Multi Modal Fusion Framework of Thermal Imaging and Low Resolution LiDAR for Pedestrian Detection in Adverse Weather
Keywords:
Adverse weather, Low resolution LiDAR, Multi modal fusion, Pedestrian detection, Thermal imagingAbstract
Background: Adverse weather conditions (fog, rain, snow, low light) severely degrade RGB and high‑resolution LiDAR-based pedestrian detection, compromising safety in autonomous driving and surveillance. Low‑resolution (4–16 beam) LiDAR is cost‑effective and privacy‑preserving but suffers from extreme spatial sparsity, while thermal imaging provides weather‑agnostic dense texture yet lacks geometric precision.
Purpose: This paper proposes a novel multi‑modal fusion framework that synergistically integrates thermal infrared imagery with low‑resolution smart LiDAR to achieve robust pedestrian detection under adverse weather.
Methods: The framework comprises a lightweight thermal CNN for heat distribution and edge extraction, a PointPillars‑style sparse convolution branch for LiDAR point clouds, and a cross‑attention fusion module augmented with a learned mask‑guided interpolation to densify sparse LiDAR features. Experiments are conducted on augmented KAIST and FLIR datasets with physics‑based fog, rain, and snow, plus real‑world validation.
Findings: The proposed method achieves 89.3% mean average precision (mAP@0.5) across all adverse conditions, outperforming thermal‑only by 15.3% and LiDAR‑only by 24.6%. Under dense fog (visibility <50 m), it retains 88.3% mAP, a mere 4.1% drop from clear weather. The system runs at 50 fps on an RTX 3090 and 25 fps on a Jetson Orin after INT8 quantization.
Conclusion: Fusing low‑resolution LiDAR with thermal imaging effectively compensates for each modality’s weaknesses, enabling all‑weather pedestrian detection at low sensor cost. Recommendation: Deploy the framework in cost‑sensitive ADAS, intelligent surveillance, and outdoor robotics where all‑weather reliability and privacy are essential. Future integration with radar or federated learning is suggested.
References
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA, 2012, pp. 3354–3361.
M. Bijelic, T. Gruber, and W. Ritter, “A benchmark for LiDAR sensors in fog: Is detection breaking down?,” 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China, 2018, pp. 760–767.
Y. Zhang, A. Carballo, H. Yang, and K. Takeda, “Perception and sensing for autonomous vehicles under adverse weather conditions: A survey,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 196, pp. 146–177, Feb. 2023.
C. Sakaridis, D. Dai, and L. Van Gool, “Semantic foggy scene understanding with synthetic data,” International Journal of Computer Vision, vol. 127, no. 9, pp. 1378–1394, 2019.
B. Jiang, J. Wang, G. Ren, M. Zhi, Z. Yu, Y. Zhang, P. Ren, and S. Jia, “Research on pedestrian detection method based on multispectral intermediate fusion using YOLOv7,” Scientific Reports, vol. 15, May 2025.
X. Wang, J. Lin, L. Yang, and S. Wang, “A review of point cloud 3D object detection methods based on deep learning,” in Computer Applications, CCF Conference of Computer Applications, Communications in Computer and Information Science, vol. 1959, Springer, 2023, pp. 30–39.
I. Gakhar, A. Gupta, A. Guha, A. Agarwal and U. Verma, "Fourier Domain Adaptation for Traffic Light Detection in Adverse Weather*," 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Honolulu, HI, USA, 2025, pp. 816-825.
M. Oliveira, R. Cerqueira, J. R. Pinto, J. Fonseca, and L. F. Teixeira, “Multimodal PointPillars for efficient object detection in autonomous vehicles,” in IEEE Transactions on Intelligent Vehicles, vol. 10, no. 1, pp. 81–91, Jan. 2025.
A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “PointPillars: Fast encoders for object detection from point clouds,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 12697–12705.
S. Hwang, J. Park, N. Kim, Y. Choi, and I. S. Kweon, “Multispectral pedestrian detection: Benchmark dataset and baseline,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015, pp. 1037–1045.
B. Ghari, A. Tourani, A. Shahbahrami, and G. Gaydadjiev, “Pedestrian detection in low-light conditions: A comprehensive survey,” Image and Vision Computing, vol. 148, 2024.
Q. Yan and Y. Wang, “ThermRad: A multi-modal dataset for robust 3D object detection under challenging conditions,” arXiv preprint arXiv:2308.10161, 2023.
C. Sakaridis, D. Dai, S. Hecker, and L. Van Gool, “Model adaptation with synthetic and real data for semantic dense foggy scene understanding,” in Computer Vision – ECCV 2018, Cham, Switzerland: Springer, vol. 11217, 2018.
S. Acharjee, S. Ganguly, A. Ali, and J. Frnda, “Comprehensive performance analysis of object detection algorithms across diverse sensor modalities,” IEEE Sensors Reviews, vol. 2, pp. 304–323, 2025.
E. Palladin, R. Dietze, P. Narayanan, M. Bijelic, and F. Heide, “SAMFusion: Sensor-adaptive multimodal fusion for 3D object detection in adverse weather,” in Computer Vision – ECCV 2024, Lecture Notes in Computer Science, Springer, 2024.
Y. Li, “Multispectral pedestrian detection in autonomous driving: A review,” IEIE Transactions on Smart Processing & Computing, vol. 10, no. 1, pp. 10–16, 2021.
H. Xu, S. Huang, Y. Yang, X. Chen, and S. Hu, “Deep learning-based pedestrian detection using RGB images and sparse LiDAR point clouds,” IEEE Transactions on Industrial Informatics, vol. 20, no. 5, pp. 7149–7161, May 2024.
J. Guang, Z. Hu, S. Wu, Q. Zhang, and J. Liu, “RPEA: A residual path network with efficient attention for 3D pedestrian detection from LiDAR point clouds,” Expert Systems with Applications, vol. 255, 2024.
Z. Tao, Y. Zhou, Y. Wan, X. Liang, and Y. Li, “Bidirectional enhancement and robust fusion for 3D object detection under complex lighting conditions,” The Visual Computer, vol. 41, no. 13, pp. 11427–11444, 2025.
H. T. Tran and I.-P. Hong, “Adaptive fusion of LiDAR and camera data for enhanced precision in 3D object detection for autonomous driving,” IEEE Access, vol. 13, pp. 155474–155481, 2025.
S. Yu, X. Wang, Y. Ci, Y. Li, M. Zhou, X. Dong, and M. Cai, “MSPE-Fusion: A multimodal 3D object detection method with multi-sensor perception enhanced fusion,” Neurocomputing, vol. 645, Art. no. 130486, 2025.
V. A. Sindagi, Y. Zhou, and O. Tuzel, “MVX-Net: Multimodal VoxelNet for 3D object detection,” 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 2019, pp. 7276–7282.
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” Computer Vision – ECCV 2020, pp. 213–229, 2020.
H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A multimodal dataset for autonomous driving,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 11621–11631.
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” in Computer Vision – ECCV 2020, Lecture Notes in Computer Science, Springer, vol. 12346, pp. 213–229, 2020.
W. E. K. Middleton, “Vision through the atmosphere,” in Geophysik II/Geophysics II, Berlin, Heidelberg: Springer, pp. 254–287, 1957.
D. Dai and L. Van Gool, “Dark model adaptation: Semantic image segmentation from daytime to nighttime,” 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 2018, pp. 3819-3824.
L. Neumann, M. Karg, S. Zhang, C. Scharfenberger, E. Piegert, S. Mistr, O. Prokofyeva, R. Thiel, A. Vedaldi, A. Zisserman, and B. Schiele, “NightOwls: A pedestrians at night dataset,” in Computer Vision – ACCV 2018, vol. 11361, Springer, pp. 691–705, 2019.
E. J. McCartney, Optics of the Atmosphere: Scattering by Molecules and Particles. New York, NY, USA: Wiley, 1976.
P. Tumas, A. Nowosielski and A. Serackis, "Pedestrian Detection in Severe Weather Conditions," in IEEE Access, vol. 8, pp. 62775-62784, 2020.
C. Sakaridis, D. Dai, and L. Van Gool, “ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding,” 2021 IEEE/CVF International Conference on Computer Vision2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021, pp. 10765–10775.
A. Pawar, P. Kadam and G. Yadav, "Night-Time Pedestrian Detection Using Deep Learning and RGB-Thermal Fusion," 2026 International Conference on Next-Gen Quantum and Advanced Computing: Algorithms, Security, and Beyond (NQComp), Bangalore, India, 2026, pp. 730-735.
Y. Yan, Y. Mao, and B. Li, “SECOND: Sparsely embedded convolutional detection,” Sensors, vol. 18, no. 10, Oct. 2018.
J. Ahn, K. Nakashima, K. Yoshino, Y. Iwashita and R. Kurazume, "Gait Sequence Upsampling using Diffusion Models for Single LiDAR Sensors," 2025 IEEE/SICE International Symposium on System Integration (SII), Munich, Germany, 2025, pp. 658-664.
J. You and Y.-K. Kim, “Up-Sampling Method for Low-Resolution LiDAR Point Cloud to Enhance 3D Object Detection in an Autonomous Driving Environment,” Sensors, vol. 23, no. 1, Dec. 2022.
X. Cai, J. Tang, H. Ding, Y. Zhao and Y. Li, "Robust Vehicle and Pedestrian Detection in Adverse Weather via a Weather-Aware Transformer," in IEEE Signal Processing Letters, vol. 33, pp. 1946-1950, 2026.
T. Guo, C. P. Huynh and M. Solh, "Domain-adaptive pedestrian detection in thermal images," 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 1660-1664.
S. Vora, A. H. Lang, B. Helou, and O. Beijbom, “PointPainting: Sequential fusion for 3D object detection,” 020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 4604–4612.
M. Lade, B. Pardhasaradhi and L. R. Cenkeramaddi, "Thermal Imaging-based Perception for Automotive Edge AI Applications using a Lightweight Custom YOLO Model," in IEEE Sensors Journal, pp. 1-1, 2026.
Y. Tatebe, D. Deguchi, Y. Kawanishi, I. Ide, H. Murase, and U. Sakai, “Can We Detect Pedestrians Using Low-Resolution LIDAR? - Integration of Multi-Frame Point-Clouds,” Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2017, vol. 5, pp. 157–164.
R. Roriz, J. Cabral and T. Gomes, "Automotive LiDAR Technology: A Survey," in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 6282-6297, July 2022.
J. Levinson and S. Thrun, “Unsupervised Calibration for Multi-Beam Lasers,” Springer Tracts in Advanced Robotics, vol. 79, pp. 179–193, 2014.
A. K. Gurumadaiah, J. Park, J. -H. Lee, J. Kim and S. Kwon, "Precise Synchronization Between LiDAR and Multiple Cameras for Autonomous Driving: An Adaptive Approach," in IEEE Transactions on Intelligent Vehicles, vol. 10, no. 3, pp. 2152-2162, March 2025.
S. Wang et al., "Temporal and Spatial Online Integrated Calibration for Camera and LiDAR," 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China, 2022, pp. 3016-3022.
P. Wang, “Research on Comparison of LiDAR and Camera in Autonomous Driving,” Journal of Physics: Conference Series, vol. 2093, no. 1, p. 012032, Nov. 2021.
A. Howard et al., "Searching for MobileNetV3," in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 1314-1324.
F. Altay and S. Velipasalar, "The Use of Thermal Cameras for Pedestrian Detection," IEEE Sensors Journal, vol. 22, no. 12, pp. 11489-11498, Jun. 2022.
C. Choy, J. Gwak and S. Savarese, "4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 3070-3079.
A. Vaswani, “Attention Is All You Need,” arXiv:1706.03762, Jun 2017.
S. Acharjee, S. Ganguly, A. Ali and J. Frnda, "Comprehensive Performance Analysis of Object Detection Algorithms Across Diverse Sensor Modalities," in IEEE Sensors Reviews, vol. 2, pp. 304-322, 2025.
K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, "Masked Autoencoders Are Scalable Vision Learners," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 15979-15988.
N. V. Afifah and S. Erniwati, “YOLOv8 for object detection: a comprehensive review of advances, techniques, and applications,” International Journal of Advanced Computing and Informatics, vol. 2, no. 1, pp. 53–61, Jul. 2025.
Z. Zheng et al., "Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation," IEEE Transactions on Cybernetics, vol. 52, no. 8, pp. 8574-8586, Aug. 2022.
S. Yu et al., “MSPE-Fusion: A multimodal 3D object detection method with multi-sensor perception enhanced fusion,” Neurocomputing, vol. 645, Sep. 2025.
H. T. Tran and I. -P. Hong, "Adaptive Fusion of LiDAR and Camera Data for Enhanced Precision in 3D Object Detection for Autonomous Driving," IEEE Access, vol. 13, pp. 155474-155481, 2025.
M. T. Bakrou, H. Selamat, and A. Aburaya, “Adaptive Transformer-Based Tri-Modal Fusion of RGB, Thermal, and LiDAR Data for Robust Object Detection in UAV Surveillance,” Communications in Computer and Information Science, vol. 2956, pp. 322–336, 2026.
S. G. Narasimhan and S. K. Nayar, “Vision and the Atmosphere,” International Journal of Computer Vision, vol. 48, no. 3, pp. 233–254, 2002.
K. Garg and S. K. Nayar, "Detection and removal of rain from videos," Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Washington, DC, USA, 2004, pp. I-I.
E. Riba, D. Mishkin, D. Ponsa, E. Rublee and G. Bradski, "Kornia: an Open Source Differentiable Computer Vision Library for PyTorch," 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass, CO, USA, 2020, pp. 3663-3672.
J. Fu et al., "Dual Attention Network for Scene Segmentation," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 3141-3149.
M. Karvat and S. Givigi, "Adver-City: Open-source multi-modal dataset for collaborative perception under adverse weather conditions," 2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC), Gold Coast, Australia, 2025, pp. 740-746.
