Assisting Blind Individuals in Navigation Using Multimodal LLM and Voice Assistance
Keywords:
Assistive navigation, Computer vision, Deep learning, DETR, LiDAR, Multimodal LLM, Object detection, Visual impairment, Voice assistance, YOLOAbstract
Visual impairment can considerably limit a person’s ability to move through unfamiliar surroundings safely and independently. Recent developments in Artificial Intelligence (AI), Computer Vision, Deep Learning, Multimodal Large Language Models (LLMs), and smartphone-based sensing technologies have facilitated the emergence of intelligent assistive navigation systems designed to support safer and more independent mobility. These systems employ object detection, distance estimation, scene understanding, voice guidance, and haptic feedback to improve environmental awareness and obstacle avoidance. This survey reviews recent research on AI-based assistive navigation systems for visually impaired individuals, with particular emphasis on object detection models, multimodal LLMs, LiDAR-based sensing, distance estimation, and real-time audio guidance. The reviewed approaches demonstrate improvements in object recognition, environmental understanding, and navigation assistance; however, several limitations remain, including dependence on specialized hardware, high computational requirements, response latency, limited offline functionality, and insufficient integration of multiple assistive features within a single application. The survey identifies these research gaps and discusses future directions toward lightweight, affordable, smartphone-based assistive navigation solutions. Integration of multimodal AI, efficient object detection, smartphone sensors, real-time distance estimation, voice assistance, vibration feedback, emergency support, and offline processing can contribute to safer, more reliable, and accessible navigation for visually impaired individuals.
References
J. E. Kim, G. Sahas, and M. Bessho, “Toward Assisting Blind Individuals in Exploring Unfamiliar Indoor Environments Using Multimodal LLM and Smartphone LiDAR,” 2025 IEEE International Conference on Consumer Electronics (ICCE), pp. 1–6, Jan. 2025.
S. Ikram, I. Sarwar Bajwa, S. Gyawali, A. Ikram, and N. Alsubaie, “Enhancing Object Detection in Assistive Technology for the Visually Impaired: A DETR-Based Approach,” IEEE Access, vol. 13, pp. 71647–71661, Apr. 2025.
G. Li, J. Xu, Z. Li, C. Chen and Z. Kan, "Sensing and Navigation of Wearable Assistance Cognitive Systems for the Visually Impaired," in IEEE Transactions on Cognitive and Developmental Systems, vol. 15, no. 1, pp. 122-133, Mar. 2023.
X. Leong and R. K. Ramasamy, "Obstacle Detection and Distance Estimation for Visually Impaired People,” IEEE Access, vol. 11, pp. 136609-136629, Nov. 2023.
M. Kobayashi and N. Motoi, “BSL: Navigation Method Considering Blind Spots Based on ROS Navigation Stack and Blind Spots Layer for Mobile Robot,” IEEE Transactions on Industry Applications, vol. 60, no. 1. 1695–1704, May 09, 2024.
J. Ni, Y. Chen, G. Tang, W. Cao, and S. X. Yang, “An Integration Model of Blind Spot Estimation and Traversable Area Detection for Indoor Robots,” IEEE Sensors Journal, vol. 25, no. 10, pp. 17850–17866, May 2025.
Y. Liu, Y. Hao, S. Gong, R.-G. Tsai, C. Qiu, and M. Wang, “Blind-Aided Target Detection Algorithm Based on Cascading Feature Pyramids with Lightweight Dual-Path Down sampling,” IEEE Access, vol. 13, pp. 173444–173459, Sep. 2025.
X. Yu and J. Saniie, “Visual Impairment Spatial Awareness System for Indoor Navigation and Daily Activities,” Journal of Imaging, vol. 11, no. 1, pp.1-9, Jan. 2025.
B. Yang, L. He, K. Liu, and Z. Yan, “VIAssist: Adapting Multi-Modal Large Language Models for Users with Visual Impairments,” 2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys), pp. 32–37, May 2024.
Z. Yang et al., “SeeWay: Vision-Language Assistive Navigation for the Visually Impaired,” 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 52–58, Oct. 2022.
R. Tapu, B. Mocanu, A. Bursuc, and T. Zaharia, “A Smartphone-Based Obstacle Detection and Classification System for Assisting Visually Impaired People,” ICCVW ’13: Proceedings of the 2013 IEEE International Conference on Computer Vision Workshops, pp. 444–451, Dec. 2013.
H. Najm, K. Elferjani, and A. Alariyibi, “Assisting Blind People Using Object Detection with Vocal Feedback,” 2022 IEEE 2nd International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA), pp. 48–52, May 2022.
V. Reddy S, B. Sathyasri, J. J. Jeya Sheela, Malarvizhi. C, S. Vanaja, and Shyam. S, “Smart Blind Assistant Using Deep Learning for the Visually Impaired Users,” 2023 Second International Conference on Augmented Intelligence and Sustainable Systems (ICAISS), pp. 680–684, Aug. 2023.
N. N. Saeed, M. A.-M. Salem, and A. Khamis, “Android-Based Object Recognition for the Visually Impaired,” 2013 IEEE 20th International Conference on Electronics, Circuits, and Systems (ICECS), pp. 645–648, Dec. 2013.
S. Md. R. Bagwan and L. J. Sankpal, “VisualPal: A Mobile App for Object Recognition for the Visually Impaired,” 2015 International Conference on Computer, Communication and Control (IC4), pp. 1–6, Sep. 2015.
