Voice-First Multimodal AI System for Visual Assistance of Blind and Low-Vision Users

Abstract

Visual impairment makes everyday tasks such as recognising objects, reading printed text, avoiding obstacles, and understanding an unfamiliar place difficult without help from another person. Advances in computer vision, vision-language models, and speech interfaces make it possible to provide part of this help through an ordinary smartphone. However, many existing systems address only one task, depend on dedicated hardware, or produce descriptions that are too long, too unreliable, or too slow to be used comfortably while moving. This paper presents a literature survey of twelve studies covering visual questions from blind users, navigation and visual-aid systems, object recognition, large-model-based description and guidance, user preferences, and the vision models used in modern assistants. The studies are reviewed and compared based on their objectives, approaches, evaluation settings, findings, and limitations. The review shows that object detection, scene description, and text recognition have each matured, and that large vision-language models can generate rich descriptions. At the same time, challenges remain in grounding descriptions in what is actually visible, keeping responses concise and actionable, supporting hands-free operation, running on commodity devices, and validating systems with the people who will use them. Based on these findings, the paper presents VisionAssist, a multimodal assistant that combines YOLO11s object detection, Florence-2 scene captioning, and EasyOCR text extraction with a priority-based response generator. Users interact through a wake phrase and spoken commands, and the system replies through text-to-speech. The pipeline is exposed through a FastAPI service and a mobile web client, with a Streamlit companion interface. VisionAssist is presented as an implemented prototype rather than a validated assistive product. The paper identifies the evaluation and design challenges that need to be addressed in future work to develop reliable, safe, and user-tested assistive technology.