Voice-First Multimodal AI System for Visual Assistance of Blind and Low-Vision Users

2026-10-13T11:22:53+00:00

Visual impairment makes everyday tasks such as recognising objects, reading printed text, avoiding obstacles, and understanding an unfamiliar place difficult without help from another person. Advances in computer vision, vision-language models, and speech interfaces make it possible to provide part of this help through an ordinary smartphone. However, many existing systems address only one task, depend on dedicated hardware, or produce descriptions that are too long, too unreliable, or too slow to be used comfortably while moving. This paper presents a literature survey of twelve studies covering visual questions from blind users, navigation and visual-aid systems, object recognition, large-model-based description and guidance, user preferences, and the vision models used in modern assistants. The studies are reviewed and compared based on their objectives, approaches, evaluation settings, findings, and limitations. The review shows that object detection, scene description, and text recognition have each matured, and that large vision-language models can generate rich descriptions. At the same time, challenges remain in grounding descriptions in what is actually visible, keeping responses concise and actionable, supporting hands-free operation, running on commodity devices, and validating systems with the people who will use them. Based on these findings, the paper presents VisionAssist, a multimodal assistant that combines YOLO11s object detection, Florence-2 scene captioning, and EasyOCR text extraction with a priority-based response generator. Users interact through a wake phrase and spoken commands, and the system replies through text-to-speech. The pipeline is exposed through a FastAPI service and a mobile web client, with a Streamlit companion interface. VisionAssist is presented as an implemented prototype rather than a validated assistive product. The paper identifies the evaluation and design challenges that need to be addressed in future work to develop reliable, safe, and user-tested assistive technology.

Mahatvak Chikitsa: Automated Skin Oncology Diagnosis with Clinical Decision Support

2026-09-25T07:11:53+00:00

Skin cancer is emerging as a significant global health issue, which makes timely detection increasingly crucial. Identifying skin-related conditions at an early stage can greatly enhance treatment success, yet differentiating between various types of skin lesions through visual inspection alone remains challenging. The project Mahatvak Chikitsa presents an intelligent web-based solution designed to assist in the detection of skin cancer by leveraging deep learning techniques on dermoscopic images. For system development, dermoscopic images from the ISIC dataset are utilized and processed using preprocessing techniques along with data augmentation to enhance image clarity and expand dataset variability. Multiple convolutional neural network architectures such as MobileNet, ResNet50, EfficientNet, VGG16, and VGG19 are implemented and compared to determine the most effective model in terms of classification accuracy and performance. A Flask-powered web interface is developed, allowing users to upload images, obtain predicted skin lesion categories, and access basic treatment-related guidance through a simple and intuitive design. This system is designed to act as an assisting tool for medical professionals rather than replacing expert diagnosis. By integrating artificial intelligence with a user-friendly web application, the project seeks to promote early-stage screening, increase awareness about skin health, and support clinicians in decision-making. Ultimately, this approach can help minimize diagnostic delays and improve patient outcomes by enabling quicker and more consistent evaluation of skin lesions.

Journal of Android and IOS Applications and Testing

Voice-First Multimodal AI System for Visual Assistance of Blind and Low-Vision Users

Mahatvak Chikitsa: Automated Skin Oncology Diagnosis with Clinical Decision Support