Voice-guided Mobile Assistance for the Visually Impaired

Audio-visual speech recognition (AVSR) is one of the most promising solutions for reliable speech recognition, particularly when audio is corrupted by noise. Additional visual information can be used for both automatic lip-reading and gesture recognition. Hand gestures are a form of non-verbal communication and can be used as a very important part of modern human–computer interaction systems. Currently, audio and video modalities are easily accessible by sensors of mobile devices. However, there is no out-of-the-box solution for automatic audio-visual speech and gesture recognition. This study introduces two deep neural network-based model architectures: one for AVSR and one for gesture recognition. The main novelty regarding audio-visual speech recognition lies in fine-tuning strategies for both visual and acoustic features and in the proposed end-to-end model, which considers three modality fusion approaches: prediction-level, feature-level, and model-level. The main novelty in gesture recognition lies in a unique set of spatio-temporal features, including those that consider lip articulation information. As there are no available datasets for the combined task, we evaluated our methods on two different large-scale corpora—LRW and AUTSL—and outperformed existing methods on both audio-visual speech recognition and gesture recognition tasks. We achieved AVSR accuracy for the LRW dataset equal to 98.76% and gesture recognition rate for the AUTSL dataset equal to 98.56%. The results obtained demonstrate not only the high performance of the proposed methodology, but also the fundamental possibility of recognizing audio-visual speech and gestures by sensors of mobile devices.

Modern mobile devices provide several functionalities and new ones are being added at a breakneck pace. Unfortunately browsing the menu and accessing the functions of a mobile phone is not a trivial task for visual impaired users. Low vision people typically rely on screen readers and voice commands. However, depending on the situations, screen readers are not ideal because blind people may need their hearing for safety, and automatic recognition of voice commands is challenging in noisy environments. Novel smart watches technologies provides an interesting opportunity to design new forms of user interaction with mobile phones. We present our first works towards the realization of a system, based on the combination of a mobile phone and a smart watch for gesture control, for assisting low vision people during daily life activities. More specifically we propose a novel approach for gesture recognition which is based on global alignment kernels and is shown to be effective in the challenging scenario of user independent recognition. This method is used to build a gesture-based user interaction module and is embedded into a system targeted to visually impaired which will also integrate several other modules. We present two of them: one for identifying wet floor signs, the other for automatic recognition of predefined logos.

As individuals age, challenges such as declining vision and memory can increase the risk of medication errors, particularly among the elderly and visually impaired. To address this issue, this research presents a deep learning-based Android application for accurate and accessible drug pill recognition. The system leverages a contrast-enhanced Convolutional Neural Network (CNN) trained on a diverse pill image dataset, achieving a test accuracy of 98%. Integrated with a REST API, the model enables real-time image classification via a smartphone camera. The application further enhances usability through voice-assisted feedback and visual pill details, promoting autonomy and medication adherence. This AI-driven solution bridges the gap between healthcare and technology, offering a practical tool to reduce medication errors and improve the quality of life for users with visual and cognitive impairments.

In recent years, gesture recognition and speech recognition, as important input methods in Human–Computer Interaction (HCI), have been widely used in the field of virtual reality. In particular, with the rapid development of deep learning, artificial intelligence, and other computer technologies, gesture recognition and speech recognition have achieved breakthrough research progress. The search platform used in this work is mainly the Google Academic and literature database Web of Science. According to the keywords related to HCI and deep learning, such as “intelligent HCI”, “speech recognition”, “gesture recognition”, and “natural language processing”, nearly 1000 studies were selected. Then, nearly 500 studies of research methods were selected and 100 studies were finally selected as the research content of this work after five years (2019–2022) of year screening. First, the current situation of the HCI intelligent system is analyzed, the realization of gesture interaction and voice interaction in HCI is summarized, and the advantages brought by deep learning are selected for research. Then, the core concepts of gesture interaction are introduced and the progress of gesture recognition and speech recognition interaction is analyzed. Furthermore, the representative applications of gesture recognition and speech recognition interaction are described. Finally, the current HCI in the direction of natural language processing is investigated. The results show that the combination of intelligent HCI and deep learning is deeply applied in gesture recognition, speech recognition, emotion recognition, and intelligent robot direction. A wide variety of recognition methods were proposed in related research fields and verified by experiments. Compared with interactive methods without deep learning, high recognition accuracy was achieved. In Human–Machine Interfaces (HMIs) with voice support, context plays an important role in improving user interfaces. Whether it is voice search, mobile communication, or children’s speech recognition, HCI combined with deep learning can maintain better robustness. The combination of convolutional neural networks and long short-term memory networks can greatly improve the accuracy and precision of action recognition. Therefore, in the future, the application field of HCI will involve more industries and greater prospects are expected.

BackgroundThe terms health app and medical app are often used interchangeably but do not necessarily mean the same thing. To better understand these terms and better regulate such technologies, we need distinct definitions of health and medical apps.ObjectiveThis study aimed to provide an overview of the definitions of health and medical apps from an interdisciplinary perspective. We summarized the core elements of the identified definitions for their holistic understanding in the context of digital public health.MethodsThe legal frameworks for medical device regulation in the United States, the European Union, and Germany formed the basis of this study. We then searched 6 databases for articles defining health or medical apps from an interdisciplinary perspective. The narrative literature review was supported by a forward and backward snowball search for more original definitions of health and medical apps. A qualitative analysis was conducted on the identified relevant aspects and core elements of each definition. On the basis of these findings, we developed a holistic definition of health and medical apps and created a decision flowchart to highlight the differences between the 2 types.ResultsThe legal framework showed that medical apps could be regulated as mobile medical devices, whereas there is no legal term for health apps. Our narrative literature review identified 204 peer-reviewed publications that offered a definition of health and medical apps. After screening for original definitions and applying the snowball method, 11.8% (24/204) of the publications were included in the qualitative analysis. Of these 24 publications, 22 (88%) provided an original definition of health apps and 11 (44%) described medical apps. The literature suggests that medical apps are a part of health apps. To describe health or medical apps, most definitions used the user group, a description of health, the device, the legal regulation, collected data, or technological functions. However, the regulation should not be a distinction criterion as it requires legal knowledge, which is neither suitable nor practical. An app’s intended medical or health use enables a clear differentiation between health and medical apps. Ultimately, the health aim of an app and its main target group are the only distinction criteria.ConclusionsHealth apps are software programs on mobile devices that process health-related data on or for their users. They can be used by every health-conscious person to maintain, improve, or manage the health of an individual or the community. As an umbrella term, health apps include medical apps. Medical apps share the same technological functions and devices. Health professionals, patients, and family caregivers are the main user groups. Medical apps are intended for clinical and medical purposes and can be legally regulated as mobile medical devices.

In an era of advancing human-computer interaction, "GestureSpeak" emerges as a pioneering project that facilitates hands-free control over computing devices through intuitive hand gestures and voice commands. Leveraging the power of MediaPipe and OpenCV in Python, this innovative system enables users to seamlessly navigate their digital environments without the need for traditional input devices. By harnessing the capabilities of computer vision and machine learning, GestureSpeak interprets and responds to users' gestures and vocal instructions, opening new frontiers in accessibility and user experience. This abstract offers a glimpse into the transformative potential of GestureSpeak in revolutionizing the way we interact with computers, making technology more accessible and intuitive for all users. Advancements in human-computer interaction have led to the development of innovative projects that redefine how users interact with technology. This paper introduces a novel integration of two projects: GestureSpeak and VoiceRobot, which together enable intuitive and hands-free control over computing devices using hand gestures and voice commands. VoiceRobot complements GestureSpeak by adding voice command capabilities to the interaction model. Powered by speech recognition technology, VoiceRobot enables users to control devices and launch applications using natural language. With VoiceRobot, initiating the GestureSpeak project is as simple as speaking a command. The combined system offers unique benefits, including improved accessibility for users with mobility impairments and a more natural way to interact with computing devices. By integrating gesture recognition and voice commands, the project enhances user engagement and efficiency, paving the way for future advancements in human-computer interaction. Launching the Project with VoiceRobot:A notable feature of this integrated system is the ability to launch the GestureGenie project using VoiceRobot. By speaking a predefined command, such as "Activate GestureGenie," users can initiate the hand gesture control system effortlessly. This capability highlights the seamless integration of gesture recognition and voice command technologies, showcasing the project's versatility and user-friendly design. In conclusion, the integration of GestureGenie and VoiceRobot represents a significant advancement in human-computer interaction, offering a comprehensive solution for hands-free device control. This abstract provides insight into the combined capabilities of these projects and their potential impact on accessibility, user experience, and the future of interactive computing.

Gesture recognition (GR) is an emerging and wide-ranging area of research. GR is extensively applied in sign language, Immersive game technology, and other computer interfaces, among others. People with visual impairments face challenges in completing tasks, including navigating environments, using technologies, and engaging in social interactions. Additionally, people face challenges in balancing their individuality with the need for protection in their day-to-day work. It is likely to recognize the communication of visually challenged and deaf people by recording their speech, and in comparison, with recent datasets, hence establishing their objectives. The conventional machine learning (ML) model attempts to utilize handcrafted features, but often underperforms in real-time environments. Deep learning (DL) models have become a sensation amongst investigators recently, making conventional ML techniques comparatively old. Therefore, this study presents a new approach, Enhancing Gesture Recognition for the Visually Impaired using Deep Learning and an Improved Snake Optimization Algorithm (EGRVI-DLISOA), in an IoT environment. The EGRVI-DLISOA approach is an advanced GR system powered by DL in an IoT environment, designed to provide real-time interpretation of gestures to assist the visually impaired. Initially, the EGRVI-DLISOA technique utilizes the Sobel filter (SF) technique for the noise elimination process. For feature extraction, the SqueezeNet model is utilized due to its efficiency in capturing meaningful features from complex visual data. For an accurate GR process, the long short-term memory (LSTM) approach is implemented. To fine-tune the hyperparameter values of the LSTM classifier, the improved snake optimization algorithm (ISOA) is utilized. The experimentation of the EGRVI-DLISOA technique is investigated under the hand gestures dataset. The comparison study of the EGRVI-DLISOA technique revealed a superior accuracy value of 98.62% compared to existing models.

Visually impaired individuals encounter significant challenges in their daily lives, particularly when navigating streets and public spaces. Walking in urban environments poses unique difficulties, as they must contend with obstacles, uneven surfaces, and traffic, all of which can create hazardous situations. With the growing prevalence of visual impairment, it is increasingly important to develop effective methods and technologies that can assist these individuals in safely and confidently navigating their surroundings. To address this issue, we propose SmartEye, a machine learning-based mobility assistant system that utilizes depth estimation and object detection. The system features a compact camera module mounted on the user’s glasses, which captures the environment in front of them. Through object detection and depth estimation algorithms, SmartEye analyzes the surroundings in real time, identifying obstacles and their distances. The outputs from both the object detection and depth estimation processes are then integrated to provide a comprehensive understanding of the user’s environment. This information is communicated to the individual through a speaker attached to the glasses, offering essential guidance and enhancing their mobility and safety while navigating public spaces. The proposed system achieved an absolute relative error of 0.068 and a mean average precision of 57.5 on a public dataset. Additionally, we conducted a real-world study by applying the SmartEye system to real-world street scenarios. The results demonstrated the system’s feasibility and effectiveness in assisting visually impaired individuals in navigating complex environments.

In accordance with the few kinds of virtual human control device, the application of speech recognition and gesture recognition in virtual human control was researched. The development status of limited collection isolated word recognition technology was introduced, the basic principle of data glove was analyzed, its correcting and data process technology were discussed. The principle of gesture recognition based on vision and its research status was introduced in detail. The realization of control virtual human with speech recognition and gesture recognition in engineering were introduced, and got anticipated result.

Gesture recognition is critical in human-computer communication. As observed, a plethora of current technological developments are in the works, including biometric authentication, which we see all the time in our smartphones. Hand gesture focus, a frequent human-computer interface in which we manage our devices by presenting our hands in front of a webcam, can benefit people of different backgrounds. Some of the efforts in human-computer interface include voice assistance and virtual mouse implementation with voice commands, fingertip recognition and hand motion tracking based on an image in a live video. Human Computer Interaction (HCI), particularly vision-based gesture and object recognition, is becoming increasingly important. Hence, we focused to design and develop a system for monitoring fingers using extreme learning-based hand gesture recognition techniques. Extreme learning helps in quickly interpreting the hand gestures with improved accuracy which would be a highly useful in the domains like healthcare, financial transactions and global business

VisionMate is a mobile accessibility application that utilizes artificial intelligence (AI) and helps visually impaired and blind people to carry out daily tasks independently. The device proposed integrates computer vision, speech recognition, and GPS to utilize the camera, microphone, and location services on a user's smartphone to assist users with a variety of tasks. VisionMate also provides a multitude of functions for people with visual impairments, including Optical Character Recognition (OCR) for reading text, Object Detection for finding people, currency detection for identifying denomination, Barcode Scanning to read product/food labels, and Voice Navigation for providing direction to users via voice command. Uses of cloud services (Google Vision API & Speech-to-Text API) to store incoming images and voice commands allow for the analysis of received images and voice commands. The image results are returned to the user via real-time sound using Text-to-Speech (TTS) technology. VisionMate will be developed using the React Native framework, and MongoDB will serve as the backend database for user management and authentication. Ultimately the goal of the proposed system is to improve access to and independence for visually impaired users by allowing users to better understand the environment using intelligent audio guidance.

Interacting with mobile devices can be challenging in adverse working environments. Using hand gestures for interaction can overcome severe usability issues that users face when working with mobile devices in industrial settings. This chapter is dedicated to the design, implementation, and evaluation of mobile information systems with hand gesture recognition as means for human computer interaction. The chapter provides a concise theoretical background on human gestural interaction and gesture recognition, guidelines for the design of gesture vocabularies, recommendations for proper implementation, and parameterization of robust and reliable recognition algorithms on energy-efficient 8-bit architectures. It additionally presents an elaborated process for participatory design and evaluation of gesture vocabularies to ensure high usability in the actual context of use. The chapter concludes with a case study that proves the suitability of the proposed framework for the design of efficient and reliable hand gesture-based user interfaces.

Purpose Speech recognition technology (SRT) is increasingly available and may provide opportunities for individuals with disabilities to participate in necessary activities and meaningful occupations. This inquiry methodically collects and reports on research related to SRT for individuals with disabilities and impairments. Materials and methods Using the Arksey and O’Malley framework and PRISMA guidelines, five databases were queried and indexes of 11 journals were hand-searched for relevant articles. Articles were included if they were published after 2005, involved SRT, written in English, and participants(s) had a disability. Articles were charted, categorised for level of evidence, and findings were summarised. Results and conclusions Of the 78 articles that were retrieved, 13 met inclusion criteria and were organised into 4 categories: SRT in primary and secondary education, in post-secondary education, for daily living, and without a specified context. No included articles considered SRT in vocational contexts. It was determined that SRT is a tool that may improve participation and independence of individuals with disabilities in multiple contexts. Most studies reviewed were non-empirical. Opportunities exist for providers to continue to develop a rigorous body of knowledge for the use of SRT to meet educational, vocational and daily living needs. IMPLICATIONS FOR REHABILITATION Speech recognition technology (SRT) provides a method of access that is widely available in mainstream technology devices such as personal computers, smartphones and mobile devices, and smart speakers/daily living aids. SRT may improve participation and independence of individuals with disabilities in contexts such as primary and secondary education, post-secondary education, and with independent living tasks. Many studies examining SRT are non-empirical, and there is an opportunity for assistive technology professionals and other providers who use assistive technology to continue to develop a rigorous body of knowledge.

La búsqueda de información a través de la World Wide Web es una tarea común hoy en día. Sin embargo, es sobre todo una experiencia textual. Este trabajo propone e implementa un entorno 3D para la representación de los resultados de búsqueda Web en un entorno en tres dimensiones (3D) que es controlada por gestos y reconocimiento de voz. Con el fin de hacer eso, se requieren dos escenarios: una solución de servidor basado en Servicios Web XML que permite ser consumidos por los clientes que están buscando información en la Web; y una aplicación en cliente que permite ser ejecutado como una interfaz gráfica de usuario en el escritorio mediante el uso de comandos de voz y gestos. Esta última aplicación se encarga de modelar gráficamente el entorno 3D de acuerdo con los resultados de búsqueda Web. Un prototipo del sistema propuesto fue implementado, lo que representa un conjunto de resultados de búsqueda Web en un entorno 3D. El escenario 3D es manejada por gestos y comandos de voz; el entorno 3D se basa en la administración de algunas transformaciones 3D tales como la rotación y el escalado de las caras interiores de un toroide. Una evaluación del prototipo en contextos de búsqueda sugiere que hay un alto nivel de aceptación del usuario del modelo multimedia propuesto. Las personas que probaron el prototipo han dicho que tiene una interfaz gráfica de usuario intuitiva, y que tanto el habla como el reconocimiento de gestos han sido una alternativa altamente funcional para las tareas relacionadas con la búsqueda Web. Sin embargo, hay algunos aspectos a mejorar, sobre todo en el apoyo de reconocimiento de voz de diferentes acentos. En general, el prototipo multimedia cumplió con las expectativas de una muestra de los usuarios finales, lo que demuestra que este tipo de interfaces tienen un gran potencial.

The number of older adults using mobile devices has doubled over recent years; however, many need assistance in learning how to use their device. To address this gap, we developed patient education materials teaching older Veterans how to download apps and the basics of mobile device and app use. For example, we developed step-by-step guides for three Veteran Affairs mobile apps that target mental health symptoms. Material development involved feedback from providers and older Veterans using a multi-step mixed methods evaluation process. Local technology and geriatric content experts provided initial feedback; all experts agreed the materials would be helpful to teach Veterans about mental health apps. We subsequently interviewed older Veterans (M = 78.5 years) who evaluated the materials. Over 50% of Veterans found the guides clear, articulate, and useful; 83.3% noted they would recommend to others. Lastly, providers who see older Veterans regularly rated the materials; 79% of providers rated the materials as helpful, with an average rating of 4.3 (1 = Strongly Disagree to 5 = Strongly Agree). Providers viewed the materials and apps as useful supplements to psychotherapy and especially useful for individuals who are unable to return to clinic. Overall, both providers and Veterans found the materials easy to understand and valuable for those new using mobile apps or devices. Findings from the evaluation process suggest the design of the materials may be vital to increasing the use of mental health mobile apps among older Veterans.