Speech recognition for command entry in multimodal interaction

BackgroundRecent advances in natural language processing and artificial intelligence have led to widespread adoption of speech recognition technologies. In consumer health applications, speech recognition is usually applied to support interactions with conversational agents for data collection, decision support, and patient monitoring. However, little is known about the use of speech recognition in consumer health applications and few studies have evaluated the efficacy of conversational agents in the hands of consumers. In other consumer-facing tools, cognitive load has been observed to be an important factor affecting the use of speech recognition technologies in tasks involving problem solving and recall. Users find it more difficult to think and speak at the same time when compared to typing, pointing, and clicking. However, the effects of speech recognition on cognitive load when performing health tasks has not yet been explored.ObjectiveThe aim of this study was to evaluate the use of speech recognition for documentation in consumer digital health tasks involving problem solving and recall.MethodsFifty university staff and students were recruited to undertake four documentation tasks with a simulated conversational agent in a computer laboratory. The tasks varied in complexity determined by the amount of problem solving and recall required (simple and complex) and the input modality (speech recognition vs keyboard and mouse). Cognitive load, task completion time, error rate, and usability were measured.ResultsCompared to using a keyboard and mouse, speech recognition significantly increased the cognitive load for complex tasks (Z=–4.08, P<.001) and simple tasks (Z=–2.24, P=.03). Complex tasks took significantly longer to complete (Z=–2.52, P=.01) and speech recognition was found to be overall less usable than a keyboard and mouse (Z=–3.30, P=.001). However, there was no effect on errors.ConclusionsUse of a keyboard and mouse was preferable to speech recognition for complex tasks involving problem solving and recall. Further studies using a broader variety of consumer digital health tasks of varying complexity are needed to investigate the contexts in which use of speech recognition is most appropriate. The effects of cognitive load on task performance and its significance also need to be investigated.

Human computer interaction (HCI) is very crucial in our day-to-day activity. Speech is one of the essential and intuitive ways to interact with machines such as Smartphone, which has multiple sensors as microphone, camera, etc. An efficient performance speech recognition system improves interaction between man and machines by making latter more receptive to user needs. Such system has Automatic speech recognition (ASR) engine, which is facing a unique challenge of accuracy in recognition rate. By integrating acoustic signal feature vectors with the visual features, a more robust audiovisual speech recognition engine (AVSR) could be developed for real environmental scenarios. This paper presents past research and development in the field of ASR and AVSR technologies. It describes key technological perspective and admiration of the fundamental progress in ASR and AVSR. The objective of this review is to summarize and compare some of the well-known methods experimented by previous researchers, and to conclude with direction on future research proficiency in HCI system using ASR and AVSR engine.

A robotic scrub nurse (RSN) designed for safe human-robot collaboration in the operating room (OR) is presented. The RSN assists the surgical staff in the OR by delivering instruments to the surgeon and operates through a multimodal interface allowing instruments to be requested through verbal commands or touchless gestures. A machine vision algorithm was designed to recognize the hand gestures performed by the user. To ensure safe human-robot collaboration, tool-tip trajectories are planned and executed to avoid collisions with the user. Experiments were conducted to test the system when speech and gesture modalities were used to interact with the robot, separately and together. The average system times were compared while performing a mock surgical task for each modality of interaction. The effects of modality training on task completion time were also studied. It was found that training results in a significant drop of 12.92% in task completion time. Experimental results show that 95.96% of the gestures used to interact with the robot were recognized correctly, and collisions with the user were completely avoided when using a new active obstacle avoidance algorithm.

Recently, one of the most common approaches used in speech recognition is deep learning. The most advanced results have been obtained with speech recognition systems created using convolutional neural network (CNN) and recurrent neural networks (RNN). Since CNNs can capture local features effectively, they are applied to tasks with relatively short-term dependencies, such as keyword detection or phoneme- level sequence recognition. This paper presents the development of a deep learning and speech command recognition system. The Google Speech Commands Dataset has been used for training. The dataset contained 65.000 one-second-long words of 30 short English words. That is, %80 of the dataset has been used in the training and %20 of the dataset has been used in the testing. The data set consists of one-second voice commands that have been converted into a spectrogram and used to train different artificial neural network (ANN) models. Various variants of CNN are used in deep learning applications. The performance of the proposed model has reached %94.60.

While auditory training in quiet has been shown to improve cochlear implant (CI) users' speech understanding in quiet, it is unclear whether training in noise will benefit speech understanding in noise. The present study investigated whether auditory training could improve CI users' speech recognition in noise and whether training with familiar stimuli in an easy listening task (closed-set digit recognition) would improve recognition of unfamiliar stimuli in a more difficult task (open-set sentence recognition). CI users' speech understanding in noise was assessed before, during, and after auditory training with a closed-set recognition task (digits identification) in speech babble. Before training was begun, recognition of digits, Hearing in Noise Test (HINT) sentences, and IEEE sentences presented in steady speech-shaped noise or multitalker speech babble was repeatedly measured to establish a stable estimate of baseline performance. After completing baseline measures, participants trained at home on their personal computers using custom software for approximately 30 mins/day, 5 days/wk, for 4 wks, for a total of 10 hrs of training. Participants were trained only to identify random sequences of three digits presented in speech babble, using a closed-set task. During training, the signal-to-noise ratio was adjusted according to subject performance; auditory and visual feedback was provided. Recognition of digits, HINT sentences, and IEEE sentences in steady noise and speech babble was remeasured after the second and fourth week of training. Training was stopped after the fourth week, and subjects returned to the laboratory 1 mo later for follow-up testing to see whether any training benefits had been retained. Mean results showed that the digit training in babble significantly improved digit recognition in babble (which was trained) and in steady noise (which was not trained). The training benefit generalized to improved HINT and IEEE sentence recognition in both types of noise. Training benefits were largely retained in follow-up measures made 1 mo after training was stopped. The results demonstrated that auditory training in noise significantly improved CI users' speech performance in noise, and that training with simple stimuli using an easy closed-set listening task improved performance with difficult stimuli and a difficult open-set listening task.

The advent of new devices, technology, machine learning techniques, and the availability of free large speech corpora results in rapid and accurate speech recognition. In the last two decades, extensive research has been initiated by researchers and different organizations to experiment with new techniques and their applications in speech processing systems. There are several speech command based applications in the area of robotics, IoT, ubiquitous computing, and different human-computer interfaces. Various researchers have worked on enhancing the efficiency of speech command based systems and used the speech command dataset. However, none of them catered to noise in the same. Noise is one of the major challenges in any speech recognition system, as real-time noise is a very versatile and unavoidable factor that affects the performance of speech recognition systems, particularly those that have not learned the noise efficiently. We thoroughly analyse the latest trends in speech recognition and evaluate the speech command dataset on different machine learning based and deep learning based techniques. A novel technique is proposed for noise robustness by augmenting noise in training data. Our proposed technique is tested on clean and noisy data along with locally generated data and achieves much better results than existing state-of-the-art techniques, thus setting a new benchmark.

This paper evaluates the performance of a multimodal interface under exerted conditions in a natural field setting. The subjects in the present study engaged in a strenuous activity while multimodally performing map-based tasks using handheld computing devices. This activity made the users breathe heavily and become fatigued during the course of the study. We found that the performance of both speech and gesture recognizers degraded as a function of exertion, while the overall multimodal success rate was stable. This stabilization is accounted for by the mutual disambiguation of modalities, which increases significantly with exertion. The system performed better for subjects with a greater level of physical fitness, as measured by their running speed, with more stable multimodal performance and a later degradation of speech and gesture recognition as compared with subjects who were less fit. The findings presented in this paper have a significant impact on design decisions for multimodal interfaces targeted towards highly mobile and exerted users in field environments.

Purpose Basic capturing of emotion on user experience of web applications and browsing is important in many ways. Quite often, online user experience is studied via tangible measures such as task completion time, surveys and comprehensive tests from which data attributes are generated. Prediction of users’ emotion and behaviour in some of these cases depends mostly on task completion time and number of clicks per given time interval. However, such approaches are generally subjective and rely heavily on distributional assumptions making the results prone to recording errors. This paper aims to propose a novel method – a window dynamic control system – that addresses the foregoing issues. Design/methodology/approach Primary data were obtained from laboratory experiments during which 44 volunteers had their synchronized physiological readings – skin conductance response, skin temperature, eye movement behaviour and users activity attributes taken by biosensors. The window-based dynamic control system (PHYCOB I) is integrated to the biosensor which collects secondary data attributes from these synchronized physiological readings and uses them for two purposes: for detection of both optimal emotional responses and users’ stress levels. The method’s novelty derives from its ability to integrate physiological readings and eye movement records to identify hidden correlates on a webpage. Findings The results from the analyses show that the control system detects basic emotions and outperforms other conventional models in terms of both accuracy and reliability, when subjected to model comparison – that is, the average recoverable natural structures for the three models with respect to accuracy and reliability are more consistent within the window-based control system environment than with the conventional methods. Research limitations/implications Graphical simulation and an example scenario are only provided for the control’s system design. Originality/value The novelty of the proposed model is its strained resistance to overfitting and its ability to automatically assess user emotion while dealing with specific web contents. The procedure can be used to predict which contents of webpages cause stress-induced emotions to users.

Mixed Reality (MR) is the next evolution of human interacting with the computer as MR has the ability to combine the physical environment and digital environment and making them coexist with each other. Interaction is still a huge research area in Augmented Reality (AR) but very less in MR, this is due to current advanced MR display techniques still not robust and intuitive enough to let the user to naturally interact with 3D content. New techniques on user interaction have been widely studied, the advanced technique in interaction when the system able to invoke more than one input modalities. Multimodal interaction undertakes to deliver intuitive multiple objects manipulation with gestures. This paper discusses the multimodal interaction technique using gesture and speech which the proposed experimental setup to implement multimodal in the MR interface. The real hand gesture is combined with speech inputs in MR to perform spatial object manipulations. The paper explains the implementation stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, spatial manipulation for selection and scaling using multimodal interaction has been invoked, and this paper ends with a discussion.

Portable devices come with different limitations in user interaction like limited display size, small keyboard, and different sorts of input and output capabilities. With the advance of speech recognition and speech synthesis technologies, their complementary use becomes attractive for mobile devices in order to implement real multimodal user interaction. However, current systems and formats do not sufficiently integrate advanced multimodal interactions. We introduce an advanced generic multimodal interaction and rendering system (MIRS) dedicated for mobile devices. MIRS incorporates efficient processing of XML specification languages for limited, mobile devices and comes with the XML-based dialog and interface specification language (DISL). DISL can be considered as an UIML subset, which is enhanced by the means of state-oriented dialog specifications. The dialog specification is based on ODSN (object oriented dialog specification notation), which has been introduced to define user interface control by means of interaction states with transition rules.

Speech recognition can be a powerful tool for use in human-computer interaction. Many researchers are investigating the use of speech recognition systems for dictation-based activities, resulting in dramatic improvements in recent years. However, this same experimentation has confirmed that recognition errors and the delays inherent with speech recognition result in unacceptably long task completion times and error rates for cursor control tasks. This study explores the potential of a speech-controlled grid-based cursor control mechanism. An experiment evaluated two alternative grid-based solutions, both using 3-3 grids. One provided a single cursor in the middle of the grid. The second allows users to select a target using any of nine cursors. The results confirm that the nine-cursor solution allowed users to select targets of varying size, distance and direction significantly faster than the one-cursor solution. Overall results are encouraging when compared to earlier evaluations of other speech-based cursor control solutions.

In this article, a learning mechanism and a multimodal interface are presented. The learning mechanism allows the robot to learn concepts of the environment by human tutelage. The multimodal interface employs mechanisms of face detection, pose estimation, object saliency and speech recognition. The learning mechanism and multimodal interface are evaluated in the context of concept learning.

We have constructed a new speech data corpus consisting of the utterances of 221 elderly Japanese people (average age: 79.2) with the aim of improving the accuracy of automatic speech recognition (ASR) for the elderly. ASR is a beneficial modality for people with impaired vision or limited hand movement, including the elderly. However, speech recognition systems using standard recognition models, especially acoustic models, have been unable to achieve satisfactory performance for the elderly. Thus, creating more accurate acoustic models of the speech of elderly users is essential for improving speech recognition for the elderly. Using our new corpus, which includes the speech of elderly people living in three regions of Japan, we conducted speech recognition experiments using a variety of DNN-HNN acoustic models. As training data for our acoustic models, we examined whether a standard adult Japanese speech corpus (JNAS), an elderly speech corpus (S-JNAS) or a spontaneous speech corpus (CSJ) was most suitable, and whether or not adaptation to the dialect of each region improved recognition results. We adapted each of our three acoustic models to all of our speech data, and then re-adapt them using speech from each region. Without adaptation, the best recognition results were obtained when using the S-JNAS trained acoustic models (total corpus: 21.85% Word Error Rate). However, after adaptation of our acoustic models to our entire corpus, the CSJ trained models achieved the lowest WERs (entire corpus: 17.42%). Moreover, after readaptation to each regional dialect, the CSJ trained acoustic model with adaptation to regional speech data showed tendencies of improved recognition rates. We plan to collect more utterances from all over Japan, so that our corpus can be used as a key resource for elderly speech recognition in Japanese. We also hope to achieve further improvement in recognition performance for elderly speech.

The purpose of this exploratory study was to probe the cognitive processes of undergraduate students at varying mental rotation ability (MRA) levels during mental rotation tasks. The study group consisted of 14 undergraduate students from the Psychological Counseling and Guidance Department and Computer Education and Instructional Technologies Department at Hacettepe University. Firstly, participants’ mental rotation ability levels were determined. Secondly, participants were grouped according to their mental rotation test scores. Then, they were given two digital tangram problems with different difficulty levels. During their interaction with Tangram problems, their eye movements were recorded. The Mann Whitney U test was run to compare differences in eye movements across different mental rotation ability level groups. As a result, it was found that while there were significant differences in eye movement metrics between different mental rotation ability levels for the easy tangram problem, there were no significant differences observed in eye movement metrics for the difficult tangram problem. In addition, examining completion time showed that while easy tangram problem completion time was significantly shorter for people with a higher score on the mental rotation test, there were no significant differences between completion times of different level mental rotation ability groups when solving the difficult tangram problem. Since the spatial ability is an important predictor of task performance and completion time, it is suggested that adaptive learning environments should be designed to compensate for low MRA level learners.

In this paper, improvement of the efficiency of data entry operations by adding a speech input function is reported. We implemented the multimodal interface on the desktop terminal for performance evaluation. An efficient and pleasant-to-use communication between man and machine is expected to result from the combination of speech input and a multimodal interface which can compensate for defects of speech recognition. In this paper, we apply the system to address data entry and evaluate its performances. Users can enter the address data by speech input, by menu selection from a display of ten words in one field, or by one-by-one selection display of the next candidate word, with input by keyboard. In the evaluation experiments, 25 operators performed 22,860 address data entry tasks and were instructed to use the speech input function in half of the tasks. From the evaluation results, the increase of operation time due to an increased number of entry data was reduced, and operation time was shortened by about 12% using the speech input function. The recognition rate of speech containing pauses is found to be significantly lower. © 1999 Scripta Technica, Syst Comp Jpn, 30(9): 64–73, 1999