Physical workload during use of speech recognition and traditional computer input devices

The use of speech recognition in the area of office automation has received little experimental evaluation. Most speech recognition studies have involved testing in military command and control or aircraft environments. Speech control of spreadsheet and word processing programs is discussed by various manufacturers and in the popular press, yet no empirical data have been presented to substantiate its use in these applications. In one of the few studies applicable to the office environment (Murray, Van Praag, and Gilfoil, 1983), the use of speech to control cursor motion was found to be less efficient than the use of traditional cursor keys. To justify the expense of adding equipment to an office workstation, a significant increase in worker productivity needs to be demonstrated. The dedicated word processing operator, involved in the task of transcribing large quantities of textual data from hardcopy or dictation tapes, was chosen for examination in this study. The majority of word processing operator keystroke activity is within the QWERTY portion of the text entry keyboard. However, many of the necessary control operations used during text entry are located on dedicated or soft function keys. These function keys are often located well away from the QWERTY portion of the keyboard, interrupting the touch typist's activity in order to locate and activate the appropriate key. In this study, the use of speech input to select operations normally assigned to function keys was evaluated. It was hypothesized that the operator would experience less distraction in the primary task of text entry when operations were selected verbally rather than by manual activation of the function keys. This would be reflected in reduced text entry time and possible reduction in keystroke errors when selecting operations. In contrast to the earlier study examining speech control of a cursor positioning task in which speech was found to be inferior to manual control, this study used speech to control discrete operations. Subjects were experienced word processing operators. The primary task was transcription of text documents requiring frequent format changes. The subjects were placed in two groups. Group I was trained to operate a word processing package in which control operations were selected by using hard and soft function keys located above and to the right of the QWERTY keypad. Group II was trained to use the identical word processing software, with the difference being that control operations were selected by using speech input. Data collected and analyzed included time to complete the text entry task, number of entry errors, number of command errors, and subjective ratings of operator satisfaction with the system used.

BackgroundRecent advances in natural language processing and artificial intelligence have led to widespread adoption of speech recognition technologies. In consumer health applications, speech recognition is usually applied to support interactions with conversational agents for data collection, decision support, and patient monitoring. However, little is known about the use of speech recognition in consumer health applications and few studies have evaluated the efficacy of conversational agents in the hands of consumers. In other consumer-facing tools, cognitive load has been observed to be an important factor affecting the use of speech recognition technologies in tasks involving problem solving and recall. Users find it more difficult to think and speak at the same time when compared to typing, pointing, and clicking. However, the effects of speech recognition on cognitive load when performing health tasks has not yet been explored.ObjectiveThe aim of this study was to evaluate the use of speech recognition for documentation in consumer digital health tasks involving problem solving and recall.MethodsFifty university staff and students were recruited to undertake four documentation tasks with a simulated conversational agent in a computer laboratory. The tasks varied in complexity determined by the amount of problem solving and recall required (simple and complex) and the input modality (speech recognition vs keyboard and mouse). Cognitive load, task completion time, error rate, and usability were measured.ResultsCompared to using a keyboard and mouse, speech recognition significantly increased the cognitive load for complex tasks (Z=–4.08, P<.001) and simple tasks (Z=–2.24, P=.03). Complex tasks took significantly longer to complete (Z=–2.52, P=.01) and speech recognition was found to be overall less usable than a keyboard and mouse (Z=–3.30, P=.001). However, there was no effect on errors.ConclusionsUse of a keyboard and mouse was preferable to speech recognition for complex tasks involving problem solving and recall. Further studies using a broader variety of consumer digital health tasks of varying complexity are needed to investigate the contexts in which use of speech recognition is most appropriate. The effects of cognitive load on task performance and its significance also need to be investigated.

Speech recognition can be a powerful tool for individuals with physical disabilities that hinder their ability to use traditional input devices. State-of-the-art speech recognition systems typically provide mechanisms for both data entry and cursor control, but the researchers continue to investigate methods of improving these interactions. Numerous researchers are investigating methods to improve the underlying technologies that make speech recognition possible and others focus on understanding the difficulties users experience using dictation-oriented applications, but few researchers have investigated the issues involved in speech-based cursor control. In this article, we describe a study that investigates the efficacy of two variations of a standard speech-based cursor control mechanism. One employs the standard mouse cursor while the second provides a predictive cursor designed to help users compensate for the delays often associated with speech recognition. As expected, larger targets and shorter distances resulted in shorter target selection times while larger targets also resulted in fewer errors. Although there were no differences between the standard and predictive cursors, a relationship between the delays associated with spoken input, the speed at which the cursor moves, and the minimum size for targets that can be reliably selected emerged that can guide the application of similar speech-based cursor control mechanisms as well as future research.

Eye tracking is a process that allows an observers gaze to be determined in real time by measuring their eye movements. Recent work has examined the possibility of using gaze control as an alternative input modality in interactive applications. Alternative means of interaction are especially important for disabled users for whom traditional techniques, such as mouse and keyboard, may not be feasible. This paper proposes a novel combination of gaze and voice commands as a means of hands free interaction in a paint style program. A drawing application is implemented which is controllable by input from gaze and voice. Voice commands are used to activate drawing which allow gaze to be used only for positioning the cursor. In previous work gaze has also been used to activate drawing using dwell time. The drawing application is evaluated using subjective responses from participant user trials. The main result indicates that although gaze and voice offered less control that traditional input devices, the participants reported that it was more enjoyable.

This article describes the findings from a small-scale research project involving 14 test subjects. Experiments were conducted with student nurses and their use of speech recognition technologies. Speech and keyboard were compared as input methods. A hypothesis is put forward and questionnaires and experiments were devised to test the hypothesis. The experiments involved a form of software monitoring, which is discussed. Results from the data analysis of the experiments are included but not the results from the analysis of the questionnaires. During the discussion, summary data from the questionnaires are included. Issues such as usability, error rates and hardware problems are discussed. A brief conclusion considers the future of the keyboard as the predominant computer interface.

Automatic measurement of speech recognition performance: a comparison of six speaker-dependent recognition devices

To conduct a replication study to validate previously identified significant risks and inefficiencies associated with the use of speech recognition (SR) for documentation within an electronic health record (EHR) system. Thirty-five emergency department clinicians undertook randomly allocated clinical documentation tasks using keyboard and mouse (KBM) or SR using a commercial EHR system. The experiment design, setting, and tasks (E2) replicated an earlier study (E1), while technical integration issues that may have led to poorer SR performance were addressed. Complex tasks were significantly slower to complete using SR (16.94%) than KBM (KBM: 191.9 s, SR: 224.4 s; p = 0.009; CI, 11.9-48.3), replicating task completion times observed in the earlier experiment. Errors (non-typographical) were significantly higher with SR compared with KBM for both simple (KBM: 3, SR: 84; p < 0.001; CI, 1.5-2.5) and complex tasks (KBM: 23, SR: 53; p = 0.001; CI, 0.5-1.0), again replicating earlier results (E1: 170, E2: 163; p = 0.660; CI, 0.0-0.0). Typographical errors were reduced significantly in the new study (E1: 465, E2: 150; p < 0.001; CI, 2.0-3.0). The results of this study replicate those reported earlier. The use of SR for clinical documentation within an EHR system appears to be consistently associated with decreased time efficiencies and increased errors. Modifications implemented to optimize SR integration in the EHR seem to have resulted in minor improvements that did not fundamentally change overall results. This replication study adds further evidence for the poor performance of SR-assisted clinical documentation within an EHR. Replication studies remain rare in informatics literature, especially where study results are unexpected or have significant implication; such studies are clearly needed to avoid overdependence on the results of a single study.

Proprioceptive information about movement is transmitted to the central nervous system by a variety of receptor types, which are widely distributed among the muscles, joints, and skin. Muscle spindles are known to be an important and reliable source of information for the perception of movement kinematics. Previous studies that focused on the characteristics of single muscle spindle firing patterns have left the impression that each receptor fires in relation to a number of kinematic variables, leaving the following question unanswered: what role is played by the ensemble of muscle spindles within the same muscle or within synergistic muscles? The study described in this paper addressed whether the perception of joint position and velocity is based on the net input of muscle spindles residing in all synergistic muscles crossing a joint. Normal human adults performed a motor coordination task that required perception of joint velocity and dynamic position at the wrist. The task was to open the left hand briskly as the right wrist was passively rotated in the flexion direction through a prescribed target angle. In randomly occurring trials, the tendons to three muscles [extensor carpi radialis (ECR), extensor carpi ulnaris (ECU), and extensor digitorum (ED)] were vibrated either individually or in different combinations during the performance of the motor task. Tendon vibration is known to distort muscle spindle firing patterns, and consequently, kinesthesia. By comparing performance errors with and without tendon vibration, the relative influences of muscle spindles residing in ECR, ECU, and ED were quantified. Vibration of the individual ECR, ECU, or ED tendons produced systematic undershoot errors in performance, consistent with the misperception of wrist velocity and dynamic position. Performance errors were larger when combinations of, rather than individual, muscle tendons were vibrated. The error resulting from simultaneous vibration of ECR and ECU was roughly equal to the sum of the errors produced by vibration of the individual tendons. These effects of vibrating synergistic tendons at the wrist suggest that kinesthesia is derived from the integrated input of muscle spindles from all synergistic muscles.

In an era of advancing human-computer interaction, "GestureSpeak" emerges as a pioneering project that facilitates hands-free control over computing devices through intuitive hand gestures and voice commands. Leveraging the power of MediaPipe and OpenCV in Python, this innovative system enables users to seamlessly navigate their digital environments without the need for traditional input devices. By harnessing the capabilities of computer vision and machine learning, GestureSpeak interprets and responds to users' gestures and vocal instructions, opening new frontiers in accessibility and user experience. This abstract offers a glimpse into the transformative potential of GestureSpeak in revolutionizing the way we interact with computers, making technology more accessible and intuitive for all users. Advancements in human-computer interaction have led to the development of innovative projects that redefine how users interact with technology. This paper introduces a novel integration of two projects: GestureSpeak and VoiceRobot, which together enable intuitive and hands-free control over computing devices using hand gestures and voice commands. VoiceRobot complements GestureSpeak by adding voice command capabilities to the interaction model. Powered by speech recognition technology, VoiceRobot enables users to control devices and launch applications using natural language. With VoiceRobot, initiating the GestureSpeak project is as simple as speaking a command. The combined system offers unique benefits, including improved accessibility for users with mobility impairments and a more natural way to interact with computing devices. By integrating gesture recognition and voice commands, the project enhances user engagement and efficiency, paving the way for future advancements in human-computer interaction. Launching the Project with VoiceRobot:A notable feature of this integrated system is the ability to launch the GestureGenie project using VoiceRobot. By speaking a predefined command, such as "Activate GestureGenie," users can initiate the hand gesture control system effortlessly. This capability highlights the seamless integration of gesture recognition and voice command technologies, showcasing the project's versatility and user-friendly design. In conclusion, the integration of GestureGenie and VoiceRobot represents a significant advancement in human-computer interaction, offering a comprehensive solution for hands-free device control. This abstract provides insight into the combined capabilities of these projects and their potential impact on accessibility, user experience, and the future of interactive computing.

Currently, the use of speech recognition is increaseingly in many applications such as mobile device interaction, interactive voice response system, voice search, voice dictation and voice identification. The heart of such applications is speech features needed to represent input signals. However, in real applications, speech signals are sampled with various sampling rates. The different sampling rates of input speech lead to the different features. This makes the speech recognition rate dropping. Therefore, this paper proposes an independent resolution descriptor based on fractal codes obtained by fractal encoding and decoding processes. The encoding process extracts fractal codes from partitioned speech signals, whereas the decoding process reconstructs independent resolution speech signals from the fractal codes. This method can effectively reconstruct speech signals at any sampling rates, especially at a higher sampling rate, which is a grand challenge. The proposed method is evaluated the performance by testing with AN4 corpus of CMU Sphinx speech recognition engine. The experimental results show that the proposed method can improve the accuracy of speech recognition, even if the sampling rate of testing speeches differs from that of training speeches.

Speech recognition devices are finding increasing use in industrial environments, and continue to receive interest for military and data processing applications. In the ten year commercial history of word recognition devices, a small but valuable body of literature has developed which can assist the human factors engineer in making best use of this technology. Similarly, an even larger, though less well structured body of informed experience with these devices has also been accumulated. Sufficient data therefore, exists to permit development of a preliminary set of human factors guidelines for the use of speech recognition devices. To be discussed will be: (1) procedures for training word recognition devices; (2) guidelines for vocabulary selection; (3) methods for evaluating the performance of word recognition devices; (4) criteria for selecting applications for word recognition. The emphasis will be on detailed factors which affect recognition accuracy in application environments, e.g., effects of noise, parameters of training and acoustic-phonetic aspects of vocabulary selection.

Recent advances in technology have led to the availability of powerful speech recognizers at low cost and to the possibility of using speech interaction in a variety of new and exciting practical applications. The purpose of this research was to investigate and develop the use of speech recognition in live television subtitling. This paper describes how the "SpeakTitle" project met the challenges of real time speech recognition and live subtitling through the development of a customisable speaker interface and use of 'Topics' for specific subject domains. In the prototype system (described in Hewitt et al., 2000; Bateman et al., 2001) output from the speech recognition system (the IBM ViaVoice R engine) is passed in to a custom-built editor from where it can be corrected and passed on to an existing subtitling system. The system was developed to the extent that it was acceptable for the production of subtitles for live television broadcasts and it has been adopted by three subtitle production facilities in the UK. The evolution of the product and the experiences of users in developing the system in a live subtitling environment are considered, and the system is analysed against industry standards. Ease-of-use and accuracy are also discussed and further research areas are identified.

Public transport vehicles are widely preferred by the mass because of the accessibility it provides. Due to its easy access, crimes like robbery, assaults and even homicides are experienced by the drivers. Hence, vehicle tracking, and alert systems are built to improve its security and safety. The existing systems are limited to physical triggering which offers minimal effectiveness because the buttons may unintentionally be pressed, or the driver is hesitant to move and unable to press the button when necessary. To eliminate the inconvenience caused by a physically triggered security system, a non-contact activation was developed with the use of speech recognition, and the Internet of Things (IoT). This study presents the evaluation of the transcription confidence level associated with background noises using the Google Speech Recognition API and the implementation of the security system in IoT. The results show that speech recognition has acquired 100% transcription accuracy around 50 dBA to 78 dBA background noise using the native language, while the tested operation latency is approximately 43 seconds during the deployment. The study paved a way for a convenient noncontact triggering security system to elevate the rapid response of crime-related incidents in public vehicle drivers through immediate notification and provision of the necessary information to authorities.

Losses of separation (LoS) are breaches of regulations that specify the minimum distance between aircraft in controlled airspace. Erroneous communications between air traffic controllers (ATCs) and pilots are leading contributors to LoS that result in elevated risk of fatal accidents. An air traffic control system that could identify communication errors promptly would, therefore, be advantageous. Establishing such a system requires a systematic characterization of communication errors to reveal how various communication arrangements and errors influence the development of LoS. Such know-how could guide the ATCs and pilots in identifying the parts of their communication processes and content that most influence the occurrence of LoS. Existing studies of LoS focus on simulation of aircraft operation processes with little quantitative analysis about how communication issues arise and result in elevated risks of LoS. This paper presents a method for supporting automatic communication error detection through integrated use of speech recognition, text analysis, and formal modeling of airport operational processes. The proposed method focuses on: identifying communication features to guide the detection of vulnerable communications; characterizing communication errors; and Bayesian Network modeling for predicting communication errors and LoS using the features derived from ATC–pilot communications. Major findings show that incorrect read-backs by pilots are highly correlated with a majority of LoS. Results indicate the proposed method could form a basis for automating communication error detection and preventing LoS. The integrated Automatic Speech Recognition and Natural Language Processing functions may be incorporated into existing aviation applications for real-time ATC–pilot communication monitoring and preventive LoS control.

David L. Weiss, MD Curtis P. Langlotz, MD, PhD The man we now know as St Lawrence lived his life as Lawrence of Rome and served as a deacon in the 3rd century Roman Empire. In 258 AD he was executed during the persecution of Christians by the Emperor Valerian. Historians detail his martyrdom as being grilled alive. Many scholars consider this account well documented while others have cast some doubt on its accuracy. They point out that as a Roman citizen under a sentence of death Lawrence would likely have been afforded the “privilege” of being beheaded. Some believe the discrepancy can be traced to the error of an early scribe. The Latin “passus est” (he died) may have been incorrectly transcribed as “assus est” (he was roasted) (1). That a seemingly trivial transcription error could have such profound historical and ecclesiastic consequences should not be surprising to a radiologist. Our specialty is rife with anecdotes of the trivial typographical error that is perhaps not quite as gruesome, but potentially life-threatening if left uncorrected. Although the first radiology report was generated over a century ago, the process of radiology reporting has changed very little until the past decade. The use of speech-to-text software has been available for some time (2), but speech recognition use did not become widespread until hardware and software matured in the late 1990s. The use of speech recognition in radiology reporting has resulted in improved turnaround time and reported cost savings (3,4), metrics that delight hospital and department administrators. Radiologists, however, have found that the software tends to decrease productivity and has the potential to cause distraction during image interpretation (5,6). In the past several decades, structured reporting systems have become available, first as research systems (7,8), then as production systems, particularly in mammography and ultrasonography (9). As with speech recognition software, current structured reporting systems are more sophisticated and acceptable as a result of advances in hardware, software, and workflow design (10). Will these new systems result in improved workflow or are these reports merely administrative agitprop designed to convince radiologists to adopt one more reporting system that may ultimately have negative consequences for the user? In the sections that follow, we will discuss these issues in a classic dialogue format, alternatively presenting the pros (C.P.L.) and cons (D.L.W) of structured reporting from two radiologists’ points of view.