Introducing Deep Mind Learning Modeling

Speech–driven virtual assistants, known as smart speakers, such as Amazon Echo and Google Home, are increasingly used. However, commercial smart speakers only support a handful of languages. Even languages for which ASR and TTS technology is available, such as many official EU member state languages, are not supported due to a commercial disinterest derived from their – relatively speaking – rather small number of speakers. This problem is even more crucial for minority languages, for which smart speakers are not expected anytime soon, or ever. In this ELG pilot project we developed a Basque–speaking smart speaker, making use of the open source smart speaker project Mycroft AI and Elhuyar Foundation’s speech technologies for Basque. Apart from getting it to speak Basque, one of our goals was to make the smart speaker privacy friendly, non–gendered and use local services, because these are usual issues of concern. The project has also served to improve the state of the art of Basque ASR and TTS technology.

Smart speakers (such as Amazon Echo and Google Home) are becoming a popular addition to homes, and it is expected that they will be used as parental support in the future. Previous studies have analyzed the history logs of the uses of their services and gathered impressions of them using questionnaire surveys and qualitative interviews. Details of how children interact with smart speakers have not been fully investigated. This study investigated children's interactions with an utterance-output device, such as a smart speaker, and focuses on the influences of the children's personality traits. The experiment was conducted in a realistic setting. The results indicate that the less nervous, more emotionally stable, or more adaptable to communication different from that at home children are, the more closely they engage in conversational play with an utterance-output device. The findings suggest that effective conversational play, which can lessen anxiety toward a novel utterance-output device, will be required in future interaction design.

Smart speakers with AI assistant, such as Amazon Echo and Google Home, and compatible IoT devices are emerging. The authors have developed an iHAC Hub that realizes coordinated operation of IoT devices that can be controlled by communication protocols. This paper proposes a cooperation method for M2M devices by using iHAC Hub and a smart speaker. In the proposed method, we provide a framework for cooperating environment sensing IoT devices and smart speaker compatible IoT devices. The iHAC Hub generates voice commands for controlling IoT devices by using the cloud service and reproduces them toward the smart speaker to realize cooperative control of the smart speaker compatible IoT device based on the environment information.

Successful communication is vital to quality of life. One group commonly facing speech and communication difficulties is individuals with intellectual disability (ID). A novel route to encourage clear speech is offered by mainstream smart speakers (e.g., Amazon Alexa and Google Home). Smart speakers offer four factors important for learning: reward immediacy, spaced practice, autonomy/intrinsic motivation and reduced social barriers. Yet the potential of smart speakers to improve speech intelligibility has not been explored before. To determine whether providing individuals with intellectual disabilities with smart speaker devices improved ratings of speech intelligibility for (1) phrases related to device use and (2) unrelated words via a semi-randomized controlled trial. In a semi-randomized controlled trial, an intervention group of adults with ID (N = 21) received smart speakers, while a control group (N = 22) did not. Before and after about 12 weeks, participants were recorded saying smart speaker-related phrases and unrelated words. Naïve participants then rated the intelligibility of the speech recordings. The group that received smart speakers made significantly larger intelligibility gains than the control group. Although the effect size was modest, this difference was found for both smart speaker-related phrases and unrelated words. While the mechanism of action remains to be determined, the presence of smart speakers in the home had a demonstrable impact on ratings of speech intelligibility, and could provide cost-effective inclusive support for speech and communication improvement, improving the quality of life of vulnerable populations. What this paper adds What is already known on the subject Speech intelligibility is a key obstacle for social relationships and quality of life across several vulnerable populations (children with speech difficulties, older adults with dementia, individuals with ID). Anecdotal reports suggest mainstream smart speakers (e.g., Amazon Alexa, Google Home), could improve speech intelligibility. What this paper adds to existing knowledge We used a semi-randomized controlled trial to show that using a smart speaker for about 12 weeks could improve ratings of speech intelligibility in adults with ID for both smart speaker-related phrases and unrelated words. What are the potential or actual clinical implications of this work? These initial findings suggest that smart speaker technology could be a novel, and inclusive, route to improving speech intelligibility in vulnerable populations.

The popularity of smart speakers with virtual personal assistants such as Amazon Alexa and Google Assistant has been growing. However, the security and privacy of users using these devices have not been thoroughly examined which can lead to severe security and privacy concerns. In this poster, we present a fingerprinting attack of skills on Amazon Echo by analyzing the encrypted network traffic and show the potential privacy leakage of users using skills on smart speakers. Skills are voice applications that can be accessed through Amazon Alexa. Skills increase the diversity of using Amazon smart speakers by providing a variety of applications that can be used from a single platform with the use of voice commands without accessing a phone or a computer. In this fingerprinting attack, an adversary can eavesdrop on the network traffic of the smart speaker and predict the skill that is used by the user without decrypting the encrypted network traffic. For this attack, we collect the dataset consisting of 10,000 traces of encrypted network traffic related to 100 popular skills. We perform feature selection to investigate the top features that contribute to privacy leakage. We implement the attack by leveraging multiple machine learning algorithms. Among the different machine learning algorithms we implement, Random Forest Classifier performs the best by achieving an accuracy of 68.32% (compared to a random guess of 1%). The experimental results show that there is a need for developers to focus on the privacy leakages of users.

BackgroundWith one in five individuals aged 65 or older living with type 2 diabetes worldwide, it is crucial to acknowledge and address the challenges faced by this population. In this context, our study aims to evaluate the efficacy of a behavioral intervention model delivered through a smart speaker on mental health and diabetes self-care in the elderly with diabetes.MethodsThis is a single-center, pragmatic, parallel two-arm open randomized clinical trial involving elderly patients with type 2 diabetes. We plan to enroll a total of 112 individuals who will be randomized 1:1 to receive the Smart Speaker EchoDot 3rd Gen device (Amazon Echo®) for home use (intervention arm) or to maintain usual care (control arm). The primary outcome is mental distress, assessed using the 20-item Self Reporting Questionnaire (SRQ-20) after a 12-week intervention period. Secondary outcomes include quality of life, adherence to diabetes self-care behaviors, perception of stress, glycemic control, blood pressure, and lipid profile. Analysis of covariance (ANCOVA) will be used to evaluate the effects of the intervention on the outcomes.DiscussionThis study assesses the effectiveness of an interactive virtual assistance system for enhancing mental health and glycemic control among elderly individuals with type 2 diabetes. The findings may introduce smart speakers as a valuable tool for promoting diabetes-related self-care in this population.Trial registrationClinicalTrials.gov NCT05329376. Registered on 15 April 2022. Enrollment began on 20 June 2023 and the last update of protocol was on 13 December 2023.

Voice user interface (VUI) plays an essential role in intelligent scenes, e.g., smart homes. It provides a hands- and eyes-free human-machine interaction between humans and Internet of Things devices. Benefiting from the development of deep learning and natural language process, the automatic speech recognition (ASR) entitles VUI to the capacity of accurate comprehension on users' intentions. With such a convenient and flexible service, users can interact with various devices as they please. Commercial VUI products have gained in popularity over recent years, such as smart speakers (e.g., Amazon Echo and Google Home), voice assistants in smartphones (e.g., Siri), and in-vehicle voice control interactions (e.g., VUIs in Tesla Model S/X/3/Y). Analysts forecast that by 2024, the deployment of VUI-based smart speakers will reach 640 million globally.

The possibility of personalizing devices and online services is important for end users living in smart environments, but existing End-User Development interfaces in this field often fail to provide users with the proper support, e.g., because they force users to deal with too many technological details. This paper explores novel approaches for personalizing IoT ecosystems via natural language and vocal interaction. We first conducted seven interviews to understand whether and how end users would converse with a conversational assistant to personalize their IoT ecosystems. Then, we designed and implemented two prototypes to define trigger-action rules through vocal and multimodal approaches. A usability study with 10 participants confirms the feasibility and effectiveness of personalizing the IoT via voice and opens the way to integrate personalization capabilities in smart speakers like Google Home and Amazon Echo.

Voice assistants embodied in smart speakers (e.g., Amazon Echo, Google Home) enable voice-based interaction that does not necessarily rely on expertise with mobile or desktop computing. Hence, these voice assistants offer new opportunities to different populations, including individuals who are not interested or able to use traditional computing devices such as computers and smartphones. To understand how older adults who use technology infrequently perceive and use these voice assistants, we conducted a 3-week field deployment of the Amazon Echo Dot in the homes of seven older adults. While some types of usage dropped over the 3-week period (e.g., playing music), we observed consistent usage for finding online information. Given that much of this information was health-related, this finding emphasizes the need to revisit concerns about credibility of information with this new interaction medium. Although features to support memory (e.g., setting timers, reminders) were initially perceived as useful, the actual usage was unexpectedly low due to reliability concerns. We discuss how these findings apply to other user groups along with design implications and recommendations for future work on voice-user interfaces.

The modern world does not stand still. We used to be surprised that technology could speak, but now voice assistants have become real family members. They do not simply turn on the alarm clock or play music. They communicate with children, help solving problems, and sometimes even take offense. Since all voice assistants have artificial intelligence, when communicating with the user, they take into account the change in their location, time of day and days of the week, search query history, previous orders in the online store, etc. However, voice assistants, which are part of modern smartphones or smart speakers, pose a threat to their owner’s personal data since their main function is to capture audio commands from the user. Generally, AI smart speakers such as Siri, Google Assistance, Google Home, and so on are moderately harmless. As voice assistants become versatile, like any other product, they can be used for the most nefarious purposes. There are many common attacks that people with bad intentions can use to hack our voice assistant. We show in our experience that a laser beam can control Google Assistance, smart speakers, and Siri. The attacker does not need to make physical contact with the victim’s equipment or interact with the victim; since the attacker’s laser can hit the smart speaker, it can send commands. In our experiments, we achieve a successful attack that allows us to transmit invisible commands by aiming lasers up to 87 m into the microphone. We have discovered the possibility of attacking Android and Siri devices using the built-in voice assistant module through the charging port.

As smart speakers such as the Amazon Echo have grown in popularity, these devices have presented new opportunities for exploring voice interaction, particularly in the domain of exercise. We have created TandemTrack, a multimodal system comprised of a mobile app and an Alexa skill whose capabilities include exercise guidance, data capture, feedback, and reminder. We propose using TandemTrack in a deployment study to explore and analyze individuals' preferences and use of voice and visual interactions, and to test TandemTrack's effectiveness as a whole in promoting consistent exercise habits while additionally comparing the effectiveness of the app against the skill. Not only does TandemTrack make important steps in addressing gaps in currently available exercise assistants, but also, more importantly, we believe this is the first study to investigate the specific value that voice interactions may provide over other technologies in promoting consistent exercise.

This paper proposes a neural network-based method for generating compact answers to open-domain why-questions (e.g., "Why was Mr. Trump elected as the president of the US?"). Unlike factoid question answering methods that provide short text spans as answers, existing work for why-question answering have aimed at answering questions by retrieving relatively long text passages, each of which often consists of several sentences, from a text archive. While the actual answer to a why-question may be expressed over several consecutive sentences, these often contain redundant and/or unrelated parts. Such answers would not be suitable for spoken dialog systems and smart speakers such as Amazon Echo, which receive much attention in these days. In this work, we aim at generating non-redundant compact answers to why-questions from answer passages retrieved from a very large web data corpora (4 billion web pages) by an already existing open-domain why-question answering system, using a novel neural network obtained by extending existing summarization methods. We also automatically generate training data using a large number of causal relations automatically extracted from 4 billion web pages by an existing supervised causality recognizer. The data is used to train our neural network, together with manually created training data. Through a series of experiments, we show that both our novel neural network and auto-generated training data improve the quality of the generated answers both in ROUGE score and in a subjective evaluation.

Since the introduction of the Amazon Echo in late 2014, smart speakers have been adopted by a wide range of households. As an “audio-first” technology, their materiality, interactivity, and ubiquity change what we mean by radioness—remediating both radio and audio streaming. This chapter investigates the radioness of smart speakers through two complementary lenses. First, it examines their impact on the music, streaming audio, and radio by analyzing media coverage of smart speakers’ adoption and impact, including market research organizations, mainstream news outlets, and the trade press. Second, it examines smart speakers as domestic artifacts, drawing on autoethnographic strategies, design thinking, and scholarship on domesticity. The domestic comfort and technological ease that smart speakers promise are animated by their nostalgic, radio-like status, but it is also their radioness that helps obscure their imbrication with the logic of contemporary big tech.

In "smart speaker'' digital assistant systems such as Google Home, there is no visual user interface, so people must learn about the system's capabilities and limitations by experimenting with different questions and commands. However, many new users give up quickly and limit their use to a few simple tasks. This is a problem for both the user and the system. Users who stop trying out new things cannot learn about new features and functionality, and the system receives less data upon which to base future improvements. Symbiosis---a mutually beneficial relationship---between AI systems like digital assistants and people is an important aspect of developing systems that are partners to humans and not just tools. In order to better understand requirements for symbiosis, we investigated the relationship between the types of digital assistant responses and users' subsequent questions, focusing on identifying interactions that were discouraging to users when speaking with a digital assistant. We conducted a user study with 20 participants who completed a series of information seeking tasks using the Google Home, and analyzed transcripts using a method based on applied conversation analysis. We found that the most common response from the Google Home, a version of "Sorry, I'm not sure how to help'', provided no feedback for participants to build on when forming their next question. However, responses that provided somewhat strange but tangentially related answers were actually more helpful for conversational grounding, which extended the interaction. We discuss the connection between grounding and symbiosis, and present recommendations for requirements for forming partnerships with digital assistants.

Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce device costs and interactive friction. This transition has been hastened by the increasing capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a hearing speech 18% WER. Our findings show that current speech-controlled interfaces are not usable by deaf and hard of hearing people. Therefore, it might be wise to pursue other methods for deaf persons to deliver natural commands to computers.