Modality and Multimodality

Simple SummaryIn a current society marked by closer relationships between humans and their pet companions, most cat owners interact with their feline partners on a daily basis. This study addresses whether, in an extraspecific interaction with humans, cats are sensitive to the communication channel used by their interlocutor. By examining three types of interactions—vocal, visual and bimodal (visual and vocal)—we found the modality of communication had a significant effect on the latency in time taken for cats to approach a human experimenter. Cats interacted significantly faster in response to visual and bimodal communication compared to vocal communication. In addition, cats displayed significantly more tail wagging when the experimenter engaged in no communication (control condition) compared to visual and bimodal communication. Taken together, our results suggest that cats display a marked preference for both visual and bimodal cues addressed by non-familiar humans compared to vocal cues only. Our findings offer further evidence for the emergence of human-compatible socio-cognitive skills in cats that favour their adaptation to a human-driven niche. Across all species, communication implies that an emitter sends signals to a receiver, through one or more channels. Cats can integrate visual and auditory signals sent by humans and modulate their behaviour according to the valence of the emotion perceived. However, the specific patterns and channels governing cat-to-human communication are poorly understood. This study addresses whether, in an extraspecific interaction, cats are sensitive to the communication channel used by their human interlocutor. We examined three types of interactions—vocal, visual, and bimodal—by coding video clips of 12 cats living in cat cafés. In a fourth (control) condition, the human interlocutor refrained from emitting any communication signal. We found that the modality of communication had a significant effect on the latency in the time taken for cats to approach the human experimenter. Cats interacted significantly faster to visual and bimodal communication compared to the “no communication” pattern, as well as to vocal communication. In addition, communication modality had a significant effect on tail-wagging behaviour. Cats displayed significantly more tail wagging when the experimenter engaged in no communication (control condition) compared to visual and bimodal communication modes, indicating that they were less comfortable in this control condition. Cats also displayed more tail wagging in response to vocal communication compared to the bimodal communication. Overall, our data suggest that cats display a marked preference for both visual and bimodal cues addressed by non-familiar humans compared to vocal cues only. Results arising from the present study may serve as a basis for practical recommendations to navigate the codes of human–cat interactions.

Humans critically depend on permanent verbal and nonverbal interaction - for aligning their mental states, for synchronizing their intentions and goals, and also for performing joint tasks, such as carrying a heavy object together, manipulating of objects in a common workspace, or handing over components and building or assembling larger structures in teams. Typically, physical interaction is initiated by a short joint planning dialog and then further accompanied by a stream of verbal utterances. For obtaining a smooth interaction flow in a given situation, humans typically use all their communication modalities and senses, and this often happens even unconsciously. As we move toward the introduction of robotic co-workers that serve humans - some of them will be humanoids; others will be of a different shape - humans will expect them to be integrated into the execution of the task at hand, just as well as if a human co-worker was involved. Such a flawless replacement will only be possible if these robots provide a number of basic action primitives, for example, handover from human to robot and vice versa. The robots must also recognize and anticipate the intention of the human by analyzing and understanding the scene as far as necessary for jointly working on the task. Most importantly, the robotic co-worker must be able to carry on a verbal and nonverbal dialog with the human partner, in parallel with and relating to the physical interaction process. In this chapter, we give an overview of the ingredients of an integrated physical interaction scenario. This includes methods to plan activities, to produce safe and human-interpretable motion, to interact through multimodal communication, to schedule actions for a joint task, and to align and synchronize the interaction by understanding human intentions. We summarize the state of the art in physical human-humanoid interaction systems and conclude by presenting three humanoid systems as case studies.

Teachers must be intentional in using a diverse range of communication modalities (i.e., visual, tactile, and verbal forms) with deafblind learners. Emphasis is placed on the methodology of a group design study that measured the variety of communication modalities used by 15 student–teacher dyads during naturally occurring classroom activities using behavioral coding of videos. Data collection for the study used small action cameras, which is a promising, noninvasive strategy that can be used by teachers to monitor how they are serving as language models to students, as well as to measure growth in students’ communication over time. During data analysis, the importance of teachers matching a learner’s expected expressive communication modality led to the development of the tactile threshold model.

Many studies have shown that apes and monkeys are adept at cross-modal matching tasks requiring the subject to identify objects in one modality when information regarding those objects has been presented in a different modality. However, much less is known about non-human primates' production of multimodal signaling in communicative contexts. Here, we present evidence from a study of 110 chimpanzees demonstrating that they select the modality of communication in accordance with variations in the attentional focus of a human interactant, which is consistent with previous research. In each trial, we presented desirable food to one of two chimpanzees, turning mid-way through the trial from facing one chimpanzee to facing the other chimpanzee, and documented their communicative displays, as the experimenter turned towards or away from the subjects. These chimpanzees varied their signals within a context-appropriate modality, displaying a range of different visual signals when a human experimenter was facing them and a range of different auditory or tactile (attention-getting) signals when the human was facing away from them; this finding extends previous research on multimodal signaling in this species. Thus, in the impoverished circumstances characteristic of captivity, complex signaling tactics are nevertheless exhibited by chimpanzees, suggesting continuity in intersubjective psychological processes in humans and apes.

This article explores evolving posthumanist representations of eros, technology, and the human condition through two body horror films: David Cronenberg’s Videodrome (1983) and Julia Ducournau’s Titane (2021). Popular depictions of eros and technology can lean towards a hypermasculine, obsessive, paranoid reading of the machine, usually resulting in the disintegration of desire and with it the human body. These accounts highlight the contested figure of the “transhuman,” an excessive, technologically enhanced version of the human capable of conquering death. Emerging from postmodern critiques against grand narratives of technological progress and neoliberalist development, the critique of the transhuman underscores the impossibility of a harmonious relationship between the human and the technical. While Videodrome depicts technology as an external force intervening in human body enhancement – destroying the essence of the human and its romanticized “humanity” – Titane speculates about nature/artifice and human/technology binaries, situating technology as inherent to the human condition. This paper traces the move from the transhuman character who seeks to dominate technology and encounters human destruction, to a posthuman who is able to integrate the mediation of technology as a fundamental aspect for thinking its humanness. Drawing on the reappraisal of “originary technicity,” the body in Titane becomes with technics and environment, offering a new framework for rethinking desire and eros beyond the body. In doing so, it resists earlier theories of desire as contained, and perhaps, repressed, in the subject. Ultimately, the film embodies the possibility of moving away from the all-too human theories of eros and machine to challenge some posthumanist limitations around essentialist oppositions between nature and technology. Posthumanist frameworks must envision a future where the entanglement of technology and the human serves as a starting point for rethinking the human condition, rather than a nostalgic excess it needs to overcome.

The present study examined differences in modality use during episodes of joint attention between hearing parent-hearing child dyads and hearing parent-deaf child dyads. Hearing children were age-matched to deaf children. Dyads were video recorded in a free play session with analyses focused on uni- and multimodality use during joint attention episodes. Results revealed that adults in hearing parent-deaf child dyads spent a significantly greater proportion of time interacting with their children using multiple communicative modalities than adults in hearing parent-hearing child dyads, who tended to use the auditory modality (e.g., oral language) most often. While these findings demonstrate that hearing parents accommodate their children’s hearing status, we observed greater overall time spent in joint attention in hearing parent-hearing child dyads than hearing parent-deaf child dyads. Our results point to important avenues for future research on how parents can better accommodate their child’s hearing status through the use of multimodal communication strategies.

Multimodal Communication Treatment with discourse via telepractice (teleMCT+D) is a modified version of Multimodal Communication Treatment for people with aphasia to expand instruction in resolving communication breakdowns to the discourse level. Due to potential barriers to telepractice with nonspoken communication modalities, we aimed to evaluate its feasibility in a single-subject design study. Two people with aphasia completed teleMCT+D across 10 weeks (two 1-hr sessions per week) and posttreatment interviews. The feasibility was evaluated for the five categories (recruitment, data collection, acceptability, resources, and responses) from Orsmond and Cohn (2015). Each category of feasibility was evaluated by several outcome measures. Findings demonstrated high feasibility for recruitment and data collection, although one participant's assessment fidelity was moderate. Participants found teleMCT+D to be acceptable. They viewed the telepractice format positively; however, minimal negative reactions and events were also reported. Regarding resources, all participants and care partners had the technology skills and materials needed to participate. Additional resources, including clinician training, supervision, and regular team meetings, assured ethical conduct. Finally, the treatment response results showed mixed effects on accuracy and nonspoken modality use but no improvement in spoken modality productions. Participants also reported that they used nonspoken modalities (e.g., gesturing) frequently for daily conversations. We discussed the advantages and challenges of teleMCT+D based on the feasibility results such as limited view and control of the environment, which limited nonspoken modality training at times. Future directions include following the suggested modifications in a large sample with various types and severities of aphasia. Preliminary evidence supporting that a telepractice format for communication breakdown resolution treatment for people with aphasia is feasible and acceptable can provide better access to underserved populations.

The digital transformation of our health care system will require not only digitization of existing tools but also a redesign of our care delivery system and collaboration with digital partners. Traditional patient journeys are reactive to symptom presentation and delayed by health care system-centric scheduling, leading to poor experience and avoidable adverse outcomes. Patient journeys will be reimagined to a digital health pathway that seamlessly integrates various care experiences from telemedicine, remote monitoring, to in-person clinic visits. Through centering the care delivery around the patients, they can have more delightful experiences and enjoy the quality of standardized condition pathways and outcomes. To design and implement digital health pathways at scale, enterprise health care systems need to develop capabilities and partnerships in human-centered design, operational workflow, clinical content management, communication channels and mechanisms, reporting and analytics, standards-based integration, security and data management, and scalability. Using a human-centered design methodology, care pathways will be built upon an understanding of the unmet needs of the patients to have a more enjoyable experience of care with improved clinical outcomes. To power this digital care pathway, enterprises will choose to build or partner for clinical content management to operationalize up-to-date, best-in-class pathways. With this clinical engine, this digital solution will engage with patients through multimodal communication modalities, including written, audio, photo, or video, throughout the patient journey. Leadership teams will review reporting and analytics functions to track that the digital care pathways will be iterated to improve patient experience, clinical metrics, and operational efficiency. On the backend, standards-based integration will allow this system to be built in conjunction with the electronic medical record and other data systems to provide safe and efficient use of the digital care solution. For protecting patient information and compliance, a security and data management strategy is critical to derisking breeches and preserving privacy. Finally, a framework of technical scalability will allow digital care pathways to proliferate throughout the enterprise and support the entire patient population. This framework empowers enterprise health care systems to avoid collecting a fragmented series of one-off solutions but develop a sustainable concerted roadmap to the future of proactive intelligent patient care.

We live in a world where it is impossible to exist without, and beyond, technologies. Despite this omnipresence, we tend to overlook their influence on us. The vigorously developing approach of postphenomenology, combining insights from phenomenology and pragmatism, focuses on the so-called technological mediation, i.e., on how technologies as mediators of human-world relations influence the appearing of both the world and the human beings in it. My analysis aims at demonstrating both the methodological weaknesses and open possibilities of postphenomenology. After summarizing its essentials, I will scrutinize, first, its ability to turn to the technological things themselves and, second, the so-called empirical turn as realized by postphenomenology. By assessing its conceptual framework from the phenomenological perspective, I hope to demonstrate that postphenomenology needs philosophical clarification and strengthening. In short, it needs a more phenomenological, and less pragmatic, approach to technology in its influence on human experience.