One Step BeyondMusical Expertise and Word Learning

The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.

Many speech and music analysis and processing schemes rely on an estimate of the fundamental frequency f0 of periodic signal components. Most established schemes apply rather unspecific signal models such as sinusoidal models to the estimation problem, which may limit time resolution and estimation accuracy. This study proposes a novel time-domain locked-loop algorithm with low computational effort and low memory footprint for f0 estimation. The loop control signal is directly derived from the input time signal, using a harmonic signal model. Theoretically, this allows for a noise-robust and rapid f0 estimation for periodic signals of arbitrary waveform, and without the requirement of a prior frequency analysis. Several simulations with short signals employing different types of periodicity and with added wide-band noise were performed to demonstrate and evaluate the basic properties of the proposed algorithm. Depending on the Signal-to-Noise Ratio (SNR), the estimator was found to converge within 3–4 signal repetitions, even at SNR close to or below 0 dB. Furthermore, it was found to follow fundamental frequency sweeps with a delay of less than one period and to track all tones of a three-tone musical chord signal simultaneously. Quasi-periodic sounds with shifted harmonics as well as signals with stochastic periodicity were robustly tracked. Mean and standard deviation of the estimation error, i.e., the difference between true and estimated f0, were at or below 1 Hz in most cases. The results suggest that the proposed algorithm may be applicable to low-delay speech and music analysis and processing.

Music is created in the listener as it is perceived and interpreted - its meaning derived from our unique sense of it; likely driving the range of interpersonal differences found in music processing. Person-specific mental representations of music are thought to unfold on multiple levels as we listen, spanning from an entire piece of music to regularities detected across notes. As we track incoming auditory information, predictions are generated at different levels for different musical aspects, leading to specific percepts and behavioral outputs, illustrating a tight coupling of cognition, perception and action. This coupling, together with a prominent role of prediction in music processing, fits well with recently described ideas about the role of predictive processing in cognitive function, which appears to be especially suitable to account for the role of mental models in musical perception and action. Investigating the cerebral correlates of constructive music imagination offers an experimentally tractable approach to clarifying how mental models of music are represented in the brain. I suggest here that mental representations underlying imagery are multimodal, informed and modulated by the body and its in- and outputs, while perception and action are informed and modulated by predictions based on mental models.

Spectral shape affects pitch perception; sounds with more high energy harmonics sound higher than sounds with low energy in higher harmonics. Flatter spectral slope corresponds to "tenser" voices while steeper spectral slope correlates to "breathier" voices in speech. In string instruments, the spectral slope differentiates sul ponticello and sul tasto. Listeners were found to integrate spectral slope cues in pitch perception in speech; however, work on music focused on cross-instrument differences, glossing over cue integration within instruments with fixed-formant frequencies. Furthermore, spectral cues and F0 co-vary in human pitch production, but are largely independent in instrumental music. It remains unclear whether music processing is as integrative as speech processing. In this study, listeners were given either speech or violin stimuli with identical pitch contour pairs, and were asked to decide whether the second contour was higher or lower in pitch compared to the first. The spectral slope of each sound was manipulated to include all combinations of “breathier”/“sul tasto” and “tenser”/“sul ponticello” sounding pairs. Results show that listeners integrate spectral slope cues in pitch perception in speech and violin stimuli similarly, with similar categoricity and shift. Overall, listeners with higher musicality have more categorical responses but no differences in shift.

Event Abstract Back to Event Evidence for language-specificity of a BCI-based language training Mariacristina Musso1*, David Hübner2, Sarah Schwarzkopf1, Cornelius Weiller1 and Michael Tangermann2 1 Universitätsklinikum Freiburg, Neurology, Germany 2 Brain State Decoding Lab, Germany Introduction: Language therapy for aphasia following stroke generally focuses on patients’ specific language deficits and is guided and controlled by an external person, e.g. by the speech therapist or a trained volunteer [1]. In contrast, closed-loop approaches based on brain-computer interfaces (BCI) have been used successfully for motor impairments after stroke [2] and are guided and controlled by the patients themselves. This study aimed at investigating if a BCI-based approach can also be designed for language training to patients with aphasia. For the first time, we developed and implemented a BCI-based training protocol for chronic aphasia patients after stroke. After attending word stimuli, patients receive feedback based on the strength of task-relevant EEG signals. Feasibility of the protocol was previously evaluated in an offline study with 20 normally-aged controls. Material and Methods: Eight patients with an infarct in the left middle cerebral artery and chronic aphasia underwent about 30 hours of effective BCI-supported online training. During each training session, patients were seated in a ring of 6 loudspeakers (AMUSE paradigm, [3]) and brain signals were measured by EEG (32 channel passive Ag/AgCl electrodes). Patients heard a cueing sentence followed by a series of 6 bisyllabic words (concrete nouns) from which one correctly finished the sentence. After each trial, they received a feedback on whether the attended word predicted by event-related potentials (ERP) responses matched the target word. Before and after the training, all patients underwent an aphasia test battery for language assessment (Aachener Aphasie Test (AAT) [4] and Snodgrass naming test) as well as tests for executive functions (TAP, Corsi, digit span and word-fluency test). Two EEG-sessions (64 EEG channels) without feedback were conducted prior to the training to calibrate the BCI system and to determine parameters of the training protocol (e.g. the stimulation onset asynchrony between two words) for each patient. In addition, one session without feedback was recorded after the training to compare ERP responses. Results: The AAT showed a significant improvement of several subtests such as naming, writing, comprehension and the token subtest. Similarly, the naming performance in the Snodgrass & Vanderwart test also significantly increased. TAP, Corsi, digit span and word-fluency revealed deficits in executive functions in each patient prior to the training. Contrary to the language-related tests, we did not observe any significant improvements in these neuropsychological categories. Further evidence for the language-specificity of the training can be derived from the ERP analysis where word ERP responses showed an increase in amplitude of the P300 and N200 component and a decrease in P300 onset latency whereas the ERP results to tone stimuli did not show significant training-induced changes. Discussion & Conclusion: Our preliminary results showed that our BCI-supported aphasia training was very effective for the recovery of language deficits with an effect that generalizes to language aspects that had not been trained directly (such as writing or reading). This novel therapeutic approach did not train executive functions, but more directly distinct aspects of language-processing. Acknowledgements Acknowledgements: This work was (partly) supported by BrainLinks-BrainTools, Cluster of Excellence funded by the German Research Foundation (DFG), grant number EXC 1086 and by Wissenschaftliche Gesellschaft Freiburg im Breisgau. References References [1] M.C. Brady, H. Kelly, J. Godwin, P. Enderby, P. Campbell, Cochrane Database Syst. Rev. (2016) CD000425. [2] A. Remsik, B. Young, R. Vermilyea, L. Kiekhoefer, J. Abrams, S. Evander Elmore, P. Schultz, V. Nair, D. Edwards, J. Williams, V. Prabhakaran, Expert Rev. Med. Devices 13 (2016) 445–454. [3] M. Schreuder, B. Blankertz, M. Tangermann, PLoS One 5 (2010) e9813. [4] W. Huber, Aachener Aphasie-Test:(AAT), Hogrefe, Verlag für Psychologie, 1983. Keywords: Speech Therapy, Speech Processing, music processing, Executive Function, Stroke Conference: Academy of Aphasia 56th Annual Meeting, Montreal, Canada, 21 Oct - 23 Oct, 2018. Presentation Type: oral presentation Topic: not eligible for a student prize Citation: Musso M, Hübner D, Schwarzkopf S, Weiller C and Tangermann M (2019). Evidence for language-specificity of a BCI-based language training. Conference Abstract: Academy of Aphasia 56th Annual Meeting. doi: 10.3389/conf.fnhum.2018.228.00069 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 05 Apr 2018; Published Online: 22 Jan 2019. * Correspondence: Dr. Mariacristina Musso, Universitätsklinikum Freiburg, Neurology, Freiburg, Germany, mariachristina.musso@uniklinik-freiburg.de Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Mariacristina Musso David Hübner Sarah Schwarzkopf Cornelius Weiller Michael Tangermann Google Mariacristina Musso David Hübner Sarah Schwarzkopf Cornelius Weiller Michael Tangermann Google Scholar Mariacristina Musso David Hübner Sarah Schwarzkopf Cornelius Weiller Michael Tangermann PubMed Mariacristina Musso David Hübner Sarah Schwarzkopf Cornelius Weiller Michael Tangermann Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

The perception of rapidly changing verbal and nonverbal auditory patterns is a fundamental prerequisite for speech and music processing. Previously, the left planum temporale (PT) has been consistently shown to support the discrimination of fast changing verbal and nonverbal sounds. Furthermore, it has been repeatedly shown that the functional and structural architecture of this supratemporal brain region differs as a function of musical training. In the present study, we used the functional magnetic resonance imaging technique, in a sample of professional musicians and nonmusicians, in order to examine the functional contribution of the left PT to the categorization of consonant-vowel syllables and their reduced-spectrum analogues. In line with our hypothesis, the musicians showed enhanced brain responses in the left PT and superior discrimination abilities in the reduced-spectrum condition. Moreover, we found a positive correlation between the responsiveness of the left PT and the performance in the reduced-spectrum condition across all subjects irrespective of musical expertise. These results have implications for our understanding of musical expertise in relation to segmental speech processing.

A growing body of research has suggested that cognitive abilities may play a role in individual differences in speech processing. The present study took advantage of a widespread linguistic phenomenon of sound change to systematically assess the relationships between speech processing and various components of attention and working memory in the auditory and visual modalities among typically developed Cantonese-speaking individuals. The individual variations in speech processing are captured in an ongoing sound change-tone merging in Hong Kong Cantonese, in which typically developed native speakers are reported to lose the distinctions between some tonal contrasts in perception and/or production. Three groups of participants were recruited, with a first group of good perception and production, a second group of good perception but poor production, and a third group of good production but poor perception. Our findings revealed that modality-independent abilities of attentional switching/control and working memory might contribute to individual differences in patterns of speech perception and production as well as discrimination latencies among typically developed speakers. The findings not only have the potential to generalize to speech processing in other languages, but also broaden our understanding of the omnipresent phenomenon of language change in all languages.

Background Over the past decade, the field of music neuroscience has surged in its examination of the efficacy of music rehabilitation. A growing area of interest within this surge has been research on music interventions for post-stroke aphasia. Brain activation during music processing and perception may overlap with activity during speech and language processing. This overlap suggests possible improvement of speech and language through activation of the multimodal music system. However, the role of music has not been investigated from the perspective of existing literature on skilled language treatment to identify gaps in evidence and outcomes. Aims We surveyed the current literature regarding the use of music-based methods for skilled aphasia rehabilitation to answer the question, “What research has been conducted on music-based, speech-language interventions for post-stroke aphasia?” Methods & procedures We conducted a systematic database search of PubMed, EBSCO Discovery Search, CINAHL Complete, CINAHL Plus with Full Text, MEDLINE Complete, MEDLINE, APA PsycArticles, and APA PsycInfo using the following search terms: (music) AND (aphasia) AND (treatment OR rehabilitation OR intervention OR therapy OR management OR approach) AND (music OR choir OR choral OR melodic OR melody OR sing OR singing OR song) AND (aphasia OR language OR dysphagia OR cognitive-communication). Outcome & results Twenty-seven studies met our inclusion criteria after full-text review. The primary use of music in skilled rehabilitative aphasia therapy and focus of music neurorehabilitation research for aphasia is melodic intonation therapy (MIT), and most studies that fell within the search parameters discussed the use, mechanisms, and adaptation of MIT within the aphasia rehabilitation space. Our review highlighted considerable inconsistency in both the administration of MIT and the reporting of characteristics of MIT used in each study. Sample sizes of included studies ranged from one to ten participants, and most studies were published from 2010 to the present. Conclusions The rise in music neurorehabilitation literature occurred without foundational research of music as an aphasia intervention to provide uniformity or example. Lack of standardization, variability of results, and inconsistency of reporting lessened the quality of evidence found in the included studies. There is a need in this field for standardization of practice and reporting, even in the exploration of new clinical applications of music interventions.

The precise mechanisms underlying tinnitus perception and distress are still not fully understood. A recent proposition is that auditory prediction errors and related memory representations may play a role in driving tinnitus perception. It is of interest to further explore this. To obtain a comprehensive narrative synthesis of current research in relation to auditory prediction and its potential role in tinnitus perception and severity. A narrative review methodological framework was followed. The key words Prediction Auditory, Memory Prediction Auditory, Tinnitus AND Memory, Tinnitus AND Prediction in Article Title, Abstract, and Keywords were extensively searched on four databases: PubMed, Scopus, SpringerLink, and PsychINFO. All study types were selected from 2000-2016 (end of 2016) and had the following exclusion criteria applied: minimum age of participants <18, nonhuman participants, and article not available in English. Reference lists of articles were reviewed to identify any further relevant studies. Articles were short listed based on title relevance. After reading the abstracts and with consensus made between coauthors, a total of 114 studies were selected for charting data. The hierarchical predictive coding model based on the Bayesian brain hypothesis, attentional modulation and top-down feedback serves as the fundamental framework in current literature for how auditory prediction may occur. Predictions are integral to speech and music processing, as well as in sequential processing and identification of auditory objects during auditory streaming. Although deviant responses are observable from middle latency time ranges, the mismatch negativity (MMN) waveform is the most commonly studied electrophysiological index of auditory irregularity detection. However, limitations may apply when interpreting findings because of the debatable origin of the MMN and its restricted ability to model real-life, more complex auditory phenomenon. Cortical oscillatory band activity may act as neurophysiological substrates for auditory prediction. Tinnitus has been modeled as an auditory object which may demonstrate incomplete processing during auditory scene analysis resulting in tinnitus salience and therefore difficulty in habituation. Within the electrophysiological domain, there is currently mixed evidence regarding oscillatory band changes in tinnitus. There are theoretical proposals for a relationship between prediction error and tinnitus but few published empirical studies.

Sounds of hammering or clapping can evoke simulation of the arm movements that have been previously associated with those sounds. This audio-motor transformation also occurs at the sequential level and plays a role in speech and music processing. The present study aimed to demonstrate how the activation pattern of the sensorimotor network was modulated by the sequential nature of the auditory input and effector. Fifteen skilled drum set players participated in our functional magnetic resonance imaging study. Prior to the scan, these drummers practiced six drumming grooves. During the scan, there were four rehearsal conditions: covertly playing the drum set under the guidance of its randomly-presented isolated stroke sounds, covertly playing the drum set along with the sounds of learned percussion music, covertly reciting the syllable representation along with this music, and covertly reciting along with the syllable representation of this music. We found greater activity in the bilateral posterior middle temporal gyri for active listening to isolated drum strokes than for active listening to learned drum music. These regions might mediate the one-to-one mappings from sounds to limb movements. Compared with subvocal rehearsals along with learned drum music, covert rehearsals of limb movements along with the same music additionally activated a lateral subregion of the left posterior planum temporale. Our results illustrate a functional specialization of the posterior temporal lobes for audio-motor processing.

Subjects who had undergone unilateral temporal-lobe excisions for the relief of epilepsy were tested in order to study the relation between musical processing and cerebral function. In one study it was found that only right temporal-lobe lesions including Heschl's gyri (the primary auditory cortex) resulted in a significant deficit in judging pitch changes when the fundamental of a complex tone was missing. In melodic discrimination tasks right-temporal lobectomy had a deleterious effect, but the excisions of Heschl's gyri from either side resulted in a more marked deficit. Other experiments involving identification of familiar distorted tunes and recognition of lyrics of songs primarily show effects of left temporal-lobe lesions. However, a case of absolute pitch ability showed no disturbance from a left temporal-lobe lesion. The results are discussed in terms of the possible contribution of different cortical regions to specific aspects of musical processing.

Musically Speaking

Source transcription of pitched polyphonic music entails providing the pitch (F0) values corresponding to each source in a separate channel. This problem is an important step towards many important problems in music and speech processing. It involves 1) estimating the multiple F0 values in each short time frame, and 2) clustering the F0 values into streams corresponding to different sources. We address the problem in an unsupervised way, with only the total number of sources given beforehand. The framework of probabilistic latent component analysis (PLCA) is used to decompose the polyphonic short-time magnitude spectra for multiple F0 estimation and source-specific feature extraction. It is further embedded into the structure of hidden Markov random fields (HMRF) for clustering the F0s into different sources. This clustering is constrained by the cognitive grouping of continuous F0 contours as well as segregation of simultaneous F0s into different source streams. Such constraints are effectively and elegantly modeled by the HMRF's. Simulated annealing varies the degree of constraints for better clustering. The paper also proposes a novel strategy using the trade-off between precision and recall of multiple F0 estimation for better clustering. Evaluations over a variety of datasets show the efficacy of the proposed algorithm and its robustness to the presence of spurious F0s while clustering. It also outperforms a state-of-the-art unsupervised source streaming algorithm in a set of comparative experiments.

The Workshop on Speech, Language and Audio in Multimedia (SLAM) positions itself at at the crossroad of multiple scientific fields (music and audio processing, speech processing, natural language processing and multimedia) to discuss and stimulate research results, projects, datasets and benchmarks initiatives where audio, speech and language are applied to multimedia data. While the first two editions were collocated with major speech events, SLAM'15 is deeply rooted in the multimedia community, opening up to computer vision and multimodal fusion. To this end, the workshop emphasizes video hyperlinking as an showcase where computer vision meets speech and language. Such techniques provide a powerful illustration of how multimedia technologies incorporating speech, language and audio can make multimedia content collections better accessible, and thereby more useful, to users.

BackgroundGiven the close relationship between hearing and cognitive function in older age, characterization of speech processing deficits along the auditory pathway is of great importance for the brain at risk of dementia. In the ascending, primary auditory pathway, signals pass through subcortical relay stations in the brainstem and midbrain before being integrated into cortical areas. A recent study suggests that the pathophysiology of MCI extends to speech encoding in the brainstem and that both cortical and subcortical markers of speech processing have predictive potential for putative Mild Cognitive Impairment (MCI). In this project, we are testing these neural speech processing markers to see if they can distinguish healthy older adults with age‐related hearing loss from those at risk for dementia.MethodsElectroencephalography (EEG, 32 channels, sampling rate = 16 384 Hz) will be recorded from N = 40 older participants (age ≥ 60 years, retired) while listening to excerpts from an audiobook. We use a novel EEG paradigm that allows us to simultaneously measure cortical and subcortical responses to natural running speech. Based on their Montreal Cognitive Assessment (MoCA) score, participants are divided into an experimental group with “putative MCI” (MoCA &lt; 26) and a control group (MoCA ≥ 26). In addition, we perform audiometric testing with pure tones (PTA) and speech‐in‐noise (SiN) tests to assess participants' hearing function.ResultsWe are currently in the middle of data collection and will be able to present preliminary results at AAIC. We expect to see altered speech processing in the experimental group, particularly driven by slower and weaker encoding at the subcortical level. In a next step, we aim to use the neural markers of speech processing that emerge from this framework as features for a diagnostic model that predicts MCI using a binary classifier, considering the hearing status of the participants.ConclusionThe main goal of this study is to investigate neural markers of speech processing for their differential diagnostic potential for the brain at risk for dementia. Depending on the results, such neurophysiological markers could prove useful for early diagnosis in the clinic in the longer term.