Neural speech tracking and auditory attention decoding in everyday life

While infants' sensitivity to visual speech cues and the benefit of these cues have been well-established by behavioural studies, there is little evidence on the effect of visual speech cues on infants' neural processing of continuous auditory speech. In this study, we investigated whether visual speech cues, such as the movements of the lips, jaw, and larynx, facilitate infants' neural speech tracking. Ten-month-old Dutch-learning infants watched videos of a speaker reciting passages in infant-directed speech while electroencephalography (EEG) was recorded. In the videos, either the full face of the speaker was displayed or the speaker's mouth and jaw were masked with a block, obstructing the visual speech cues. To assess neural tracking, speech-brain coherence (SBC) was calculated, focusing particularly on the stress and syllabic rates (1-1.75 and 2.5-3.5Hz respectively in our stimuli). First, overall, SBC was compared to surrogate data, and then, differences in SBC in the two conditions were tested at the frequencies of interest. Our results indicated that infants show significant tracking at both stress and syllabic rates. However, no differences were identified between the two conditions, meaning that infants' neural tracking was not modulated further by the presence of visual speech cues. Furthermore, we demonstrated that infants' neural tracking of low-frequency information is related to their subsequent vocabulary development at 18 months. Overall, this study provides evidence that infants' neural tracking of speech is not necessarily impaired when visual speech cues are not fully visible and that neural tracking may be a potential mechanism in successful language acquisition.

The brain tracks incoming speech by aligning the phase of its low-frequency endogenous oscillatory activity to the slow amplitude modulations in the speech signal and is modulated by various bottom-up and top-down factors such as speech acoustics or selective attention. However, it is unclear whether speech tracking can be influenced by voice characteristics such bottom-up driven voice identity or the top-down feature of familiarity with the speaker’s voice. Speech from familiar speakers is more intelligible, so higher speech tracking response can be expected for speech from familiar compared to unfamiliar speakers. To test this, we familiarized 39 young adult listeners with four female voices. We used high-variability training containing sentence stimuli from different speaking styles and audio qualities to ensure listeners can recognize voices reliably across different stimuli types and form more stable voice representations. After this, we recorded EEG while participants listened to sentences from familiar and unfamiliar speakers in quiet and in multitalker babble noise. To quantify neural speech tracking, we computed cross-correlation between the neural response and the amplitude envelopes of the presented sentences. Results revealed that lab-trained familiarity with female voices does not affect neural speech tracking. We also found no differences in sentence comprehension scores for familiar and unfamiliar voices suggesting that lab-trained familiarity does not benefit sentence comprehension. However, we found a significant effect of individual speakers on speech tracking suggesting that neural speech tracking is modulated by voice identity. We also found a relationship between voice recognition and speech tracking: the better a particular speaker was recognized, the higher the speech tracking response was to the respective speaker. Our results further support the notion that voice identity is reflected in early acoustic and later linguistic processing stages in the brain and that linguistic and speaker-specific voice information are processed jointly on the neural level.

Auditory attention decoding (AAD) is the process of identifying the attended speech in a multi-talker environment using brain signals, typically recorded through electroencephalography (EEG). Over the past decade, AAD has undergone continuous development, driven by its promising application in neuro-steered hearing devices. Most AAD algorithms are relying on the increase in neural entrainment to the envelope of attended speech, as compared to unattended speech, typically using a two-step approach. First, the algorithm predicts representations of the attended speech signal envelopes; second, it identifies the attended speech by finding the highest correlation between the predictions and the representations of the actual speech signals. In this study, we proposed a novel end-to-end neural network architecture, named AADNet, which combines these two stages into a direct approach to address the AAD problem. We compare the proposed network against traditional stimulus decoding-based approaches, including linear stimulus reconstruction, canonical correlation analysis, and an alternative non-linear stimulus reconstruction using three different datasets. AADNet shows a significant performance improvement for both subject-specific and subject-independent models. Notably, the average subject-independent classification accuracies for different analysis window lengths range from 56.3% (1 s) to 78.1% (20 s), 57.5% (1 s) to 89.4% (40 s), and 56.0% (1 s) to 82.6% (40 s) for three validated datasets, respectively, showing a significantly improved ability to generalize to data from unseen subjects. These results highlight the potential of deep learning models for advancing AAD, with promising implications for future hearing aids, assistive devices, and clinical assessments.

The auditory attention decoding (AAD) approach was proposed to determine the identity of the attended talker in a multi-talker scenario by analyzing electroencephalography (EEG) data. Although the linear model-based method has been widely used in AAD, the linear assumption was considered oversimplified and the decoding accuracy remained lower for shorter decoding windows. Recently, nonlinear models based on deep neural networks (DNN) have been proposed to solve this problem. However, these models did not fully utilize both the spatial and temporal features of EEG, and the interpretability of DNN models was rarely investigated. In this paper, we proposed novel convolutional recurrent neural network (CRNN) based regression model and classification model, and compared them with both the linear model and the state-of-the-art DNN models. Results showed that, our proposed CRNN-based classification model outperformed others for shorter decoding windows (around 90% for 2 s and 5 s). Although worse than classification models, the decoding accuracy of the proposed CRNN-based regression model was about 5% greater than other regression models. The interpretability of DNN models was also investigated by visualizing layers' weight.

Speech perception in crowded environments is challenging for hearing-impaired listeners. Assistive hearing devices cannot lower interfering speakers without knowing which speaker the listener is focusing on. One possible solution is auditory attention decoding in which the brainwaves of listeners are compared with sound sources to determine the attended source, which can then be amplified to facilitate hearing. In realistic situations, however, only mixed audio is available. We utilize a novel speech separation algorithm to automatically separate speakers in mixed audio, with no need for the speakers to have prior training. Our results show that auditory attention decoding with automatically separated speakers is as accurate and fast as using clean speech sounds. The proposed method significantly improves the subjective and objective quality of the attended speaker. Our study addresses a major obstacle in actualization of auditory attention decoding that can assist hearing-impaired listeners and reduce listening effort for normal-hearing subjects.

This study investigates the neural encoding of speech features in hearing aid users using electroencephalography (EEG) during a simulated cocktail party scenario. The objective was to investigate neural tracking of various acoustic and linguistic features and how hearing aid noise reduction influenced this tracking. The features analyzed included the acoustic envelope, phonetic features, word onset, and word surprisal, the latter derived from GPT-2. Temporal Response Functions (TRFs) were used to correlate these features with EEG signals, revealing how the brain tracks attended (target) versus unattended (masker) speech. TRFs were estimated using a boosting algorithm, with speech features as predictors and EEG signals as responses. Results revealed a significant distinction between target and masker speech. The acoustic envelope showed the strongest correlation with EEG responses. Distinct tracking patterns were observed: the acoustic envelope and phonetic features correlated with early processing stages, while word onset and word suprisal were linked to later stages. Noise reduction further influenced the tracking of these features. These findings improve our understanding of how hearing aid users process speech and provide insight for developing hearing aids that adapt to individual neural responses.

Auditory attention decoding (AAD) utilizes electroencephalography (EEG) to detect an individual's attentional focus in noisy, multi-speaker environments, serving as a cornerstone for next-generation neuro-steered hearing aids. Although deep learning has recently surpassed traditional linear models in decoding accuracy, a holistic overview connecting preprocessing pipelines with advanced model architectures is lacking. This systematic review addresses this deficiency by analyzing 82 studies selected via PRISMA guidelines. We comprehensively dissect the AAD framework, from data processing to the evolution of architectures like convolutional neural networks (CNNs), graph neural networks (GNNs), and Transformers. Beyond reviewing trends, we synthesize practical design guidance, recommending the alignment of preprocessing complexity with model capacity and the use of hybrid architectures to capture spatiotemporal dynamics. Crucially, this study highlights persistent challenges impeding real-world transferability, including the reliance on high-density EEG montages incompatible with wearables, the computational latency on edge devices, and the lack of realistic acoustic scenarios. By identifying these bottlenecks and synthesizing effective design choices, this review offers a structured reference for developing robust, low-latency, and subject-independent auditory attention detection systems suitable for practical brain-computer interface applications.

The performance of speech enhancement algorithms in a multi-speaker scenario depends on correctly identifying the target speaker to be enhanced. Auditory attention decoding (AAD) methods allow to identify the target speaker which the listener is attending to from single-trial EEG recordings. In this paper we propose a cognitive-driven multi-microphone speech enhancement system, which combines a neural-network-based mask estimator, weighted minimum power distortionless response convolutional beamformers and AAD. The proposed system allows to enhance the attended speaker and jointly suppress reverberation, the interfering speaker and ambient noise. To control the suppression of the interfering speaker, we also propose an extension incorporating an interference suppression constraint. The experimental results show that the proposed system outperforms the state-of-the-art cognitive-driven speech enhancement systems in reverberant and noisy conditions.

Objective. This study aimed to investigate the potential of contrastive learning to improve auditory attention decoding (AAD) using electroencephalography (EEG) data in challenging cocktail-party scenarios with competing speech and background noise.Approach. Three different models were implemented for comparison: a baseline linear model (LM), a non-LM without contrastive learning (NLM), and a non-LM with contrastive learning (NLMwCL). The EEG data and speech envelopes were used to train these models. The NLMwCL model used SigLIP, a variant of CLIP loss, to embed the data. The speech envelopes were reconstructed from the models and compared with the attended and ignored speech envelopes to assess reconstruction accuracy, measured as the correlation between the reconstructed and actual speech envelopes. These reconstruction accuracies were then compared to classify attention. All models were evaluated in 34 listeners with hearing impairment.Results. The reconstruction accuracy for attended and ignored speech, along with attention classification accuracy, was calculated for each model across various time windows. The NLMwCL consistently outperformed the other models in both speech reconstruction and attention classification. For a 3-second time window, the NLMwCL model achieved a mean attended speech reconstruction accuracy of 0.105 and a mean attention classification accuracy of 68.0%, while the NLM model scored 0.096 and 64.4%, and the LM achieved 0.084 and 62.6%, respectively.Significance. These findings demonstrate the promise of contrastive learning in improving AAD and highlight the potential of EEG-based tools for clinical applications, and progress in hearing technology, particularly in the design of new neuro-steered signal processing algorithms.

In complex sound scenes, where multiple sounds are present around a listener, selective attention to one auditory stream is hypothesized to synchronize low-frequency brain activity with the envelope of the attended streams. Recent research has employed stimulus reconstruction from neural data to decode to which auditory stream a listener is paying attention. This could be used to create an auditory attention decoder (AAD), that could be embedded in smart headphones or hearing aids, that would adapt the sound processing based on the attention of the user. However, most of these studies use full scalp electroencephalogram, which is not suitable for implementations in audio devices. To that aim, a smaller EEG device, with fewer electrodes could be used. In the present study, we explore the performance of an AAD based on a smaller number of electrodes during speech and music listening. Participants were presented with two sounds simultaneously, and where asked to attend to one while ignoring the other, and their cortical response was continuously recorded during the lsitening. Using a greedy approach based on reconstruction accuracy, a subset of EEG electrodes that are optimized for linear stimulus reconstruction were selected. The goal of this study is to explore the performance of a linear AAD when reducing the number of electrodes. Results suggest that four well-selected electrodes can be sufficient for a miniaturized AAD as it performs as well as a 64-channels setup. The channels selected vary depending on the type of sound attended, suggesting that different electrodes placement should be used to decode attention during music listening and speech listening.

Objective. Unobtrusive electroencephalography (EEG) monitoring in everyday life requires the availability of highly miniaturized EEG devices (mini-EEGs), which ideally consist of a wireless node with a small scalp area footprint, in which the electrodes, amplifier and wireless radio are embedded. By attaching a multitude of mini-EEGs at relevant positions on the scalp, a wireless ‘EEG sensor network’ (WESN) can be formed. However, each mini-EEG in the network only has access to its own local electrodes, thereby recording local scalp potentials with short inter-electrode distances. This is unlike using traditional cap-EEG, which by the virtue of re-referencing can measure EEG across arbitrarily large distances on the scalp. We evaluate the implications and limitations of such far-driven miniaturization on neural decoding performance. Approach. We collected 255-channel EEG data in an auditory attention decoding (AAD) task. As opposed to previous studies with a lower channel density, this new high-density dataset allows emulation of mini-EEGs with inter-electrode distances down to 1 cm in order to identify and quantify the lower bound on miniaturization for EEG-based stimulus decoding. Main results. We demonstrate that the performance remains reasonably stable for inter-electrode distances down to 3 cm, but decreases quickly for shorter distances if the mini-EEG nodes can be placed at optimal scalp locations and orientations selected by a data-driven algorithm. Significance. The results indicate the potential for the use of mini-EEGs in a WESN context for AAD applications and provide guidance on inter-electrode distances while designing such devices for neuro-steered hearing devices.

Objective. To develop an efficient, embedded electroencephalogram (EEG) channel selection approach for deep neural networks, allowing us to match the channel selection to the target model, while avoiding the large computational burdens of wrapper approaches in conjunction with neural networks. Approach. We employ a concrete selector layer to jointly optimize the EEG channel selection and network parameters. This layer uses a Gumbel-softmax trick to build continuous relaxations of the discrete parameters involved in the selection process, allowing them be learned in an end-to-end manner with traditional backpropagation. As the selection layer was often observed to include the same channel twice in a certain selection, we propose a regularization function to mitigate this behavior. We validate this method on two different EEG tasks: motor execution and auditory attention decoding. For each task, we compare the performance of the Gumbel-softmax method with a baseline EEG channel selection approach tailored towards this specific task: mutual information and greedy forward selection with the utility metric respectively. Main results. Our experiments show that the proposed framework is generally applicable, while performing at least as well as (and often better than) these state-of-the-art, task-specific approaches. Significance. The proposed method offers an efficient, task- and model-independent approach to jointly learn the optimal EEG channels along with the neural network weights.

This study evaluates the feasibility of using a dry-electrode electroencephalography (EEG) system for auditory cognitive neuroscience research by comparing its performance with a conventional wet-electrode system during auditory tasks. EEG data were collected using the Biosemi Active Two (wet-electrode system) and the Wearable Sensing DSI-24 (dry-electrode system) as participants completed a Speech-in-Noise task based on the Coordinate Response Measure (CRM) corpus. Participants identified the color and number of target sounds while masker sounds varied in direction and signal-to-noise ratio (SNR). Results reveal no significant differences in SNR between the dry- and wet-electrode systems. This indicates that the dry-electrode system can reliably capture auditory neural responses, making it a suitable tool for auditory neuroscience research. The use of auditory stimuli in this study highlights the applicability of dry-electrode systems for investigating brain responses to complex auditory environments, such as speech-in-noise. These findings suggest that dry-electrode EEG systems offer a practical and efficient alternative for auditory cognitive neuroscience, enabling easier setup and improved participant comfort without compromising data quality. This advancement holds promise for applications in auditory attention decoding, brain-computer interfaces, and clinical research.

Auditory Attention Decoding in Four-Talker Environment with EEG

Importance: Developmental language problems must be detected early to capitalize on the high effectiveness of early intervention. Because developmental language problems are not formally diagnosed until preschool years, a method must be developed and validated to forecast language problems from as early as infancy. Objective: To develop and validate predictive models of language problems using neural data collected from as early as infancy and language problems detected at up to 36 months of age.Design: A cohort study with initial electroencephalography (EEG) neural encoding of speech data collected between 3 weeks and 24 months of age and language outcome data collected between 7 and 36 months of age. The data reported were collected between 2016 and 2024. The cutoff date for analysis was March 25, 2024.Setting: Laboratory and home. Participants: A total of 726 healthy children from a community sample were recruited, 534 of whom with complete EEG and follow-up language data were included in the final analysis.Exposure: Early- and long-latency EEG responses to three speech stimuli (two native and one non-native) were collected. No intervention was involved.Main Outcome Measures: The Chinese Communicative Development Inventory – Cantonese version (CCDI-C) and the language subscale of the Bayley Scales of Infant and Toddler Development, third edition (Bayley-III).Results: Random Forest was used as the machine learning algorithm to classify children into binary groups of below or at or above 16th percentile on the outcome measures. Different predictive models were constructed and compared, including those with and without EEG and clinical measures. Models with non-neural measures (e.g., gestational age and birth weight) predicted a language outcome above chance level. Models with EEG measures alone outperformed any models without neural features, with sensitivity and Area Under Curve (AUC) scoring well above 90% for the best models. When validated using unseen data, both sensitivity and AUC remained at 88%.Conclusion & Relevance: EEG neural encoding of speech can be a new procedure for screening language developmental problems as early as infancy. Families with children who are screened positive may consider early intervention in the form of parent coaching to promote better language development.