Auditory-GAN: deep learning framework for improved auditory spatial attention detection.

DGSD: Dynamical graph self-distillation for EEG-based auditory spatial attention detection

In the context of a multi-speaker "cocktail party" scenario where listeners selectively focus on specific speakers, human auditory attention networks have shown a strong correlation with Electroencephalography (EEG) measurements. However, current EEG-based auditory attention detection (AAD) methods, mostly using artificial neural networks (ANN), face limitations on edge computing platforms due to extended decision windows, high power consumption, and substantial memory requirements linked to multiple EEG channels. This paper introduces a novel hybrid convolutional-spiking neural network (CNN-SNN) architecture, inspired by the auditory cortex, combining EEG data with multi-speaker speech envelopes, enabling effective auditory attention decoding within 0.5-s timeframes. Our approach reduces EEG channels, minimizes computational operations, and quantizes weight parameters while maintaining high accuracy. We validate this approach on our dataset and compare it to state-of-the-art methods on a publicly available dataset. CNN-SNN demonstrates superior performance, achieving up to 10% increase in decoding accuracy, while using 87.5% fewer EEG channels and 75% smaller bit precision for weight quantization compared to existing methods. These results offer promise for edge computing applications, such as hearing aids, emphasizing short decision windows, minimal EEG channels, and strict power and memory constraints.

Attention is one of many human cognitive functions that are essential in everyday life. Given our limited processing capacity, attention helps us focus only on what matters. Focusing attention on one speaker in an environment with many speakers is a critical ability of the human auditory system. This paper proposes a new end-to-end method based on the combined transformer and graph convolutional neural network (TraGCNN) that can effectively detect auditory attention from electroencephalograms (EEGs). This approach eliminates the need for manual feature extraction, which is often time-consuming and subjective. Here, the first EEG signals are converted to graphs. We then extract attention information from these graphs using spatial and temporal approaches. Finally, our models are trained with these data. Our model can detect auditory attention in both the spatial and temporal domains. Here, the EEG input is first processed by transformer layers to obtain a sequential representation of EEG based on attention onsets. Then, a family of graph convolutional layers is used to find the most active electrodes using the spatial position of electrodes. Finally, the corresponding EEG features of active electrodes are fed into the graph attention layers to detect auditory attention. The Fuglsang 2020 dataset is used in the experiments to train and test the proposed and baseline systems. The new TraGCNN approach, as compared with state-of-the-art attention classification methods from the literature, yields the highest performance in terms of accuracy (80.12%) as a classification metric. Additionally, the proposed model results in higher performance than our previously graph-based model for different lengths of EEG segments. The new TraGCNN approach is advantageous because attenuation detection is achieved from EEG signals of subjects without requiring speech stimuli, as is the case with conventional auditory attention detection methods. Furthermore, examining the proposed model for different lengths of EEG segments shows that the model is faster than our previous graph-based detection method in terms of computational complexity. The findings of this study have important implications for the understanding and assessment of auditory attention, which is crucial for many applications, such as brain–computer interface (BCI) systems, speech separation, and neuro-steered hearing aid development.

Objective. We consider the problem of Auditory Attention Detection (AAD), where the goal is to detect which speaker a person is attending to, in a multi-speaker environment, based on neural activity. This work aims to analyze the influence of head-related filtering and ear-specific decoding on the performance of an AAD algorithm. Approach. We recorded high-density EEG of 16 normal-hearing subjects as they listened to two speech streams while tasked to attend to the speaker in either their left or right ear. The attended ear was switched between trials. The speech stimuli were administered either dichotically, or after filtering using Head-Related Transfer Functions (HRTFs). A spatio-temporal decoder was trained and used to reconstruct the attended stimulus envelope, and the correlations between the reconstructed and the original stimulus envelopes were used to perform AAD, and arrive at a percentage correct score over all trials. Main results. We found that the HRTF condition resulted in significantly higher AAD performance than the dichotic condition. However, speech intelligibility, measured under the same set of conditions, was lower for the HRTF filtered stimuli. We also found that decoders trained and tested for a specific attended ear performed better, compared to decoders trained and tested for both left and right attended ear simultaneously. In the context of the decoders supporting hearing prostheses, the former approach is less realistic, and studies in which each subject always had to attend to the same ear may find over-optimistic results. Significance. This work shows the importance of using realistic binaural listening conditions and training on a balanced set of experimental conditions to obtain results that are more representative for the true AAD performance in practical applications.

Focusing on a single source within a complex auditory scene is challenging. M/EEG-based auditory attention detection (AAD) allows to detect which stream an individual is attending to within a set of multiple concurrent streams. The high interindividual variability in the auditory attention detection performance often is attributed to physiological factors and signal-to-noise ratio of neural data. We hypothesize that executive functions-in particular sustained attention, working memory, and attentional inhibition-may partly explain the variability in auditory attention detection performance, because they support the cognitive processes required when listening to complex auditory scenes. We chose a particularly challenging auditory scene by presenting dichotically polyphonic classical piano excerpts that lasted 1min each. Two different excerpts were presented simultaneously, one in each ear. Forty-one participants, with different degrees of musical expertise, listened to these complex auditory scenes focusing on one ear while we recorded the EEG. Participants also completed several tasks assessing executive functions. As expected, EEG-based auditory attention detection was greater for attended than unattended stimuli. Importantly, attentional inhibition ability did explain 6% of the reconstruction accuracy and 8% of the classification accuracy. No other executive function was a significant predictor of reconstruction or classification accuracies. No clear effect of musical expertise was found on reconstruction and classification performance. In conclusion, cognitive factors seem to impact the robustness of the neural auditory representation and hence the performance of EEG-based decoding approaches. Taking advantage of this relation could be useful to improve next-generation hearing aids.

State-of-the-art hearing prostheses are equipped with acoustic noise reduction algorithms to improve speech intelligibility. However, cocktail party scenarios with multiple speakers pose a major challenge since it is difficult for the algorithm to determine which speaker it should enhance. To address this problem, electroencephalography (EEG) signals can be used to perform auditory attention detection (AAD), i.e., to detect which speaker the listener is attending to. Taking a step further towards realization of a neuro-steered hearing prosthesis, we worked on AAD-assisted noise suppression in a competing-speakers scenario in the presence of babble noise. We use an EEG-informed AAD module in combination with a blind source separation algorithm to extract the per-speaker envelopes from the microphone recordings, as well as a multi-channel Wiener filter to extract the denoised speech signal(s). Using a new algorithm pipeline, we obtain better AAD accuracies, and a better robustness to variations in speaker positions and signal-to-noise ratios (SNR), compared to previously reported results. Furthermore, the algorithm allows to switch more swiftly to the other speaker’s stream when there is a switch in attention.

This paper considers the auditory attention detection (AAD) paradigm, where the goal is to determine which of two simultaneous speakers a person is attending to. The paradigm relies on recordings of the listener's brain activity, e.g., from electroencephalography (EEG). To perform AAD, decoded EEG signals are typically correlated with the temporal envelopes of the speech signals of the separate speakers. In this paper, we study how the inclusion of various degrees of auditory modelling in this speech envelope extraction process affects the AAD performance, where the best performance is found for an auditory-inspired linear filter bank followed by power law compression. These two modelling stages are computationally cheap, which is important for implementation in wearable devices, such as future neuro-steered auditory prostheses. We also introduce a more natural way to combine recordings (over trials and subjects) to train the decoder, which reduces the dependence of the algorithm on regularization parameters. Finally, we investigate the simultaneous design of the EEG decoder and the audio subband envelope recombination weights vector using either a norm-constrained least squares or a canonical correlation analysis, but conclude that this increases computational complexity without improving AAD performance.

The efficiency of a learning process directly depends on how well the students are attentive. Detecting the level of attention can help to improve the learning quality. In recent years, there have been several attempts to leverage EEG signal processing as a tool to detect whether a student is attentive or not. In such work, the level of attention is determined by analyzing the EEG power spectrum, which is mostly followed by machine learning approaches. However, the efficiency of such methods for detecting auditory attention of blind or visually-impaired students has not been analyzed. This study aims to investigate such a scenario. To this end, we propose an EEG recording protocol to simulate the learning process of visually-impaired students as closely as possible. Ten different university students participated in the experiment. In the proposed protocol, the EEG signals were recorded by using EMOTIV EPOC+ wireless EEG headset. Then, the power spectrum of the recorded EEG signals was analyzed, and the most relevant features were extracted using the Fisher feature selection method. Then, Linear SVM, RBF SVM, KNN, and LDA classifiers were used to evaluate the proposed protocol. The results of the classification showed that the level of auditory attention could be detected up to 89% accuracy.

Existing research has revealed that auditory attention can be tracked from ongoing electroencephalography (EEG) signals. The aim of this novel study was to investigate the identification of peoples’ attention to a specific auditory object from single-trial EEG signals via entropy measures and machine learning. Approximate entropy (ApEn), sample entropy (SampEn), composite multiscale entropy (CmpMSE) and fuzzy entropy (FuzzyEn) were used to extract the informative features of EEG signals under three kinds of auditory object-specific attention (Rest, Auditory Object1 Attention (AOA1) and Auditory Object2 Attention (AOA2)). The linear discriminant analysis and support vector machine (SVM), were used to construct two auditory attention classifiers. The statistical results of entropy measures indicated that there were significant differences in the values of ApEn, SampEn, CmpMSE and FuzzyEn between Rest, AOA1 and AOA2. For the SVM-based auditory attention classifier, the auditory object-specific attention of Rest, AOA1 and AOA2 could be identified from EEG signals using ApEn, SampEn, CmpMSE and FuzzyEn as features and the identification rates were significantly different from chance level. The optimal identification was achieved by the SVM-based auditory attention classifier using CmpMSE with the scale factor τ = 10. This study demonstrated a novel solution to identify the auditory object-specific attention from single-trial EEG signals without the need to access the auditory stimulus.

Auditory spatial attention detection (ASAD) is used to determine the direction of a listener's attention to a speaker by analyzing her/his electroencephalographic (EEG) signals. This study aimed to further improve the performance of ASAD with a short decision window (i.e., <1 s) rather than with long decision windows ranging from 1 to 5 seconds in previous studies. An end-to-end temporal attention network (i.e., TAnet) was introduced in this work. TAnet employs a multi-head attention (MHA) mechanism, which can more effectively capture the interactions among time steps in collected EEG signals and efficiently assign corresponding weights to those EEG time steps. Experiments demonstrated that, compared with the CNN-based method and recent ASAD methods, TAnet provided improved decoding performance in the KUL dataset, with decoding accuracies of 92.4% (decision window 0.1 s), 94.9% (0.25 s), 95.1% (0.3 s), 95.4% (0.4 s), and 95.5% (0.5 s) with short decision windows (i.e., <1 s). As a new ASAD model with a short decision window, TAnet can potentially facilitate the design of EEG-controlled intelligent hearing aids and sound recognition systems.

Objective: To establish the modality specificity and generality of selective attention networks.Method: Forty-eight young adults completed a battery of four auditory and visual selective attention tests based upon the Attention Network framework: the visual and auditory Attention Network Tests (vANT, aANT), the Test of Everyday Attention (TEA), and the Test of Attention in Listening (TAiL). These provided independent measures for auditory and visual alerting, orienting, and conflict resolution networks. The measures were subjected to an exploratory factor analysis to assess underlying attention constructs.Results: The analysis yielded a four-component solution. The first component comprised of a range of measures from the TEA and was labeled “general attention.” The third component was labeled “auditory attention,” as it only contained measures from the TAiL using pitch as the attended stimulus feature. The second and fourth components were labeled as “spatial orienting” and “spatial conflict,” respectively—they were comprised of orienting and conflict resolution measures from the vANT, aANT, and TAiL attend-location task—all tasks based upon spatial judgments (e.g., the direction of a target arrow or sound location).Conclusions: These results do not support our a-priori hypothesis that attention networks are either modality specific or supramodal. Auditory attention separated into selectively attending to spatial and non-spatial features, with the auditory spatial attention loading onto the same factor as visual spatial attention, suggesting spatial attention is supramodal. However, since our study did not include a non-spatial measure of visual attention, further research will be required to ascertain whether non-spatial attention is modality-specific.

Auditory attention switch detection (AASD) poses significant challenges for adaptive neurotechnologies, particularly under electroencephalogram (EEG) with low signal-to-noise ratios (SNRs). However, the performance of existing methods is limited due to insufficient feature discriminability and high detection delay. To solve the problem, this paper proposes a Hierarchical Spatiotemporal Network (HSTN) for detecting auditory attention switch from EEG signals. The model employs a hierarchical spatiotemporal encoder to extract spatiotemporal features of EEG signals, integrates short-term transient and long-term dependency information through a multi-scale gated attention mechanism, and synchronously optimizes auditory attention switch detection and auditory attention decoding tasks via a multi-task joint training strategy. Experimental results demonstrate that HSTN significantly outperforms baseline models in both auditory attention switch detection (AASD F1=0.89, accuracy 88.6%) and auditory attention decoding tasks (AAD accuracy 89.3%), with superior model parameter efficiency and inference time. Ablation experiments further validate the critical roles of multi-task learning, gated attention, and multi-scale convolutions. This study provides an efficient solution for auditory attention switch detection in complex auditory scenarios.Clinical Relevance-The study confirms that spatiotemporal feature encoding combined with multi-task joint training significantly enhances performance in EEG attention switch detection, providing a practical technical framework for enabling dynamic sound source enhancement in intelligent hearing aids and auditory brain-computer interface systems.

Attention is a fundamental aspect of human cognitive function and is crucial for essential activities such as learning, social interaction, and routine tasks. Notably, Auditory attention involves complex interactions and collaboration among multiple brain networks. Recognizing the impairment of auditory attention, comprehending its underlying mechanisms, and identifying the activated brain regions essential for the development of treatments and interventions for individuals facing auditory attention deficits, emphasizes the significance of investigating these matters. In the current study, we conducted a review by searching for the full text of 53 articles published related to auditory attention, mechanisms, and networks in databases like Science Direct, Google Scholar, ProQuest, and PubMed using the keywords Attention, Auditory Attention, Auditory Attention Impairment, theories of attention were investigated in the years 2000 to 2023 And focused on articles that provided discussions within this research domain. The studies have demonstrated that auditory attention exceeds being an acoustic attribute and assumes a fundamental role in complex acoustic environments, information processing, and even speech comprehension. In the context of this study, we have conducted a review and summary of the proposed theories related to attention and the brain networks involved in different forms of auditory attention. In conclusion, the integration of auditory attention assessments, behavioral observations, and an understanding of the neural mechanisms and brain regions implicated in auditory attention proves to be an effective approach for the diagnosis and treatment of attention-related disorders.

Recent work in our laboratory has suggested that human caudal lateral frontal cortex contains four interleaved regions in each hemisphere that exhibit strong sensory-specific biases in attention tasks (Michalka et al, 2014). Two visually-biased attention regions, superior and inferior pre-central sulcus (sPCS, iPCS), anatomically alternate with two auditory-biased attention regions, caudal inferior frontal sulcus (cIFS) and the transverse gyrus intersection the precentral sulcus (tgPCS). These small regions were identified in fMRI studies in a small number of individual subjects. Here, we have investigated these regions and their putative networks by mining the WashU-Minn Human Connectome Project (HCP) dataset. We used data from the 482 HCP participants with both diffusion-weighted imaging and resting-state fMRI. We defined seed regions from our individual subject data in a task that contrasted auditory and visual spatial attention. Probabilistic activation maps were constructed and thresholded to generate ROIs. These ROIs served as seed regions for resting state and tractography analyses of the HCP dataset. Stronger functional connectivity was observed for the sPCS and iPCS than for tgPCS and cIFS with superior parietal lobule visual attention regions, and conversely stronger connectivity was observed for the tgPCS and cIFS than for sPCS and iPCS with superior temporal lobe auditory attention regions. A similar pattern was observed with tractography for all ROIs, except for tgPCS. We next analyzed the whole-brain connectivity patterns of these ROIs using a multivariate approach; we found that the modality of sensory-bias can be predicted well above chance in both hemispheres at a voxelwise scale (L:71%, R:80%), using only the connectivity pattern of an individual voxel. A long-term goal of this analysis is to develop reliable methods for identifying fine-scale brain networks in large population datasets, which could have important clinical applications. Our preliminary results reveal both successes and challenges of these efforts. Meeting abstract presented at VSS 2015

Spatial auditory attention decoding (Sp-AAD) holds great promise for brain-computer interfaces (BCIs). However, studies have shown that the high performance of Sp-AAD relies heavily on eye gaze artifacts rather than actual auditory attention features. For this reason, this study focuses on verifying whether EEG signals contain sufficient discriminative features for attending target speaker identity without eye gaze artifacts. In this study, we proposed an EEG-Mixup data optimization method to suppress trial-specific features in EEG data by adjusting the data distribution and generating soft labels through linear interpolation. In addition, a lightweight EEG-MLP model containing only 2.5k parameters was designed, which showed significant advantages over the latest SOTA model (DenseNet-3D) in cross-trial scenarios. It is shown that the model's generalization ability can be significantly improved by optimizing the data without increasing the data volume; meanwhile, the lightweight model demonstrates higher computational efficiency and inference speed in specific tasks. This study provides important theoretical and practical references for future optimization applications of BCI systems.Clinical Relevance- This study demonstrates the potential of lightweight EEG-based methods for attending target speaker identity without relying on eye gaze artifacts, providing a foundation for future auditory brain-computer interface systems.