AADNet: An End-to-End Deep Learning Model for Auditory Attention Decoding.

Speech perception in crowded environments is challenging for hearing-impaired listeners. Assistive hearing devices cannot lower interfering speakers without knowing which speaker the listener is focusing on. One possible solution is auditory attention decoding in which the brainwaves of listeners are compared with sound sources to determine the attended source, which can then be amplified to facilitate hearing. In realistic situations, however, only mixed audio is available. We utilize a novel speech separation algorithm to automatically separate speakers in mixed audio, with no need for the speakers to have prior training. Our results show that auditory attention decoding with automatically separated speakers is as accurate and fast as using clean speech sounds. The proposed method significantly improves the subjective and objective quality of the attended speaker. Our study addresses a major obstacle in actualization of auditory attention decoding that can assist hearing-impaired listeners and reduce listening effort for normal-hearing subjects.

In complex sound scenes, where multiple sounds are present around a listener, selective attention to one auditory stream is hypothesized to synchronize low-frequency brain activity with the envelope of the attended streams. Recent research has employed stimulus reconstruction from neural data to decode to which auditory stream a listener is paying attention. This could be used to create an auditory attention decoder (AAD), that could be embedded in smart headphones or hearing aids, that would adapt the sound processing based on the attention of the user. However, most of these studies use full scalp electroencephalogram, which is not suitable for implementations in audio devices. To that aim, a smaller EEG device, with fewer electrodes could be used. In the present study, we explore the performance of an AAD based on a smaller number of electrodes during speech and music listening. Participants were presented with two sounds simultaneously, and where asked to attend to one while ignoring the other, and their cortical response was continuously recorded during the lsitening. Using a greedy approach based on reconstruction accuracy, a subset of EEG electrodes that are optimized for linear stimulus reconstruction were selected. The goal of this study is to explore the performance of a linear AAD when reducing the number of electrodes. Results suggest that four well-selected electrodes can be sufficient for a miniaturized AAD as it performs as well as a 64-channels setup. The channels selected vary depending on the type of sound attended, suggesting that different electrodes placement should be used to decode attention during music listening and speech listening.

The human brain’s remarkable ability to focus on individual speakers in multi-talker environments reveals sophisticated auditory attention mechanisms. Decoding such auditory attention from electroencephalogram (EEG) signals presents substantial challenges, particularly for achieving high-precision decoding within sub-second temporal resolution. Here we introduce a lightweight neural network that integrates convolutional layers, residual connections, and attention mechanisms to significantly improve auditory attention decoding (AAD) from sub-second electroencephalogram (EEG) signals. The proposed model achieves state-of-the-art accuracies of 95.4% on the KUL dataset and 94.6% on the DTU dataset within a challenging 0.2-second decision window. Notably, our architecture contains only 0.057 M trainable parameters, a greater than 16 -fold reduction compared to baseline models, establishing a computationally efficient framework for real-time neural decoding. This work marks a substantial advancement in developing practical, high-precision brain-computer interfaces.

IntroductionIn our complex world, the auditory system plays a crucial role in perceiving and processing our environment. Humans are able to segment and stream concurrent auditory objects, allowing them to focus on specific sounds, such as speech, and suppress irrelevant auditory objects. The attentional enhancement or suppression of sound processing is evident in neural data through a phenomenon called neural speech tracking. Previous studies have identified correlates of neural speech tracking in electroencephalography (EEG) data, but EEG measures are susceptible to motion artefacts, and the association between neural data and auditory objects is vulnerable to distraction.MethodsThe current study investigated EEG-based auditory attention decoding in realistic everyday scenarios. N=20 participants were exposed to the sound of a busy cafeteria or walked along busy and quiet streets while listening to one or two simultaneous speech streams. We also investigated the robustness of neural speech tracking estimates within subjects. Linear decoding models were used to determine the magnitude of neural speech tracking.ResultsThe results confirmed that neural speech tracking was strongest in single speaker scenarios. In dual speaker conditions, there was significantly stronger neural speech tracking for the attended speaker compared to the ignored speaker, even in complex environments such as a busy cafeteria or outdoor settings.DiscussionIn conclusion, EEG-based attention decoding is feasible in highly complex and realistic everyday conditions while humans behave naturally.

HighlightsWhat are the main findings?We evaluate deception detection from five-channel wearable EEG on the public LieWaves dataset (27 subjects, Emotiv Insight headset) under both subject-dependent (same-subject) and subject-independent (cross-subject) protocols.In the subject-independent configuration, a compact ResNet-SE model trained on raw overlapping windows and evaluated with a single session-level decision threshold estimated from cross-validated scores attains 66.70% session-level accuracy (AUC = 57.80%) on subjects not seen by the model that predicts them.In the subject-dependent configuration, an overlapping short-window Res-TCN-SE-Attention model that fuses raw EEG with DWT-based spectral and handcrafted band-power and Hjorth features reaches 99.94% window-level accuracy on held-out windows from held-out sessions of the same individuals under a heavily overlapping, subject-dependent but session-disjoint split. This near-ceiling window-level figure should be interpreted only as an optimistic upper bound in an autocorrelated, subject-dependent regime, and not as an estimate of deployment-relevant deception-detection ability.What are the implications of the main findings?Five-channel wearable EEG thus carries modest but above-chance information for lie–truth discrimination in cross-subject settings, suggesting that portable screening might be technically feasible only in tightly controlled scenarios. At the same time, the pronounced generalization gap between subject-dependent and subject-independent models underscores that substantial additional work on robustness, calibration, and external validation is required before real-world deployment.By making our subject-wise splits, windowing rules, decision thresholds, and session-level aggregation explicit, we provide a concrete, reproducible benchmark for five-channel wearable EEG deception detection and highlight the need for standardized, clearly reported evaluation protocols.Deception detection with low-channel wearable EEG requires protocols that generalize across people while remaining practical for portable devices. Using the public LieWaves dataset (27 subjects recorded with a five-channel Emotiv Insight headset), we evaluate to what extent five-channel head-mounted EEG can support lie–truth discrimination under both subject-independent and subject-dependent evaluations. For the subject-independent setting, we train a compact Residual Network with Squeeze-and-Excitation blocks (ResNet-SE) model on raw overlapping windows with focal loss, light data augmentation, and grouped cross-validation by subject; out-of-fold window probabilities are averaged per session and converted to labels using a single decision threshold estimated from the cross-validated session scores. For the subject-dependent setting, we adopt an overlapping short-window Residual Temporal Convolutional Network with Squeeze-and-Excitation and Attention (Res-TCN-SE-Attention) model that fuses raw EEG with discrete wavelet transform (DWT)-based spectral and handcrafted band-power and Hjorth features, using an 80/10/10 split at the recording/session level (stratified by session label), so that all windows from a given session are assigned to a single subset; because each subject contributes two sessions, the same subject may still appear across subsets via different sessions. The subject-independent model attains 66.70% session-level accuracy with an AUC of 0.58 on unseen subjects, underscoring the difficulty of person-independent generalization from low-channel wearable EEG. Because practical deployment requires generalization to previously unseen individuals, we treat the subject-independent evaluation as the primary estimate of real-world generalization. In contrast, the subject-dependent pipeline reaches 99.94% window-level accuracy under the overlapping sliding-window (OSW) setting with a session-disjoint split (no session contributes windows to more than one subset). This near-ceiling performance reflects the optimistic nature of subject-dependent evaluation with highly overlapping windows, even when avoiding within-session train–test overlap, and should not be interpreted as a meaningful indicator of deception-detection capability under realistic deployment constraints. These results suggest limited, above-chance separability between lie and truth sessions in LieWaves using a five-channel wearable EEG under the studied protocol; however, performance remains far from deployment-ready and is strongly shaped by evaluation design. Explicit reporting of both protocols, together with clear rules for windowing, aggregation, and threshold selection, supports more reproducible and comparable benchmarking.

Publication in the conference proceedings of EUSIPCO, Antalya, Turkey, 2005

Speech signal has various features of speakers. This feature is extracted from speech signal processing. The speaker is identified by the speaker identification system. In this paper, we propose the speaker identification system that uses the incremental learning based on neural network. Recorded speech signal through the microphone is blocked to the frame of 1024 speech samples. Energy is divided speech signal to voiced signal and unvoiced signal. The extracted 12 orders LPC cpestrum coefficients are used with input data for neural network. The speakers are identified with the speaker identification system using the neural network. The neural network has the structure of MLP which consists of 12 input nodes, 8 hidden nodes, and 4 output nodes. The number of output node means the identified speakers. The first output node is excited to the first speaker. Incremental learning begins when the new speaker is identified. Incremental learning is the learning algorithm that already learned weights are remembered and only the new weights that are created as adding new speaker are trained. It is learning algorithm that overcomes the fault of neural network. The neural network repeats the learning when the new speaker is entered to it. The architecture of neural network is extended with the number of speakers. Therefore, this system can learn without the restricted number of speakers.

Feature extraction from speech representation is one of the processes in speech recognition. Parametric modeling is a dominant approach to model speech signals. Within a localized interval, speech representation is equivalent to a noise driven output from an all-pole system that can be estimated using linear prediction. Besides the characteristics of speech, temporal variability of speech signal model is also due to the computation of linear prediction coefficients. Thus, an alternative representation is proposed based on the Gabor coefficients. In this paper, a comparison is made with the linear prediction coefficients to show the consistency of the parameters that are generated for implementation in the speech recognition system.

Real-time walking gait terrain classification from foot-mounted Inertial Measurement Unit using Convolutional Long Short-Term Memory neural network

Before the advancement of deep learning technology, the brain signals are to be analysed manually by the neuroscientists on how the brain signals reacts in proportion with the human body. This process is very time consuming and unreliable. Therefore, this project aims to develop a brain signal detection system based on deep learning algorithm in response to the output of EEG device on the imagery finger movements. These fingers include thumb, index, middle, ring and little of right hand. There are 4 CNN classification models being developed in this project. They differ with each other in terms of the pre-processing requirements and the neural network architecture. The best results for offline classification obtained in this project are 69.07% and 82.83% respectively in terms of average accuracy from 6-class and 2-class tests. Moreover, this project has also developed a proof of concept for applying the trained models in online or real-time classification.

Facial expression may establish communication between physically disabled people and assistive devices. Different types of facial expression including eye wink, smile, eye blink, looking up and looking down can be extracted from the brain signal. In this study, the possibility of controlling assistive devices using the individual’s wink has been investigated. Brain signals from the five subjects have been captured to recognize the left wink, right wink, and no wink. The brain signals have been captured using Emotiv Insight which consists of five channels. Fast Fourier transform and the sample range have been computed to extract the features. The extracted features have been classified with the help of different machine learning algorithms. Here, support vector machine (SVM), linear discriminant analysis (LDA) and K-nearest neighbor (K-NN) have been employed to classify the features sets. The performance of the classifier in terms of accuracy, confusion matrix, true positive and false positive rate and the area under curve (AUC)—receiver operating characteristics (ROC) have been evaluated. In the case of sample range, the highest training and testing accuracies are 98.9% and 96.7% respectively which have been achieved by two classifiers namely, SVM and K-NN. The achieved results indicate that the person’s wink can be utilized in controlling assistive devices.

Color vision deficiency (CVD), a common visual impairment, affects individuals' ability to differentiate between various colors due to malfunctioning or absent color photoreceptors in the retina. Currently available diagnostic tests require a behavioral response, rendering them unsuitable for individuals with limited physical and communication abilities, such as those with locked-in syndrome. This study introduces a novel, non-invasive method that employs brain signals, specifically Steady-State Visually Evoked Potentials (SSVEPs), along with Ishihara plates to diagnose CVD. This method aims to provide an alternative diagnostic tool that addresses the limitations of current tests. Electroencephalography (EEG) recordings were obtained from 16 subjects, including 5 with CVD (specifically Deuteranomaly), using channels O1, O2, Pz, and Cz. The subjects were exposed to visual stimuli at frequencies of 15 Hz and 18 Hz to assess the proposed method. The subjects focused on specific visual stimuli in response to questions related to the Ishihara plates. Their responses were analyzed to determine the presence of CVD. Feature extraction was performed using Power Spectral Density (PSD), Canonical Correlation Analysis (CCA), and a combined PSD + CCA, followed by classification to categorize subjects into two classes: normal vision and CVD. The results indicate that the proposed method effectively diagnoses CVD in individuals with limited communication abilities. The classification accuracy of SSVEP exceeded 75% across the three classifiers: Decision Tree (DT), K-Nearest Neighbors (KNN), and Support Vector Machine (SVM). The SVM classifier demonstrated higher accuracy compared to the other classifiers, exceeding 90%. These observations suggest that the SVM classifier, utilizing the combined feature set of PSD + CCA, may be the most effective in this classification task. These findings demonstrate that the proposed method is an accurate and reliable diagnostic tool for CVD, particularly for individuals unable to communicate.

Speech signal processing on graphs: The graph frequency analysis and an improved graph Wiener filtering method

Scene change detection is the process of identifying the differences between the multi-temporal image scenes at the semantic level, which has significant potential in the application of urban development and land management. In this paper, we propose a novel deep convolution canonical correlation analysis neural network (DCCANet) architecture, which could consider the spectral-spatial-temporal correlation feature for scene change detection in remote sensing images. For this purpose, we put together the convolutional neural networks (CNNs) and the deep canonical correlation analysis (DCCA) into the end-to-end network. The CNN could get the spectral-spatial feature information for scene representation, while the later could enhance the temporal correlation by nonlinear high- dimensional transformations between the multi-temporal image scenes for scene change detection. Experiments with high-resolution remote sensing image scene datasets demonstrated that our proposed approach can get a better performance in scene classification and change detection.

Analysis of brain signals is essential to the study of mental states and various neurological conditions. The two most prevalent noninvasive signals for measuring brain activities are electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, characterized by its higher sampling frequency, captures more temporal features, while fNIRS, with a greater number of channels, provides richer spatial information. Although a few previous studies have explored the use of multimodal deep-learning models to analyze brain activity for both EEG and fNIRS, subject-independent training-testing split analysis remains underexplored. The results of the subject-independent setting directly show the model's ability on unseen subjects, which is crucial for real-world applications. In this paper, we introduce EF-Net, a new CNN-based multimodal deep-learning model. We evaluate EF-Net on an EEG-fNIRS word generation (WG) dataset on the mental state recognition task, primarily focusing on the subject-independent setting. For completeness, we report results in the subject-dependent and subject-semidependent settings as well. We compare our model with five baseline approaches, including three traditional machine learning methods and two deep learning methods. EF-Net demonstrates superior performance in both accuracy and F1 score, surpassing these baselines. Our model achieves F1 scores of 99.36%, 98.31%, and 65.05% in the subject-dependent, subject-semidependent, and subject-independent settings, respectively, surpassing the best baseline F1 scores by 1.83%, 4.34%, and 2.13% These results highlight EF-Net's capability to effectively learn and interpret mental states and brain activity across different and unseen subjects.