Impact of target probability on single-trial EEG target detection in a difficult rapid serial visual presentation task

Objective. Brain–computer interface (BCI) systemsdirectly translate human intentions to instructions for machines by decoding the neural signals. The rapid serial visual presentation (RSVP) task is a typical paradigm of BCIs, in which subjects can detect the targets in the high-speed serial images. There are still two main challenges in electroencephalography (EEG) classification for RSVP tasks: inter-trial variability of event-related potentials (ERPs) and limited trial number of EEG training data.Approach.This study proposed an algorithm of discriminant analysis and classification for interval ERPs (DACIE) in RSVP tasks. Firstly, an interval model of ERPs was exploited to solve the inter-trial variability problem. Secondly, a spatial structured sparsity regularization was utilized to reinforce the important channels, which provided a spatial region of interest (sROI). Meanwhile, a temporal auto-weighting technique was conducted to emphasize the important discriminant components, which obtained a temporal regions of interest (tROIs). Thirdly, classification features were obtained by the discriminant eigenvalue analysis to avoid the ill-conditioned estimation of covariance matrix caused by fewer training trials.Main results.EEG datasets of 12 subjects in RSVP tasks were analyzed to evaluate the classification performance of proposed algorithm. The average accuracy rate, true positive rate, false positive rate and AUC value are 96.9%, 81.6%, 2.8% and 0.938, respectively. Compared with several state-of-the-art algorithms, the proposed algorithm can provide significantly better classification performance.Significance.The interval model of ERPs was exploited in a spatial linear discriminant framework to overcome the inter-trial variability. The sROIs and tROIs were explored to reinforce the pivotal channels and temporal components. And the proposed algorithm can provide good performance with fewer training trials.

Best practice for single-trial detection of event-related potentials: Application to brain-computer interfaces

Electroencephalography (EEG) datasets are characterized by low signal-to-noise signals and unquantifiable noisy labels, which hinder the classification performance in rapid serial visual presentation (RSVP) tasks. Previous approaches primarily relied on supervised learning (SL), which may result in overfitting and reduced generalization performance. In this paper, we propose a novel multi-task collaborative network (MTCN) that integrates both SL and self-supervised learning (SSL) to extract more generalized EEG representations. The original SL task, i.e., the RSVP EEG classification task, is used to capture initial representations and establish classification thresholds for targets and non-targets. Two SSL tasks, including the masked temporal/spatial recognition task, are designed to enhance temporal dynamics extraction and capture the inherent spatial relationships among brain regions, respectively. The MTCN simultaneously learns from multiple tasks to derive a comprehensive representation that captures the essence of all tasks, thus mitigating the risk of overfitting and enhancing generalization performance. Moreover, to facilitate collaboration between SL and SSL, MTCN explicitly decomposes features into task-specific features and task-shared features, leveraging both label information with SL and feature information with SSL. Experiments conducted on THU, CAS, and GIST datasets illustrate the significant advantages of learning more generalized features in RSVP tasks. Our code is publicly accessible at https://github.com/Tammie-Li/MTCN.

Neuroscience studies have demonstrated the phase-locked characteristics of some early event-related potential (ERP) components evoked by stimuli. In this study, we propose a phase preservation neural network (PPNN) to learn phase information to improve the Electroencephalography (EEG) classification in a rapid serial visual presentation (RSVP) task. The PPNN consists of three major modules that can produce spatial and temporal representations with the high discriminative ability of the EEG features for classification. We first adopt a stack of dilated temporal convolution layers to extract temporal dynamics while avoiding the loss of phase information. Considering the intrinsic channel dependence of the EEG data, a spatial convolution layer is then applied to obtain the spatial-temporal representation of the input EEG signal. Finally, a fully connected layer is adopted to extract higher-level features for the final classification. The experiments are conducted on two public and one collected EEG datasets from the RSVP task, in which we evaluated the performance and explored the capability of phase preservation of our PPNN model and visualized the extracted features. The experimental results indicate the superiority of the proposed PPNN when compared with previous methods, suggesting the PPNN is a robust model for EEG classification in RSVP task.

We describe a method, using linear discrimination, for detecting single-trial EEG signatures of object recognition events in a rapid serial visual presentation (RSVP) task. We record EEG using a high spatial density array (87 electrodes) during the rapid presentation (50-200 msec per image) of natural images. Subjects were instructed to release a button when they recognized a target image (an image with a person/people). Trials consisted of 100 images each, with a 50% chance of a single target being in a trial. Subject EEG was analyzed on a single-trial basis with an optimal spatial linear discriminator learned at multiple time windows after the presentation of an image. Linear discrimination enables the estimation of a forward model and thus allows for an approximate localization of the discriminating activity. Results show multiple loci for discriminating activity (e.g. motor and visual). Using these detected EEG signatures, we show that in many cases we can detect targets more accurately than the overt response (button release) and that such signatures can be used to prioritize images for high-throughput search.

Evidence for adaptive event integration has previously been provided using the Rapid Serial Visual Presentation (RSVP) task. However, it is not straightforward to generalize this finding to other types of tasks that measure temporal integration, because integration in such tasks is known to vary, depending on the method that is used. This variability has been seen as an indication that integration may result from more than a single type of perceptual persistence, and that different integration tasks may not tap into same type of persistence. Therefore, we investigated whether adaptive control of integration in the RSVP task can be replicated using another technique for measuring temporal integration, which may rely more on low-level mechanisms, namely the dot-array integration or Missing Element Task (MET). As in the RSVP studies, stimulus speed expectancy was presently manipulated. The results indicated that integration performance in the MET was not subject to adaptive control. We argue that this discrepancy with previous RSVP studies can most likely be attributed to a specific difference in the type of persistence underlying task performance. Temporal integration in the MET might rely mostly on visible persistence, while for the RSVP task integration relies more on informational persistence. The present findings suggest that, contrary to informational persistence, visible persistence may not be susceptible to adaptive control.

Brain-computer interface (BCI) based on rapid serial visual presentation (RSVP) is an efficient information detection technology by detecting event related brain response evoked by target stimuli. In the protocol design of the RSVP-BCI a range of parameters could influence the task difficulty, which may result in the changes of mental workload for subjects. This paper focused on the presentation rate in the RSVP paradigm aiming to investigate its influence on mental workload, and the separability of brain states during RSVP tasks with different setup of presentation rate. 64-channel Electroencephalographic (EEG) data were recorded during RSVP tasks with three levels of presentation rate in ten healthy subjects. The results show that different presentation rates indeed contribute to significant differences on mental workload revealed by one-way repeated measures analysis of variance (ANOVA) on z-scored RSME. Higher presentation rate results in the significant decrease on both behavioral and single-trial recognition performance of target images. Classification results on three levels of mental workload show that the mean accuracy reaches 65.5% and the highest accuracy reaches 88.3%. This work implies that mental workload induced by different presentation rates during RSVP tasks could be accurately recognized, and provides a possible method to monitor the mental workload in the application areas of RSVP-BCI.

Collaborative brain-computer interfaces (cBCIs) have demonstrated significant improvements in single-trial electroencephalogram (EEG) classification performance in rapid serial visual presentation (RSVP) tasks. However, it remains unclear how to effectively organize multiple collaborators into sub-groups to optimize system performance. This study introduces a novel sub-group partition strategy for RSVP-based cBCI systems. We first developed intra-individual and inter-individual neural response reproducibility (IINRR) as a metric to estimate subgroup capability in RSVP tasks. Based on this metric, we propose an IINRR-based partition strategy to optimize sub-group composition. Additionally, we introduce a metric called collaborative information processing rate (CIPR) to evaluate overall system performance. Our experiments verified the effectiveness of the proposed strategy on a public RSVP-based cBCI dataset. The results showed that our strategy consistently outperformed random partitioning in both within-session and cross-session scenarios, achieving higher classification performance and system efficiency. These findings suggest the strategy's potential for optimizing group mode in practical RSVP-based cBCI applications.

The extent to which letter-by-letter reading results from a specific orthographic deficit, as compared with a nonspecific disturbance in basic visuoperceptual mechanisms, is unclear. The current study directly compared processing of letters and digits in a letter-by-letter reader, G.M., using a rapid serial visual presentation (RSVP) task and a speeded matching task. Comparisons were made to a group of six brain-damaged individuals without reading deficits. In the RSVP task, G.M. had increased difficulty reporting the target identities when they were letters, as compared with digits. Although this general pattern was also evident in the control group, the magnitude of the letter-digit accuracy difference was greater in G.M. Similarly, in the matching task, G.M. was slower to match letters than digits, relative to the control group, although his response times to both item types were increased. These data suggest that letter-by-letter reading, at least in this case, results from a visuoperceptual encoding deficit that particularly affects letters, but also extends to processing of digits to a lesser extent. Results are consistent with the notion that a left occipitotemporal area is specialized for letter processing with greater bilaterality in the visual processing of digits.

Objective. Single-trial electroencephalography (EEG) classification is of great importance in the rapid serial visual presentation (RSVP) task. Convolutional neural networks (CNNs), as one of the mainstream deep learning methods, have been proven to be effective in extracting RSVP EEG features. However, most existing CNN models for EEG classification do not consider the phase-locked characteristic of event-related potential (ERP) components very well in the architecture design. Here, we propose a novel CNN model to make better use of the phase-locked characteristic to extract spatiotemporal features for single-trial RSVP EEG classification. Based on the phase-locked characteristic, the spatial distributions of the main ERP component in different periods can be learned separately. Approach. In this work, we propose a novel CNN model to achieve superior performance on single-trial RSVP EEG classification. We introduce the combination of the standard convolutional layer, the permute layer and the depthwise convolutional layer to separately operate the spatial convolution in different periods, which more fully utilizes the phase-locked characteristic of ERPs for classification. We compare our model with several traditional and deep-learning methods in the classification performance. Moreover, we use spatial topography and saliency map to visually analyze the ERP features extracted by our model. Main results. The results show that our model obtains better classification performance than those of reference methods. The spatial topographies of each subject exhibit the typical ERP spatial distribution in different time periods. And the saliency map of each subject illustrates the discriminant electrodes and the meaningful temporal features. Significance. Our model is designed with better consideration of the phase-locked ERP characteristic and reaches excellent performance on single-trial RSVP EEG classification.

: The current study evaluated the relationship between reaction time performance and the visual target-evoked P3 event-related potential (ERP) using a dynamic rapid serial visual presentation (RSVP) task. Electroencephalography (EEG) and reaction time measures were obtained while observers monitored an RSVP stream of short videos for infrequent targets presented among frequent nontarget distractors. Observers reaction time quartiles were used as ERP binning parameters for target- and response-locked EEG epochs. This procedure allowed us to assess neural activity related to perceptual and response processing across different levels of overt performance. P3 amplitude and latency were evaluated for both stimulus and response-locked averages. The results showed that the peak latency of the stimulus-locked P3 was maximal over central parietal electrode Pz and was significantly different between each quartile; however, these latency differences were absent when P3 latency was measured in the response-locked data. The P3 amplitude analysis revealed no significant differences between stimulus and response-locked averages within each quartile. Overall, the results suggest the peak latency of the P3 obtained in the current study reflects processes more associated with motor planning and response execution.

Fatigue has an impact on the target recognition accuracy in small target detection task based on the rapid serial visual presentation (RSVP) paradigm. It is necessary to evaluate brain fatigue objectively to adjust RSVP parameters, so as to prolong the effective task execution time. This paper proposes a method to evaluate the degree of brain fatigue based on reaction time and Oxyhemoglobin (HBO) index of functional near-infrared spectroscopy (fNIRS). A 100-min RSVP task was designed to induce the fatigue state. The fNIRS data of 17 channels in the prefrontal cortex (PFC) region and 440 groups of auditory key reaction time data of 11 subjects were collected and analyzed. The degree of brain fatigue can be divided into conscious state and fatigue state by combining the subjective evaluation scale, probability density distribution and cumulative distribution of reaction time, and the concentration change of HBO. The results show that, in the RSVP task, with the increase of brain fatigue, the reaction time of human gradually increases and the overall distribution is lognormal, the reaction time of HBO reduces and the concentration increases in PFC at the same time. Therefore, the reaction time index and fNIRS index can be used as potential indicators to evaluate the brain fatigue model in RSVP small target detection task, and provide the basis for the later research on the monitoring and regulation of brain fatigue.

Event-related potentials (ERPs) are often used in brain-computer interfaces (BCIs) for communication or system control for enhancing or regaining control for motor-disabled persons. Especially results from single-trial EEG classification approaches for BCIs support correlations between single-trial ERP detection performance and ERP expression. Hence, BCIs can be considered as a paradigm shift contributing to new methods with strong influence on both neuroscience and clinical applications. Here, we investigate the relevance of the choice of training data and classifier transfer for the interpretability of results from single-trial ERP detection. In our experiments, subjects performed a visual-motor oddball task with motor-task relevant infrequent (targets), motor-task irrelevant infrequent (deviants), and motor-task irrelevant frequent (standards) stimuli. Under dual-task condition, a secondary senso-motor task was performed, compared to the simple-task condition. For evaluation, average ERP analysis and single-trial detection analysis with different numbers of electrodes were performed. Further, classifier transfer was investigated between simple and dual task. Parietal positive ERPs evoked by target stimuli (but not by deviants) were expressed stronger under dual-task condition, which is discussed as an increase of task emphasis and brain processes involved in task coordination and change of task set. Highest classification performance was found for targets irrespective whether all 62, 6 or 2 parietal electrodes were used. Further, higher detection performance of targets compared to standards was achieved under dual-task compared to simple-task condition in case of training on data from 2 parietal electrodes corresponding to results of ERP average analysis. Classifier transfer between tasks improves classification performance in case that training took place on more varying examples (from dual task). In summary, we showed that P300 and overlaying parietal positive ERPs can successfully be detected while subjects are performing additional ongoing motor activity. This supports single-trial detection of ERPs evoked by target events to, e.g., infer a patient's attentional state during therapeutic intervention.

Restrained eaters (RE) are individuals that regularly show disinhibited food intake despite attempts of cognitive dietary restraint. This disinhibition is often attributed to an abundance of food cues in the environment that repeatedly draw attention, although the literature remains ambiguous. One source of ambiguity is the influence of different food cue modalities (pictures, words) on attentional processes. The current study therefore investigates the effect different modalities have on attentional bias to food cues in RE (n=70) compared to unrestrained eaters (URE; n=74). After a standardized lunch, participants completed a Rapid Serial Visual Presentation task with concurrent electroencephalography. Trials consisted of streams of pictures or words, with task-irrelevant food and animal distractors at lag 2 before an animal target appeared. Model comparisons indicated a significant Group×Distractor Type interaction. Specifically, RE were more accurate at identifying targets that were preceded by food compared to animal stimuli than URE. Exploratory analyses indicated that this was especially due to food pictures. This finding was not accompanied by changes in event related potentials. Furthermore, we found a significant Modality×Distractor Type interaction. In the picture, but not in the word condition, participants correctly identified targets preceded by a food distractor more often than when preceded by an animal distractor. This finding was accompanied by an increased P300 after food cues. In all, we showed that RE more effectively ignore food cues, especially in the image condition. This contributes to a body of research that demonstrates increased inhibitory control over food cues in RE. Future studies have to determine under which conditions inhibition of food cues is successful in RE.

Rapid Serial Visual Presentation (RSVP)-based Brain-Computer Interface (BCI) is an efficient information detection technology by detecting event-related brain responses evoked by target visual stimuli. However, a time-consuming calibration procedure is needed before a new user can use this system. Thus, it is important to reduce calibration efforts for BCI applications. In this article, we propose a multi-source conditional adversarial domain adaptation with the correlation metric learning (mCADA-C) framework that utilizes data from other subjects to reduce the data requirement from the new subject for training the model. This model utilizes adversarial training to enable a CNN-based feature extraction network to extract common features from different domains. A correlation metric learning (CML) loss is proposed to constrain the correlation of features based on class and domain to maximize the intra-class similarity and minimize inter-class similarity. Also, a multi-source framework with a source selection strategy is adopted to integrate the results of multiple domain adaptation. We constructed an RSVP-based dataset that includes 11 subjects each performing three RSVP experiments on three different days. The experimental results demonstrate that our proposed method can achieve 87.72% cross-subject balanced-accuracy under one block calibration. The results indicate our method can realize a higher performance with less calibration efforts.