Multi-Task Collaborative Network: Bridge the Supervised and Self-Supervised Learning for EEG Classification in RSVP Tasks.

Objective. Brain–computer interface (BCI) systemsdirectly translate human intentions to instructions for machines by decoding the neural signals. The rapid serial visual presentation (RSVP) task is a typical paradigm of BCIs, in which subjects can detect the targets in the high-speed serial images. There are still two main challenges in electroencephalography (EEG) classification for RSVP tasks: inter-trial variability of event-related potentials (ERPs) and limited trial number of EEG training data.Approach.This study proposed an algorithm of discriminant analysis and classification for interval ERPs (DACIE) in RSVP tasks. Firstly, an interval model of ERPs was exploited to solve the inter-trial variability problem. Secondly, a spatial structured sparsity regularization was utilized to reinforce the important channels, which provided a spatial region of interest (sROI). Meanwhile, a temporal auto-weighting technique was conducted to emphasize the important discriminant components, which obtained a temporal regions of interest (tROIs). Thirdly, classification features were obtained by the discriminant eigenvalue analysis to avoid the ill-conditioned estimation of covariance matrix caused by fewer training trials.Main results.EEG datasets of 12 subjects in RSVP tasks were analyzed to evaluate the classification performance of proposed algorithm. The average accuracy rate, true positive rate, false positive rate and AUC value are 96.9%, 81.6%, 2.8% and 0.938, respectively. Compared with several state-of-the-art algorithms, the proposed algorithm can provide significantly better classification performance.Significance.The interval model of ERPs was exploited in a spatial linear discriminant framework to overcome the inter-trial variability. The sROIs and tROIs were explored to reinforce the pivotal channels and temporal components. And the proposed algorithm can provide good performance with fewer training trials.

Neuroscience studies have demonstrated the phase-locked characteristics of some early event-related potential (ERP) components evoked by stimuli. In this study, we propose a phase preservation neural network (PPNN) to learn phase information to improve the Electroencephalography (EEG) classification in a rapid serial visual presentation (RSVP) task. The PPNN consists of three major modules that can produce spatial and temporal representations with the high discriminative ability of the EEG features for classification. We first adopt a stack of dilated temporal convolution layers to extract temporal dynamics while avoiding the loss of phase information. Considering the intrinsic channel dependence of the EEG data, a spatial convolution layer is then applied to obtain the spatial-temporal representation of the input EEG signal. Finally, a fully connected layer is adopted to extract higher-level features for the final classification. The experiments are conducted on two public and one collected EEG datasets from the RSVP task, in which we evaluated the performance and explored the capability of phase preservation of our PPNN model and visualized the extracted features. The experimental results indicate the superiority of the proposed PPNN when compared with previous methods, suggesting the PPNN is a robust model for EEG classification in RSVP task.

Evidence for adaptive event integration has previously been provided using the Rapid Serial Visual Presentation (RSVP) task. However, it is not straightforward to generalize this finding to other types of tasks that measure temporal integration, because integration in such tasks is known to vary, depending on the method that is used. This variability has been seen as an indication that integration may result from more than a single type of perceptual persistence, and that different integration tasks may not tap into same type of persistence. Therefore, we investigated whether adaptive control of integration in the RSVP task can be replicated using another technique for measuring temporal integration, which may rely more on low-level mechanisms, namely the dot-array integration or Missing Element Task (MET). As in the RSVP studies, stimulus speed expectancy was presently manipulated. The results indicated that integration performance in the MET was not subject to adaptive control. We argue that this discrepancy with previous RSVP studies can most likely be attributed to a specific difference in the type of persistence underlying task performance. Temporal integration in the MET might rely mostly on visible persistence, while for the RSVP task integration relies more on informational persistence. The present findings suggest that, contrary to informational persistence, visible persistence may not be susceptible to adaptive control.

We describe a method, using linear discrimination, for detecting single-trial EEG signatures of object recognition events in a rapid serial visual presentation (RSVP) task. We record EEG using a high spatial density array (87 electrodes) during the rapid presentation (50-200 msec per image) of natural images. Subjects were instructed to release a button when they recognized a target image (an image with a person/people). Trials consisted of 100 images each, with a 50% chance of a single target being in a trial. Subject EEG was analyzed on a single-trial basis with an optimal spatial linear discriminator learned at multiple time windows after the presentation of an image. Linear discrimination enables the estimation of a forward model and thus allows for an approximate localization of the discriminating activity. Results show multiple loci for discriminating activity (e.g. motor and visual). Using these detected EEG signatures, we show that in many cases we can detect targets more accurately than the overt response (button release) and that such signatures can be used to prioritize images for high-throughput search.

The extent to which letter-by-letter reading results from a specific orthographic deficit, as compared with a nonspecific disturbance in basic visuoperceptual mechanisms, is unclear. The current study directly compared processing of letters and digits in a letter-by-letter reader, G.M., using a rapid serial visual presentation (RSVP) task and a speeded matching task. Comparisons were made to a group of six brain-damaged individuals without reading deficits. In the RSVP task, G.M. had increased difficulty reporting the target identities when they were letters, as compared with digits. Although this general pattern was also evident in the control group, the magnitude of the letter-digit accuracy difference was greater in G.M. Similarly, in the matching task, G.M. was slower to match letters than digits, relative to the control group, although his response times to both item types were increased. These data suggest that letter-by-letter reading, at least in this case, results from a visuoperceptual encoding deficit that particularly affects letters, but also extends to processing of digits to a lesser extent. Results are consistent with the notion that a left occipitotemporal area is specialized for letter processing with greater bilaterality in the visual processing of digits.

Collaborative brain-computer interfaces (cBCIs) have demonstrated significant improvements in single-trial electroencephalogram (EEG) classification performance in rapid serial visual presentation (RSVP) tasks. However, it remains unclear how to effectively organize multiple collaborators into sub-groups to optimize system performance. This study introduces a novel sub-group partition strategy for RSVP-based cBCI systems. We first developed intra-individual and inter-individual neural response reproducibility (IINRR) as a metric to estimate subgroup capability in RSVP tasks. Based on this metric, we propose an IINRR-based partition strategy to optimize sub-group composition. Additionally, we introduce a metric called collaborative information processing rate (CIPR) to evaluate overall system performance. Our experiments verified the effectiveness of the proposed strategy on a public RSVP-based cBCI dataset. The results showed that our strategy consistently outperformed random partitioning in both within-session and cross-session scenarios, achieving higher classification performance and system efficiency. These findings suggest the strategy's potential for optimizing group mode in practical RSVP-based cBCI applications.

In non-invasive brain-computer interface (BCI), the analysis of event-related potentials (ERP) has typically focused on averaged trials, a current trend is to analyze single-trial evoked response individually with new approaches in pattern recognition and signal processing. Such single trial detection requires a robust response that can be detected in a variety task conditions. Here, we investigated the influence of target probability, a key factor known to influence the amplitude of the evoked response, on single trial target classification in a difficult rapid serial visual presentation (RSVP) task. Our classification approach for detecting target vs. non target responses, considers spatial filters obtained through the maximization of the signal to signal-plus-noise ratio, and then uses the resulting information as inputs to a Bayesian discriminant analysis. The method is evaluated across eight healthy subjects, on four probability conditions (P=0.05, 0.10, 0.25, 0.50). We show that the target probability has a statistically significant effect on both the behavioral performance and the target detection. The best mean area under the ROC curve is achieved with P=0.10, AUC=0.82. These results suggest that optimal performance of ERP detection in RSVP tasks is critically dependent on target probability.

Brain-computer interface (BCI) based on rapid serial visual presentation (RSVP) is an efficient information detection technology by detecting event related brain response evoked by target stimuli. In the protocol design of the RSVP-BCI a range of parameters could influence the task difficulty, which may result in the changes of mental workload for subjects. This paper focused on the presentation rate in the RSVP paradigm aiming to investigate its influence on mental workload, and the separability of brain states during RSVP tasks with different setup of presentation rate. 64-channel Electroencephalographic (EEG) data were recorded during RSVP tasks with three levels of presentation rate in ten healthy subjects. The results show that different presentation rates indeed contribute to significant differences on mental workload revealed by one-way repeated measures analysis of variance (ANOVA) on z-scored RSME. Higher presentation rate results in the significant decrease on both behavioral and single-trial recognition performance of target images. Classification results on three levels of mental workload show that the mean accuracy reaches 65.5% and the highest accuracy reaches 88.3%. This work implies that mental workload induced by different presentation rates during RSVP tasks could be accurately recognized, and provides a possible method to monitor the mental workload in the application areas of RSVP-BCI.

Objective. Single-trial electroencephalography (EEG) classification is of great importance in the rapid serial visual presentation (RSVP) task. Convolutional neural networks (CNNs), as one of the mainstream deep learning methods, have been proven to be effective in extracting RSVP EEG features. However, most existing CNN models for EEG classification do not consider the phase-locked characteristic of event-related potential (ERP) components very well in the architecture design. Here, we propose a novel CNN model to make better use of the phase-locked characteristic to extract spatiotemporal features for single-trial RSVP EEG classification. Based on the phase-locked characteristic, the spatial distributions of the main ERP component in different periods can be learned separately. Approach. In this work, we propose a novel CNN model to achieve superior performance on single-trial RSVP EEG classification. We introduce the combination of the standard convolutional layer, the permute layer and the depthwise convolutional layer to separately operate the spatial convolution in different periods, which more fully utilizes the phase-locked characteristic of ERPs for classification. We compare our model with several traditional and deep-learning methods in the classification performance. Moreover, we use spatial topography and saliency map to visually analyze the ERP features extracted by our model. Main results. The results show that our model obtains better classification performance than those of reference methods. The spatial topographies of each subject exhibit the typical ERP spatial distribution in different time periods. And the saliency map of each subject illustrates the discriminant electrodes and the meaningful temporal features. Significance. Our model is designed with better consideration of the phase-locked ERP characteristic and reaches excellent performance on single-trial RSVP EEG classification.

Fatigue has an impact on the target recognition accuracy in small target detection task based on the rapid serial visual presentation (RSVP) paradigm. It is necessary to evaluate brain fatigue objectively to adjust RSVP parameters, so as to prolong the effective task execution time. This paper proposes a method to evaluate the degree of brain fatigue based on reaction time and Oxyhemoglobin (HBO) index of functional near-infrared spectroscopy (fNIRS). A 100-min RSVP task was designed to induce the fatigue state. The fNIRS data of 17 channels in the prefrontal cortex (PFC) region and 440 groups of auditory key reaction time data of 11 subjects were collected and analyzed. The degree of brain fatigue can be divided into conscious state and fatigue state by combining the subjective evaluation scale, probability density distribution and cumulative distribution of reaction time, and the concentration change of HBO. The results show that, in the RSVP task, with the increase of brain fatigue, the reaction time of human gradually increases and the overall distribution is lognormal, the reaction time of HBO reduces and the concentration increases in PFC at the same time. Therefore, the reaction time index and fNIRS index can be used as potential indicators to evaluate the brain fatigue model in RSVP small target detection task, and provide the basis for the later research on the monitoring and regulation of brain fatigue.

: The current study evaluated the relationship between reaction time performance and the visual target-evoked P3 event-related potential (ERP) using a dynamic rapid serial visual presentation (RSVP) task. Electroencephalography (EEG) and reaction time measures were obtained while observers monitored an RSVP stream of short videos for infrequent targets presented among frequent nontarget distractors. Observers reaction time quartiles were used as ERP binning parameters for target- and response-locked EEG epochs. This procedure allowed us to assess neural activity related to perceptual and response processing across different levels of overt performance. P3 amplitude and latency were evaluated for both stimulus and response-locked averages. The results showed that the peak latency of the stimulus-locked P3 was maximal over central parietal electrode Pz and was significantly different between each quartile; however, these latency differences were absent when P3 latency was measured in the response-locked data. The P3 amplitude analysis revealed no significant differences between stimulus and response-locked averages within each quartile. Overall, the results suggest the peak latency of the P3 obtained in the current study reflects processes more associated with motor planning and response execution.

Through rapid serial visual presentation (RSVP), we asked Ss to identify a partially specified letter (target) and then to detect the presence or absence of a fully specified letter (probe). Whereas targets are accurately identified, probes are poorly detected when they are presented during a 270-ms interval beginning 180 ms after the target. Probes presented immediately after the target or later in the RSVP stream are accurately detected. This temporary reduction in probe detection was not found in conditions in which a brief blank interval followed the target or Ss were not required to identify the target. The data suggest that the presentation of stimuli after the target but before target-identification processes are complete produces interference at a letter-recognition stage. This interference may cause the temporary suppression of visual attention mechanisms observed in the present study.

The perceptual load theory of attention posits that attentional selection occurs early in processing when a task is perceptually demanding but occurs late in processing otherwise. We used a frequency-tagged steady-state evoked potential paradigm to investigate the modality specificity of perceptual load-induced distractor filtering and the nature of neural-competitive interactions between task and distractor stimuli. EEG data were recorded while participants monitored a stream of stimuli occurring in rapid serial visual presentation (RSVP) for the appearance of previously assigned targets. Perceptual load was manipulated by assigning targets that were identifiable by color alone (low load) or by the conjunction of color and orientation (high load). The RSVP task was performed alone and in the presence of task-irrelevant visual and auditory distractors. The RSVP stimuli, visual distractors, and auditory distractors were "tagged" by modulating each at a unique frequency (2.5, 8.5, and 40.0 Hz, respectively), which allowed each to be analyzed separately in the frequency domain. We report three important findings regarding the neural mechanisms of perceptual load. First, we replicated previous findings of within-modality distractor filtering and demonstrated a reduction in visual distractor signals with high perceptual load. Second, auditory steady-state distractor signals were unaffected by manipulations of visual perceptual load, consistent with the idea that perceptual load-induced distractor filtering is modality specific. Third, analysis of task-related signals revealed that visual distractors competed with task stimuli for representation and that increased perceptual load appeared to resolve this competition in favor of the task stimulus.

Rapid Serial Visual Presentation (RSVP)-based Brain-Computer Interface (BCI) is an efficient information detection technology by detecting event-related brain responses evoked by target visual stimuli. However, a time-consuming calibration procedure is needed before a new user can use this system. Thus, it is important to reduce calibration efforts for BCI applications. In this article, we propose a multi-source conditional adversarial domain adaptation with the correlation metric learning (mCADA-C) framework that utilizes data from other subjects to reduce the data requirement from the new subject for training the model. This model utilizes adversarial training to enable a CNN-based feature extraction network to extract common features from different domains. A correlation metric learning (CML) loss is proposed to constrain the correlation of features based on class and domain to maximize the intra-class similarity and minimize inter-class similarity. Also, a multi-source framework with a source selection strategy is adopted to integrate the results of multiple domain adaptation. We constructed an RSVP-based dataset that includes 11 subjects each performing three RSVP experiments on three different days. The experimental results demonstrate that our proposed method can achieve 87.72% cross-subject balanced-accuracy under one block calibration. The results indicate our method can realize a higher performance with less calibration efforts.

The aim of this study was to investigate visual processing speeds in children. A rapid serial visual presentation (RSVP) task with schematic faces as stimuli was given to ninety-nine 6–10-year-old children as well as a short form of the WISC-III. Participants were asked to determine whether a happy face stimulus was embedded in a stream of distracter stimuli. Presentation time was gradually reduced from 500 ms per stimulus to 100 ms per stimulus, in 50 ms steps. The data revealed that (i) RSVP speed increases with age, (ii) children aged 8 years and over can discriminate stimuli presented every 100 ms—the speed typically used with RSVP procedures in adult and adolescent populations, and (iii) RSVP speed is significantly correlated with digit span and object assembly. In consequence, the RSVP paradigm presented here is appropriate for use in further investigations of processes of temporal attention within this cohort.