- Research Article
521
- 10.1016/j.cub.2013.01.044
Action Video Games Make Dyslexic Children Read Better
- Feb 28, 2013
- Current Biology
- Sandro Franceschini + 5 more +5
Action Video Games Make Dyslexic Children Read Better
Visual attention has been successfully applied in structural prediction tasks such as visual captioning and question answering. Existing visual attention models are generally spatial, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image. However, we argue that such spatial attention does not necessarily conform to the attention mechanism --- a dynamic feature extractor that combines contextual fixations over time, as CNN features are naturally spatial, channel-wise and multi-layer. In this paper, we introduce a novel convolutional neural network dubbed SCA-CNN that incorporates Spatial and Channel-wise Attentions in a CNN. In the task of image captioning, SCA-CNN dynamically modulates the sentence generation context in multi-layer feature maps, encoding where (i.e., attentive spatial locations at multiple layers) and what (i.e., attentive channels) the visual attention is. We evaluate the proposed SCA-CNN architecture on three benchmark image captioning datasets: Flickr8K, Flickr30K, and MSCOCO. It is consistently observed that SCA-CNN significantly outperforms state-of-the-art visual attention-based image captioning methods.
Action Video Games Make Dyslexic Children Read Better
Action Video Games Make Dyslexic Children Read Better
Channel and spatial attention mechanism for fashion image captioning
<div class="page" title="Page 1"><div class="layoutArea"><div class="column"><p><span lang="EN-US">Image captioning aims to automatically generate one or more description sentences for a given input image. Most of the existing captioning methods use encoder-decoder model which mainly focus on recognizing and capturing the relationship between objects appearing in the input image. However, when generating captions for fashion images, it is important to not only describe the items and their relationships, but also mention attribute features of clothes (shape, texture, style, fabric, and more). In this study, one novel model is proposed for fashion image captioning task which can capture not only the items and their relationship, but also their attribute features. Two different attention mechanisms (spatial-attention and channel-wise attention) is incorporated to the traditional encoder-decoder model, which dynamically interprets the caption sentence in multi-layer feature map in addition to the depth dimension of the feature map. We evaluate our proposed architecture on Fashion-Gen using three different metrics (CIDEr, ROUGE-L, and BLEU-1), and achieve the scores of 89.7, 50.6 and 45.6, respectively. Based on experiments, our proposed method shows significant performance improvement for the task of fashion-image captioning, and outperforms other state-of-the-art image captioning methods.</span></p></div></div></div>
Read moreMulti attention module for visual tracking
Multi attention module for visual tracking
Context-Aware Visual Policy Network for Fine-Grained Image Captioning
With the maturity of visual detection techniques, we are more ambitious in describing visual content with open-vocabulary, fine-grained and free-form language, i.e., the task of image captioning. In particular, we are interested in generating longer, richer and more fine-grained sentences and paragraphs as image descriptions. Image captioning can be translated to the task of sequential language prediction given visual content, where the output sequence forms natural language description with plausible grammar. However, existing image captioning methods focus only on language policy while not visual policy, and thus fail to capture visual context that are crucial for compositional reasoning such as object relationships (e.g., "man riding horse") and visual comparisons (e.g., "small(er) cat"). This issue is especially severe when generating longer sequences such as a paragraph. To fill the gap, we propose a Context-Aware Visual Policy network (CAVP) for fine-grained image-to-language generation: image sentence captioning and image paragraph captioning. During captioning, CAVP explicitly considers the previous visual attentions as context, and decides whether the context is used for the current word/sentence generation given the current visual attention. Compared against traditional visual attention mechanism that only fixes a single visual region at each step, CAVP can attend to complex visual compositions over time. The whole image captioning model-CAVP and its subsequent language policy network-can be efficiently optimized end-to-end by using an actor-critic policy gradient method. We have demonstrated the effectiveness of CAVP by state-of-the-art performances on MS-COCO and Stanford captioning datasets, using various metrics and sensible visualizations of qualitative visual context.
Read moreFrom Pixels to Objects: Cubic Visual Attention for Visual Question Answering
Recently, attention-based Visual Question Answering (VQA) has achieved great success by utilizing question to selectively target different visual areas that are related to the answer. Existing visual attention models are generally planar, i.e., different channels of the last conv-layer feature map of an image share the same weight. This conflicts with the attention mechanism because CNN features are naturally spatial and channel-wise. Also, visual attention models are usually conducted on pixel-level, which may cause region discontinuous problem. In this paper we propose a Cubic Visual Attention (CVA) model by successfully applying a novel channel and spatial attention on object regions to improve VQA task. Specifically, instead of attending to pixels, we first take advantage of the object proposal networks to generate a set of object candidates and extract their associated conv features. Then, we utilize the question to guide channel attention and spatial attention calculation based on the con-layer feature map. Finally, the attended visual features and the question are combined to infer the answer. We assess the performance of our proposed CVA on three public image QA datasets, including COCO-QA, VQA and Visual7W. Experimental results show that our proposed method significantly outperforms the state-of-the-arts.
Read moreAttention Mechanism for Fashion Image Captioning
Image captioning tries to make computer learn to understand the visual content of a given image and produce one or more description sentences. Recently, with the help from rapid development of deep learning, image captioning has become an active research. The encoder-decoder architecture is commonly considered as the baseline method to address image captioning. This model focuses on recognizing objects and the relationship among these objects. However, with fashion images, the generated sentence should not only describe items inside the input image, but also mention item attributes such as texture, fabric, shape, style, etc. This requirement of fashion image captioning is not able to be solved by the image-captioning method based on the traditional encoder-decoder architecture. Our study addresses this issue by proposing an image captioning model based on the attention mechanism for fashion image which is able to cover both items and the relationship among the detailed attributes of items. We introduce an efficient framework for fashion image captioning that incorporates spatial attention inside the traditional encoder-decoder architecture. Our model generates fashion image captions using the spatial attention mechanism, which dynamically modulates the sentence generation context in multi-layer feature maps. Experiments were conducted on Fashion-Gen, one of state-of-the-art fashion image dataset, and achieved CIDEr/ROUGE-L/BLEU-4 scores of 0.913, 0.502, and 0.221, respectively. Based on experiments, it is consistently presented that our proposed model has significantly improved the performance of fashion-image captioning task, and it even surpasses the baseline methods on the same fashion benchmark dataset.
Read moreKnowing What it is: Semantic-Enhanced Dual Attention Transformer
Attention has become an indispensable component of the models of various multimedia tasks like Image Captioning (IC) and Visual Question Answering (VQA). However, most existing attention modules are designed for capturing the spatial dependency, and are still insufficient in semantic understanding, e.g.}, the categories of objects and their attributes, which is also critical for image captioning. To compensate for this defect, we propose a novel attention module termed Channel-wise Attention Block (CAB) to model channel-wise dependency for both visual modality and linguistic modality, thereby improving semantic learning and multi-modal reasoning simultaneously. Specifically, CAB has two novel designs to tickle with the high overhead of channel-wise attention, which are the reduction-reconstruction block structure and the gating-based attention prediction. Based on CAB, we further propose a novel Semantic-enhanced Dual Attention Transformer (termed SDATR), which combines the merits of spatial and channel-wise attentions. To validate SDATR, we conduct extensive experiments on the MS COCO dataset and yield new state-of-the-art performance of 134.5 CIDEr score on COCO Karpathy test split and 136.0 CIDEr score on the official online testing server. To examine the generalization of SDATR, we also apply it to the task of visual question answering, where superior performance gains are also witnessed.
Read moreHierarchical LSTMs with Adaptive Attention for Visual Captioning.
Recent progress has been made in using attention based encoder-decoder framework for image and video captioning. Most existing decoders apply the attention mechanism to every generated word including both visual words (e.g., "gun" and "shooting") and non-visual words (e.g., "the", "a"). However, these non-visual words can be easily predicted using natural language model without considering visual signals or attention. Imposing attention mechanism on non-visual words could mislead and decrease the overall performance of visual captioning. Furthermore, the hierarchy of LSTMs enables more complex representation of visual data, capturing information at different scales. Considering these issues, we propose a hierarchical LSTM with adaptive attention (hLSTMat) approach for image and video captioning. Specifically, the proposed framework utilizes the spatial or temporal attention for selecting specific regions or frames to predict the related words, while the adaptive attention is for deciding whether to depend on the visual information or the language context information. Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and high-level language context information to support the caption generation. We design the hLSTMat model as a general framework, and we first instantiate it for the task of video captioning. Then, we further instantiate our hLSTMarefine it and apply it to the imioning task. To demonstrate the effectiveness of our proposed framework, we test our method on both video and image captioning tasks. Experimental results show that our approach achieves the state-of-the-art performance for most of the evaluation metrics on both tasks. The effect of important components is also well exploited in the ablation study.
Read moreA Survey on Image Captioning datasets and Evaluation Metrics
In the task of image captioning, a natural language explanation is generated for a given image. It uses the subfields of artificial intelligence: computer vision and language generation. Convolutional Neural Network (CNN) is generally applied to capture image features and language processing models such as Recurrent Neural Network for sentence generation. In this paper, various datasets and evaluation metrics which are useful for image captioning task are discussed. Also, the datasets and evaluation metrics applied by the state-of-the-art image captioning models is summarized.
Read moreBengali Image Captioning with Visual Attention
Attention based approaches has been manifested to be an effective method in image captioning. However, attention can be used on text called semantic attention or on image which in known as spatial attention. We chose to implement the later as the main problem of image captioning is not being able to detect objects in image properly. In this work, we develop an approach which extracts features from images using two different convolutional neural network and combines the features with an attention model in order to generate caption with an RNN. We adapted Xception and InceptionV3 as our CNN and GRU as our RNN. Moreover, we Evaluated our proposed model on Flickr8k dataset translated into Bengali. So that captions can be generated in Bengali using visual attention.
Read moreAttention systems and neural responses to visual and auditory stimuli: an fMRI study
Visual orienting attention has been researched thoroughly using visual cues, and spatial attention and temporal attention have been compared using brain-imaging data [1]. However, while several studies have reported on auditory attention, the attention was not oriented using a cue. In addition, fewer studies have compared spatial and temporal attention using auditory stimuli. We compared visual and auditory orienting attention using a visual cue. We asked whether a general system exists for auditory spatial and temporal attention that conforms to that of visual spatial and temporal attention. We measured brain activity in 16 healthy volunteers using functional magnetic resonance imaging (fMRI). The attention was manipulated spatially and temporally using a visual cue for both visual and auditory tasks in a factorial design. Symbolic central cues oriented the subjects toward spatial location only (left or right), toward the temporal interval only (300 or 1800 ms), or gave no information about the spatial location or temporal interval. The subjects were also scanned under a resting baseline condition during which they clicked the reaction key ten times. Behavioral data showed the same tendency in visual and auditory tasks. The reaction time for spatial location attention was faster than that for temporal interval attention. The brain-imaging data showed a partial overlap between visual spatial attention and auditory spatial attention. In addition, it revealed a partial overlap between the neural systems involved in the performance of visual spatial attention versus visual temporal attention. The parietal and frontal cortices were activated in both the visual and auditory spatial attention tasks. The parietal cortex was activated in the visual temporal attention task only and not in the auditory temporal task.
Read moreWeakly Labelled Audio Tagging Via Convolutional Networks with Spatial and Channel-Wise Attention
Multiple instance learning (MIL) with convolutional neural networks (CNNs) has been proposed recently for weakly labelled audio tagging. However, features from the various CNN filtering channels and spatial regions are often treated equally, which may limit its performance in event prediction. In this paper, we propose a novel attention mechanism, namely, spatial and channel-wise attention (SCA). For spatial attention, we divide it into global and local submodules with the former to capture the event-related spatial regions and the latter to estimate the onset and offset of the events. Considering the variations in CNN channels, channel-wise attention is also exploited to recognize different sound scenes. The proposed SCA can be employed into any CNNs seamlessly with affordable overheads and is end-to-end trainable fashion. Extensive experiments on weakly labelled dataset Audioset show that the proposed SCA with CNNs achieves a state-of-the-art mean average precision (mAP) of 0.390.
Read moreMEDA: Multi-output Encoder-Decoder for Spatial Attention in Convolutional Neural Networks
Utilizing channel-wise spatial attention mechanisms to emphasize special parts of an input image is an effective method to improve the performance of convolutional neural networks (CNNs). There are multiple effective implementations of attention mechanism. One is adding squeeze-and-excitation (SE) blocks to the CNN structure that selectively emphasize the most informative channels and suppress the relatively less informative channels by taking advantage of channel dependence. Another method is adding convolutional block attention module (CBAM) to implement both channel-wise and spatial attention mechanisms to select important pixels of the feature maps while emphasizing informative channels. In this paper, we propose an encoder-decoder architecture based on the idea of letting the channel-wise and spatial attention blocks share the same latent space representation. Instead of separating the channel-wise and spatial attention modules into two independent parts in CBAM, we combine them into one encoder-decoder architecture with two outputs. To evaluate the performance of the proposed algorithm, we apply it to different CNN architectures and test it on image classification and semantic segmentation. Through comparing the resulting structure equipped with MEDA blocks against other attention module, we show that the proposed method achieves better performance across different test scenarios.
Read moreStructure Preserving Convolutional Attention for Image Captioning
In the task of image captioning, learning the attentive image regions is necessary to adaptively and precisely focus on the object semantics relevant to each decoded word. In this paper, we propose a convolutional attention module that can preserve the spatial structure of the image by performing the convolution operation directly on the 2D feature maps. The proposed attention mechanism contains two components: convolutional spatial attention and cross-channel attention, aiming to determine the intended regions to describe the image along the spatial and channel dimensions, respectively. Both of the two attentions are calculated at each decoding step. In order to preserve the spatial structure, instead of operating on the vector representation of each image grid, the two attention components are both computed directly on the entire feature maps with convolution operations. Experiments on two large-scale datasets (MSCOCO and Flickr30K) demonstrate the outstanding performance of our proposed method.
Read moreDecision letter: Distinct roles of forward and backward alpha-band waves in spatial visual attention
Article Figures and data Abstract Editor's evaluation Introduction Results Discussion Methods Data availability References Decision letter Author response Article and author information Metrics Abstract Previous research has associated alpha-band [8–12 Hz] oscillations with inhibitory functions: for instance, several studies showed that visual attention increases alpha-band power in the hemisphere ipsilateral to the attended location. However, other studies demonstrated that alpha oscillations positively correlate with visual perception, hinting at different processes underlying their dynamics. Here, using an approach based on traveling waves, we demonstrate that there are two functionally distinct alpha-band oscillations propagating in different directions. We analyzed EEG recordings from three datasets of human participants performing a covert visual attention task (one new dataset with N = 16, two previously published datasets with N = 16 and N = 31). Participants were instructed to detect a brief target by covertly attending to the screen’s left or right side. Our analysis reveals two distinct processes: allocating attention to one hemifield increases top-down alpha-band waves propagating from frontal to occipital regions ipsilateral to the attended location, both with and without visual stimulation. These top-down oscillatory waves correlate positively with alpha-band power in frontal and occipital regions. Yet, different alpha-band waves propagate from occipital to frontal regions and contralateral to the attended location. Crucially, these forward waves were present only during visual stimulation, suggesting a separate mechanism related to visual processing. Together, these results reveal two distinct processes reflected by different propagation directions, demonstrating the importance of considering oscillations as traveling waves when characterizing their functional role. Editor's evaluation Alamia and colleagues investigate the direction of traveling waves in the α frequency band during visual spatial attention. The authors' novel perspective adopted here is important to understand the functional relevance of α oscillations for spatial attention. The observed pattern of results is consistent with distinct roles for travelling α waves in spatially opposite directions and makes a solid case for considering this new perspective on α rhythms in human cognitive function. https://doi.org/10.7554/eLife.85035.sa0 Decision letter Reviews on Sciety eLife's review process Introduction Brain oscillations are related to several cognitive functions, as they orchestrate neuronal activity at distinct temporal and spatial scales (Buzsáki and Draguhn, 2004; Buzsáki, 2009). Alpha-band oscillations [8–12 Hz] are the most prevailing rhythms in electrophysiological (EEG) recordings, spreading through most cortical regions. Several studies investigated their functional role in various cognitive processes (Palva and Palva, 2007; Palva and Palva, 2011), providing mixed results. On the one hand, some studies showed that alpha-band oscillations might filter sensory information, regulating excitation and inhibition of sensory-specific brain regions (Jensen and Mazaheri, 2010; Mathewson et al., 2011; Klimesch, 2012; Sadaghiani and Kleinschmidt, 2016). Accordingly, researchers interpreted alpha oscillations as a top-down mechanism involved in inhibitory control and timing of cortical processing (Klimesch et al., 2007), as well as modulating cortical excitability (Jensen and Mazaheri, 2010; Mathewson et al., 2011). Experimental studies corroborated this hypothesis, demonstrating how the phase of alpha-band oscillation affects visual perception (Busch et al., 2009; Fakche et al., 2022; but see also Ruzzoli et al., 2019). Another highly replicated result regarding the inhibitory role of alpha oscillations consists in the hemispheric modulation in occipital regions associated with visual attention, having an increase of power ipsilateral to the attended hemifield, and a corresponding decrease contralaterally (Worden et al., 2000; Sauseng et al., 2005; Kelly et al., 2006; Thut et al., 2006; Händel et al., 2011). On the other hand, other experimental studies have related alpha-band oscillations in occipital and parietal regions to perceptual processing and visual memory (Bonnefond and Jensen, 2012; VanRullen, 2016; Pang et al., 2020; Luo et al., 2021). For example, reverse-correlation techniques reveal that the visual system reverberates sensory information in the alpha-band for as long as 1 s, in what has been dubbed ‘perceptual echoes’ (VanRullen and Macdonald, 2012). Importantly, these echoes are a clear signature of sensory processing as they reflect the input’s precise time course, are modulated by attention and have been dissociated from inhibitory alpha power modulation (VanRullen and Macdonald, 2012; VanRullen, 2016; Brüers and VanRullen, 2018; Schwenk et al., 2020). Altogether, these experimental evidences support distinct and contradictory conclusions about alpha-band oscillation’s functional role(s), which remains an open debate. Here, we address this question from a different perspective that interprets alpha-band oscillations as traveling waves (Muller et al., 2018; Alamia and VanRullen, 2019), thus considering their spatial component, and their propagation direction. Considering the case of visual attention, we tested the hypothesis that two functionally distinct alpha-band oscillations propagate along the frontal–occipital line in opposite directions. This compelling hypothesis about the different functional roles of alpha-band traveling waves derives from our previous studies (Alamia and VanRullen, 2019; Pang et al., 2020), in which we showed how visual perception modulates alpha waves, that is forward waves during visual stimulation, backward waves when the stimulus was off. In addition, this hypothesis is in line with previous studies suggesting that distinct alpha-band oscillations are related to specific cognitive processes (Gulbinaite et al., 2017; Deng et al., 2019; Schuhmann et al., 2019; Sokoliuk et al., 2019; Kasten et al., 2020). In this study, we analyzed three datasets, two publicly available (Foster et al., 2017; Feldmann-Wüstefeld and Vogel, 2019, see below), and one collected specifically for this study. In all datasets, participants attended either to the left or the right hemifield, while keeping central fixation. Our results confirmed the hemispheric modulation of alpha-band oscillations in posterior regions (Worden et al., 2000; Sauseng et al., 2005; Kelly et al., 2006; Thut et al., 2006; Händel et al., 2011) and revealed two distinct alpha-band traveling waves propagating in opposite directions. First, visual attention increases top-down alpha-band waves propagating from frontal to occipital regions ipsilateral to the attended location, and such waves correlate positively with alpha power in frontal and occipital regions. Moreover, our analysis demonstrates that visual attention also modulates contralateral forward waves, that is, waves propagating from occipital to frontal areas. Importantly, the attentional modulation of forward waves is crucially dependent on sustained sensory processing, as this modulation disappears in the absence of visual stimulation. In contrast, alpha-band top-down waves are present and modulated by visual attention irrespective of the presence or absence of concurrent sensory stimulation. These results demonstrate two distinct alpha-band oscillatory waves propagating in opposite directions, seemingly underlying different cognitive processes. The well-known lateralization effect observed in alpha-band can be interpreted as top-down traveling waves, and it is most likely related to inhibitory processes, in line with previous studies (Jensen and Mazaheri, 2010; Händel et al., 2011). However, different alpha-band oscillations propagate in a forward direction and are directly related to sensory processing, reconciling previous evidence linking alpha-band oscillations with visual processing (VanRullen and Macdonald, 2012; Lozano-Soldevilla and VanRullen, 2019). Results Traveling waves’ spectral profile The goal of the study was to investigate how visual attention modulates alpha-band traveling waves in the hemisphere contra- and ipsilateral to the attended location. To test this, we considered 11 lines of electrodes running from occipital to frontal regions (Figure 1C), 5 for each hemisphere and 1 midline. It is important to note that the spatial resolution of these lines is not critical for our analysis, as we do not expect significant differences within each hemisphere. However, before testing how visual attention modulates traveling waves, we explored the amount of waves propagating forward (FW) and backward (BW) as a function of their temporal frequency (see Figure 5 and methods for a detailed description of the analysis). Figure 1D shows the spectral profile of FW and BW waves in the midline (along the Oz–Fz axis) and the contra- and ipsilateral lines: confirming previous experimental studies (Alamia and VanRullen, 2019; Pang et al., 2020), we found that alpha-band oscillatory waves propagate in both directions during visual stimulation, whereas theta (4–7 Hz) and high-beta/gamma (24–45 Hz) bands propagate mostly bottom-up from occipital to frontal regions, and low-beta (13–23 Hz) waves flow in the top-down direction. Interestingly, this pattern of results confirms previous studies using different methods, in which higher frequency bands (i.e., high-beta/gamma) have been associated with forward processing, whereas low-beta and alpha frequencies have been related to top-down processing (Bastos et al., 2012; Bastos et al., 2015; van Kerkoerle et al., 2014; Michalareas et al., 2016; but see also Schneider et al., 2021b). Figure 1 Download asset Open asset Experimental design and waves’ spectral profile. (A) Each trial lasted 5 s, in which two flickering stimuli were presented to both hemifield. Participants were instructed to attend either the left or the right hemifield, as indicated by a central cue. In some trials, a target or a distractor appeared for 100 ms as a square either in the attended or unattended location. (B) The target and distractor luminance changed over trials due to the QUEST algorithm, which kept participants’ performance around 80%. (C) We quantified traveling waves along 11 electrodes lines, running along the anterior–posterior axis. These lines were located in the contralateral or the ipsilateral hemisphere to the attended location. (D) The amount of waves in dB computed for forward (in blue) and backward (in red) waves in the midline (central subplots, thinner lines represent standard errors of the mean) and in the ipsi- and contralateral hemisphere (left and right panels, respectively). These waves were computed on trials without target or distractors. Positive (negative) values reflect more (less) waves than the chance level (as quantified by the surrogate distribution), whereas values around 0 indicate no difference between the real and the null distribution. (E) Simulated data providing a schematic representation of forward and backward waves in the time domain in a given line of electrodes (from more frontal E1 to more occipital E7). A positive and a negative phase shift characterized forward and backward waves, respectively. Attending to visual stimuli modulates traveling waves In this analysis, we investigated how covert visual attention influences the traveling wave pattern. We focused on trials where neither a target nor a distractor was presented. First, we quantified the amount of traveling waves in the contra- and ipsilateral hemispheres to the attentional allocation. As shown in Figure 2 (left column), we found a strong lateralization effect revealing an increase (respectively, decrease) of contralateral (ipsilateral) forward waves in the alpha-band, and the opposite pattern in waves propagating in the opposite direction. These results were confirmed by a Bayesian analysis of variance (ANOVA), considering as factors DIRECTION (FW or BW), LINES (distance from the midline), and LATERALITY (contra vs. ipsi). The results revealed strong evidence in favor of the interaction between DIRECTION and LATERALITY factors (BF10 = 31.230, estimated error ~1%, η2 = 0.08 as estimated from a classical ANOVA), whereas all other factors and their interactions revealed evidence in favor of the absence of an effect (BFs10 < 0.3). We also found no significant effect in the other frequency bands (as shown in Figure 1D, namely theta, low, and high beta), hence we focused the following analyses on alpha-band oscillatory waves. These results demonstrate that the direction of alpha-band oscillatory traveling waves shows a laterality effect during a task involving covert selective attention. Figure 2 Download asset Open asset Traveling waves block analysis. Each column in the figure represents a different EEG dataset involving experiments with visual stimulation (left and middle columns) and without visual stimulation (right column). In the upper panels, the net amount of forward (blue) and backward (red) waves is represented along different lines of electrodes, normalized to the midline. The left and central panels reveal an increase (decrease) of forward (backward) waves contralateral to the attended location when participants attended to visual stimulation. The right column shows that when participants attended an empty screen (data from Foster et al., 2017), only backward waves were modulated by visual attention, and no effect was observed in the forward waves without visual stimulation. Error bars represent standard errors of the mean. The middle row shows schematic representations of the screen during the tasks: the central panel illustrates the task from Feldmann-Wüstefeld and Vogel, 2019, where D and T stand for Distractor and Target, respectively. In the task from Foster et al., 2017, the screen was empty, as the eight circles were not displayed during the task but here illustrate the stimulus positions (Foster et al., 2017). The lower panels represent the lines of electrodes in all datasets. Backward waves correlate with alpha-band power Previous studies investigating the role of alpha-band oscillations in visual attention reported a lateralization effect in the spectral power of alpha-band oscillations (Worden et al., 2000; Sauseng et al., 2005; Kelly et al., 2006; Thut et al., 2006; Jensen and Mazaheri, 2010; Händel et al., 2011). One may then wonder about the relationship between alpha power and traveling waves. To address this question, we investigated whether the oscillatory activity we observe propagating through the cortex relates to the spectral power in either occipital or frontal regions. We computed the averaged alpha-band power in frontal and occipital areas, contra- and ipsilaterally to the target presentation, considering the same electrodes used for quantifying the traveling waves (see Figure 3A). Interestingly, we found a significant positive correlation between alpha-band power in both occipital and frontal regions and backward waves, but not with forward waves (Figure 3A and Table 1). Next, we considered the lateralization effect in the alpha-band, as shown in Figure 3B (topographic plot in the right panel) and well replicated in previous studies (Sauseng et al., 2005; Thut et al., 2006; Händel et al., 2011). We wondered whether we could observe a correlation between such lateralization, defined as the difference between alpha-band power when attention is allocated to one side of the screen and to the other side, and the effect we reported in the traveling waves (Figure 2). As shown in Figure 3B (left panels), our results demonstrate a lack of correlation for both backward and forward waves in both frontal and occipital regions (all |r| < 0.1, BF10~ = 0.3). Table 1 Correlation with alpha-band power. The table reports the Pearson’s correlation coefficient and the Bayes Factor (BF10 supporting the alternative hypothesis, that is the presence of a correlation) between frontal and occipital electrodes and forward (FW) and backward (BW) waves, in both contra- and ipsilateral hemispheres. Values in bold reflect Bayes Factors providing strong evidence in favor of the alternative hypothesis. All correlations were computed on trials when neither a target nor a distractor was displayed. Pearsonr (BF₁₀)FWBWCONTRAIPSICONTRAIPSI OCC.CONTRA−0.297 (0.549)−0.350 (0.697)0.720 (28.519)0.698 (19.503)IPSI−0.305 (0.566)−0.342 (0.669)0.786 (116.990)0.746 (47.512) FRONT.CONTRA−0.222 (0.422)−0.252 (0.465)0.772 (84.225)0.712 (24.645)IPSI−0.327 (0.625)−0.354 (0.710)0.747 (48.448)0.705 (21.841) Figure 3 Download asset Open asset Correlation with alpha-band power. (A) Panel A reveals a correlation between backward waves and alpha power (static, standing power, i.e., measured via wavelets transform), in both frontal and occipital areas, in both hemispheres. We did not observe such correlation with forward waves. The plot to the right reveals the topographic distribution of alpha power when participants attended to the right hemifield (we included the ‘left’ condition by flipping the electrodes symmetrically to the midline). The white dots indicate the electrodes used for the correlation. (B) The plots to the left show the correlation between the laterality effect in the alpha power and in the waves (laterality measured as the mean difference between contra- and ipsilateral hemispheres for both alpha power and the waves – for the waves we computed the difference using lines of electrodes symmetrical to the midline). We did not observe any correlation in neither forward nor backward waves, with neither frontal nor occipital alpha power. The topography to the right reveals a lateralization effect in the alpha power (attention to the left minus attention to the right), confirming the presence of alpha power lateralization, in line with previous studies (Sauseng et al., 2005; Thut et al., 2006; Händel et al., 2011). (C) Panel C shows the trial-by-trial correlation coefficients averaged across participants for different conditions (as indicated in the x-axis). Confirming the results in panel A, we found a positive correlation across participants between backward waves and alpha power, specifically in the contralateral hemisphere. We also observed a positive global effect of the laterality condition across participants in the forward waves, even though the combined p values for the trial-by-trial correlation did not reach the significant threshold. Error bars represent standard errors of the mean. To further investigate the relation between alpha-band traveling waves and alpha power, we performed the same analysis focusing on the correlation within each participant. In particular, we correlated trial-by-trial forward and backward waves with alpha-band power for each subject, obtaining correlation coefficients ‘r’ and their respective p values. As in the previous analysis, we correlated forward and backward waves with frontal and occipital electrodes in both contro- and ipsilateral hemispheres. We applied the Fisher method (Fisher, 1992, see Methods for details) to combine all subjects’ p values in every conditions. Overall, we found a significant effect of all combined p values (p < 0.0001), except in the lateralization condition (contra- minus ipsilateral hemisphere), similar to our previous analysis. Additionally, we tested for a consistent positive or negative distribution of the correlation coefficients. As shown in Figure 3C, the results support a significant correlation between backward waves and alpha power in the hemisphere contralateral to the attended location (BF10 = 10.7 and BF10 = 7.4 for occipital and frontal regions, respectively; all other BF10 were between 1 and 2, providing inconclusive evidence). Interestingly, this analysis also revealed a small but consistent effect in the correlation between lateralization effects, as we reported a consistently positive correlation in the contra- minus ipsilateral difference between forward waves and alpha power (BF10 ~ 5 for both frontal and occipital electrodes). However, it is important to notice that the combined p values obtained using the Fisher method did not reach the significance threshold in the lateralization condition, reducing the relevance of this specific result. Covert attention modulates forward waves only with visual stimuli To confirm our previous results, we replicated the same traveling waves analysis on two publicly available EEG datasets in which participants performed similar attentional tasks (experiment 1 of Foster et al., 2017 and experiment 1 of Feldmann-Wüstefeld and Vogel, 2019). In the first experiment from the Feldmann-Wüstefeld and Vogel dataset, participants were instructed to perform a visual working memory task in which, while keeping a central fixation, they had to memorize a set of items while ignoring a group of distracting stimuli. We focused our analysis on those trials in which the visual items to remember were placed either to the right or the left side of the screen, while the distractors were either in the upper or lower part of the screen (we pulled together the trials with either two or four distractors, as this factor was irrelevant for our analysis). The stimuli were shown for 200 ms, and we computed the amount of forward and backward waves in the 500 ms following stimulus onset. As shown in Figure 2 (central column), the analysis confirmed our previous results, demonstrating a strong interaction between the factors DIRECTION and LATERALITY (BF10 = 667, error ~2%; independently, the factors DIRECTION and LATERALITY had BF10 = 0.2 and BF10 = 0.4, respectively). These results confirmed that spatial attention modulates both forward and backward waves in the presence of visual stimulation. Next, we analyzed another publicly available dataset from Foster et al., 2017. In the first experiment of Foster’s study, participants completed a spatial cueing task, requiring them to identify a digit among distractor letters. After a central cue was displayed for 250 ms, participants attended one of eight locations for 1000 ms before the onset of the target and distractors. As in our design, participants allocated attention to different locations to the left or right of the screen while keeping central fixation. However, unlike in our and in Feldmann-Wüstefeld’s study, no stimulus was displayed while participants were attending one of the possible locations. Here, we assessed the amount of waves in the 1000 ms before the onset of the stimulus during attention allocation, when no visual stimuli were shown on the screen. Remarkably, as shown in Figure 2 (right panel), our analysis demonstrated an effect of the lateralization (LATERALITY: BF10 = 3.571, error ~1%), revealing more waves contralateral to the attended location, but inconclusive results regarding the interaction between DIRECTION and LATERALITY (BF10 = 2.056, error ~1%). However, using a classical ANOVA (i.e., without modeling the slope of the random terms), the interaction between DIRECTION and LATERALITY proved significant (F(1,16) = 9.81, p = 0.003, η2 = 0.13). In addition, when testing LATERALITY separately for forward and backward waves, we observed an effect in the backward waves (BF10 = 3.497, error <0.01%) but not in the forward waves (BF10 = 0.231, error <0.01%, supporting evidence in favor of the absence of an effect). In addition, as analyzed in our dataset, we tested the correlation between backward waves and alpha-band power in occipital (electrodes: PO3 and PO4) and frontal (electrodes: F3 and F4) regions. We found evidence of a positive correlation between contra- and ipsilateral backward waves, and occipital (all Pearson’s = 0.4, all = but inconclusive evidence in the frontal (all Pearson’s = all = 2). These results those from our dataset, having a amount of lines, and power (see Figure 2, lower All in we could confirm our previous that covert visual attention modulates top-down oscillatory waves, this effect even in the absence of visual stimulation. In addition, we that the lateralization effect we reported in the forward waves in our dataset in the Foster is related to the visual stimulation during the attentional allocation, in line with our previous results demonstrating that oscillatory bottom-up waves reflect sensory processing (Alamia and VanRullen, 2019; Pang et al., 2020). and distractors FW waves, but not In our previous analysis, based on a of trials in which neither a target nor a distractor we demonstrated that sustained attention to one hemifield oscillatory alpha-band waves propagating forward in the contralateral hemisphere and backward in the ipsilateral We whether the of a specific such as the onset of a target or a distractor could the of oscillatory waves. For this we replicated the same analysis on those trials either a target or a distractor each performed trials in each to the amount of waves to the onset of these The upper panels of Figure reveal the amount of forward and backward waves contralateral to the that the and distractors appeared in the attended and unattended respectively. A Bayesian ANOVA reveals no difference between and distractors BF10 = error ~1%), or their interaction BF10 = error as shown in the panel of Figure This result reveals that both target and distractor forward waves propagating contralateral to the hemifield where they Next, we investigated whether the waves in the hemisphere contralateral to the attended hemifield correlate with the performance in the target QUEST kept the the experiment around Remarkably, we found an effect the and as revealed by a significant interaction of the DIRECTION and factors BF10 = error as shown in the panel of Figure Interestingly, Figure reveals the amount of waves ms before and the onset of the how a target is related to a decrease in forward (backward) waves contralateral (ipsilateral) to the attended location, due to attentional during each Figure Download asset Open asset analysis. (A) The figure shows the amount of forward (in blue) and backward (in red) contralateral waves around the onset of the or and (right Error bars are standard error of the mean. We found an interaction effect when we analyzed the (B) The represent the amount of waves in the 11 lines of electrodes and around the onset time for forward and backward waves, and for and The opposite pattern for before the target that are due to a of attentional than sensory and that attentional is characterized by contralateral forward waves and ipsilateral backward waves. Discussion Previous studies demonstrated that selective attention modulates alpha-band oscillations in occipital and parietal regions (Worden et al., 2000; Sauseng et al., 2005; Kelly et al., 2006; Thut et al., 2006; Händel et al., 2011), their in inhibitory Here, we a novel perspective on these results by oscillations as traveling waves (Muller et al., thus considering their spatial on of the temporal Our results revealed two distinct alpha-band waves propagating in opposite attention modulates oscillations traveling from occipital to frontal regions only in the presence of visual stimulation, thus forward waves to visual processing and VanRullen, 2019; Pang et al., whereas oscillations propagating in the top-down direction were modulated by attention irrespective of the presence or absence of concurrent visual as in standard studies of alpha power lateralization (Worden et al., 2000; Sauseng et al., 2005; Kelly et al., 2006; Thut et al., 2006; Händel et al., 2011), this attentional modulation involved both an decrease of alpha waves contralateral to the attended location, and an ipsilateral In line with previous studies (Gulbinaite et al., 2017; Deng et al., 2019; Schuhmann et al., 2019; Sokoliuk et al., 2019; Kasten et al., 2020), our results support the that distinct alpha-band oscillations are involved in separate cognitive processes. A study from Sokoliuk et al., demonstrated two different of alpha-band oscillations during a located in visual areas, the of and with attention, whereas another one attentional and in parietal regions. et also demonstrated that but not occipital alpha-band oscillations are for the oscillatory
Read more