Convolutional neural networks uncover the dynamics of human visual memory representations over time.

How distinct are visual memory representations from visual perception? Although evidence suggests that briefly remembered stimuli are represented within early visual cortices, the degree to which these memory traces resemble true visual representations remains something of a mystery. Here, we tested whether both visual memory and perception succumb to a seemingly ubiquitous neural computation: normalization. Observers were asked to remember the contrast of visual stimuli, which were pitted against each other to promote normalization either in perception or in visual memory. Our results revealed robust normalization between visual representations in perception, yet no signature of normalization occurring between working memory stores—neither between representations in memory nor between memory representations and visual inputs. These results provide unique insight into the nature of visual memory representations, illustrating that visual memory representations follow a different set of computational rules, bypassing normalization, a canonical visual computation.

Our ability to actively maintain information in visual memory is strikingly limited. There is considerable debate about why this is so. As with many questions in psychology, the debate is framed dichotomously: Is visual working memory limited because it is supported by only a small handful of discrete "slots" into which visual representations are placed, or is it because there is an insufficient supply of a "resource" that is flexibly shared among visual representations? Here, we argue that this dichotomous framing obscures a set of at least eight underlying questions. Separately considering each question reveals a rich hypothesis space that will be useful for building a comprehensive model of visual working memory. The questions regard (1) an upper limit on the number of represented items, (2) the quantization of the memory commodity, (3) the relationship between how many items are stored and how well they are stored, (4) whether the number of stored items completely determines the fidelity of a representation (vs. fidelity being stochastic or variable), (5) the flexibility with which the memory commodity can be assigned or reassigned to items, (6) the format of the memory representation, (7) how working memories are formed, and (8) how memory representations are used to make responses in behavioral tasks. We reframe the debate in terms of these eight underlying questions, placing slot and resource models as poles in a more expansive theoretical space.

Recent research has found visual object memory can be stored as part of a larger scene representation rather than independently of scene context. The present study examined how spatial and nonspatial contextual information modulate visual object memory. Two experiments tested participants’ visual memory by using a change detection task in which a target object's orientation was either the same as it appeared during initial viewing or changed. In addition, we examined the effect of spatial and nonspatial contextual manipulations on change detection performance. The results revealed that visual object representations can be maintained reliably after viewing arrays of objects. Moreover, change detection performance was significantly higher when either spatial or nonspatial contextual information remained the same in the test image. We concluded that while processing complex visual stimuli such as object arrays, visual object memory can be stored as part of a comprehensive scene representation, and both spatial and nonspatial contextual changes modulate visual memory retrieval and comparison.

It is generally assumed that the encoding of a single event generates multiple memory representations, which contribute differently to subsequent episodic memory. We used functional magnetic resonance imaging (fMRI) and representational similarity analysis to examine how visual and semantic representations predicted subsequent memory for single item encoding (e.g., seeing an orange). Three levels of visual representations corresponding to early, middle, and late visual processing stages were based on a deep neural network. Three levels of semantic representations were based on normative observed ("is round"), taxonomic ("is a fruit"), and encyclopedic features ("is sweet"). We identified brain regions where each representation type predicted later perceptual memory, conceptual memory, or both (general memory). Participants encoded objects during fMRI, and then completed both a word-based conceptual and picture-based perceptual memory test. Visual representations predicted subsequent perceptual memory in visual cortices, but also facilitated conceptual and general memory in more anterior regions. Semantic representations, in turn, predicted perceptual memory in visual cortex, conceptual memory in the perirhinal and inferior prefrontal cortex, and general memory in the angular gyrus. These results suggest that the contribution of visual and semantic representations to subsequent memory effects depends on a complex interaction between representation, test type, and storage location.

In this article, we consider the problem of human head movement recognition (HMR) using linear frequency-modulated continuous-wave (LFMCW) radar. The multidomain fusion network is proposed to improve the HMR performance, which extracts and fuses the multidomain features from the range and time–frequency (TF) domain. Specifically, the 2-D convolutional neural network (2D-CNN) structure is applied to extract range domain features, and the 3-D convolutional neural network (3D-CNN) structure is designed to extract multitype TF representation (TFR) plots features cooperatively. After that, the learnable convolution weights are used for the adaptive fusion of multidomain features. In addition, the attention mechanism is employed to remove redundant information in the fused features. Finally, experimental results on real data verify the effectiveness of the proposed method.

In order to identify pests quickly and accurately in agricultural environment, a deep learning model based on convolutional neural network is proposed to locate and identify pests. In this paper, by accelerating the color attenuation of the whole image, combining with the spatial influence between the hyperpixel regions, the saliency value of each super region is calculated, and then the potential area of the pest target is provided. Finally, the pest target location and segmentation are combined with grabcut algorithm. For the segmented pest target, the optimized convolution neural network is used to express and classify, and the structure of convolution neural network is further reduced.

Visual search is assumed to be guided by an active visual working memory representation of what we are currently looking for. This attentional template can be dissociated from accessory memory representations that are only needed prospectively, for a future task, and that until then should be prevented from guiding attention. Little is known about how the brain sequentially prioritizes, and switches between memory representations for successive task goals. We measured EEG while human observers performed two consecutive working memory-guided visual search tasks. Prior to the first search task, a cue instructed observers which item to look for first (current template), and which second (prospective template). During the first delay, leading up to the first of the two searches, posterior alpha power (8-14 Hz) was more suppressed contralateral to the memory item. These lateralized alpha dynamics were stronger if the lateralized item was the imminent search target (current template), than when it was the subsequent search target (prospective template). To investigate the electrophysiological mechanisms underlying the switch in priority, on 40% of the trials an auditory cue replaced the first search, telling observers that at this point the first template could be dropped, in favor of now prioritizing the second, prospective template. Dropping the first template resulted in clear alpha enhancement, while turning the former prospective template into the current search target resulted in alpha suppression. Furthermore, this switch in posterior alpha lateralization was predicted by an increase in frontal delta/low theta (2–6 Hz) power. This increase in low-frequency power in frontal regions also predicted faster response times on the second search task. We thus obtained evidence for large-scale network interactions during the flexible "juggling" between priority states of multiple memory items in between search tasks. Meeting abstract presented at VSS 2018

Due to the irregular and sporadic nature of wildfires, continuous monitoring of large areas is required. Since geostationary satellite sensors can observe large areas with high temporal resolution, they are suitable for monitoring wildfires in real time. However, the threshold algorithm currently employed for satellite-based active fire detection has poor performance in sensors with low spatial resolution. In addition, the algorithm does not account for environmental conditions that affect wildfire detection, resulting in poor generalization performance for large areas. This study examines the viability of an adaptive active fire detection model by combining satellite and numerical model data with deep learning. A model for active fire detection was developed using commonly employed brightness temperature-related variables (key variables) and local environmental variables (sub variables). Key variables are the cross spectral and spatial differences between the MIR (central wavelength of 3.85 m) and 2 TIR (central wavelengths of 9.63 and 11.20 m) channels of the Advanced Himawari Imager (AHI). Sub variables include Solar zenith angle (SOZ) and satellite zenith angle (SAZ) of AHI, skin temperature (ST) and relative humidity (RH) of European Centre for Medium-Range Weather Forecasts (ECMWF) Reanalysis v5 (ERA5)-land data. Four processes (confidence, frequency, land cover, and continuity tests) were used to extract reference fire samples from Moderate Resolution Imaging Spectroradiometer (MODIS) and Visible Infrared Imaging Radiometer Suite (VIIRS) active fire products. To consider the different properties of key and sub variables, a 2-way convolutional neural network (CNN) structure was developed. To evaluate the influence of environmental variables, a CNN model without sub variables was adopted as a control model. The 2-way CNN (recall of 0.86, precision of 0.96, and standard deviation of recall of 0.13) was more robust at five focus sites than the control CNN (recall of 0.82, precision of 0.97, and standard deviation of recall of 0.163). Despite having a lower spatial resolution than MODIS/VIIRS, 2-way CNN outperformed other satellite-based active fire products (MODIS, VIIRS, AHI, and Advanced Meteorological Imager) in terms of detection capacity. The control CNN demonstrated poor performance under certain environmental conditions (high RH, high SAZ, and transition time between day and night), but 2-way CNN mitigates this tendency. In particular, the use of RH improved detection sensitivity, and SAZ contributed to the spatial robustness. This study demonstrated the significance of environmental conditions in active fire detection and proposed a suitable CNN structure for this intent. Based on the findings of this study, higher-level adaptive active fire monitoring under diverse environmental conditions will be possible together with explainable artificial intelligence.

How are images that have been assembled from their constituting elements maintained as a coherent representation in visual working memory (vWM)? Here, we compared two conditions of vWM maintenance that only differed in how vWM contents had been created. Participants maintained images that they either had to assemble from single features or that they had perceived as complete objects. Object complexity varied between two and four features. We analyzed electroencephalogram phase coupling as a measure of cortical connectivity in a time interval immediately before a probe stimulus appeared. We assumed that during this time both groups maintained essentially the same images, but that images constructed from their elements would require more neural coupling than images based on a complete percept. Increased coupling between frontal and parietal-to-occipital cortical sources was found for the maintenance of constructed in comparison to nonconstructed objects in the theta, alpha, beta, and gamma frequency bands. A similar pattern was found for an increase in vWM load (2 vs. 4 features) for nonconstructed objects. Under increased construction load (2 vs. 4 features for constructed images), the pattern was restricted to fronto-parietal couplings, suggesting that the fronto-parietal attention network is coping with the higher attentional demands involved in maintaining constructed images, but without increasing the communication with the occipital visual buffer in which the visual representations are assumed to be stored. We conclude from these findings that the maintenance of constructed images in vWM requires additional attentional processes to keep object elements together as a coherent representation. Hum Brain Mapp 37:4349-4362, 2016. © 2016 Wiley Periodicals, Inc.

An attribution graph-based interpretable method for CNNs

Stochastic Schemata Exploiter (SSE), which is one of Evolutionary Computations, is designed to find the optimal solution of the function. When comparing it with Genetic Algorithm (GA), which is a population evolutionary computation, SSE has interesting features; quick convergence and smaller number of control parameters. In this study, SSE is applied for designing hyperparameters and structure of Convolutional Neural Network (CNN). The validity of the proposal algorithm is discussed for determining CNN in experiments using CIFAR-10. The results show that SSE can find the better parameters and structure of CNN than GA.

In general, Convolutional Neural Networks (CNNs) have a complex network structure consisted of heavy layers with huge number of parameters such as the convolutional, pooling, relu-activation, and fully-connected layers. Due to the complexity and computation load, CNNs are trained on a cloud environment. There are a couple of drawbacks on learning and performing on the cloud such as security problem of personal information and dependency of communication state. Recently, CNNs are directly trained at the mobile devices in order to alleviate those two drawbacks. Due to the resource limitation of the mobile devices, the structure of CNNs needs to be compressed or to reduce training overhead. In this paper, we propose an on-device partial learning technique with the following benefits: (1) does not require additional neural network structures, and (2) reduces unnecessary computation overhead. We select a subset of influential weights from a trained network to accommodate the new classification class. The selection is made based on the information of the contribution of each weight to output, which is measured using the entropy concept. In the experimental section, we demonstrate and analyze our method with a CNN image classifier using two datasets such as Mixed National Institute of Standards and Technology image data and Microsoft Common Objection in Context data. As a result, the computational resources at LeNet-5 and AlexNet showed performance improvements of 1.7× and 2.3×, respectively, and memory resources demonstrated performance improvements of 1.4× and 1.6×, respectively.

Although it is well accepted that the formation of visual working memory (VWM) representations from simple static features is a rapid and effortless process that completes within several hundred milliseconds, the storage of motion information in VWM within that time scale can be challenging due to the limited processing capacity of the visual system. Memory formation can also be demanding especially when motion stimuli are visually complex. Here, we investigated whether the formation of VWM representations of motion direction is more gradual than that of static orientation and examined the effects of stimulus complexity on that process. To address these issues, we examined how the number and the precision of stored items in VWM develop over time by using a continuous report procedure. Results showed that while a viewing duration of several seconds was required for the successful storage of multiple motion directions in VWM regardless of motion complexity, the accumulation of memory precision was much slower when the motion stimulus was visually complex (Experiments 1 & 2). Additional experiments showed that the difference in memory performance for simple and complex motion stimuli cannot be explained by differences in signal-to-noise levels of the stimulus (Experiment 3). These results demonstrate remarkable temporal limitations in the formation of VWM representations for dynamic objects, and further show how this process is affected by stimulus properties such as visual complexity and signal-to-noise levels.

Attention has frequently been regarded as an emergent property of linking sensory representations to action plans. It has recently been proposed that similar mechanisms may operate within visual working memory (VWM), such that linking an object in VWM to an action plan strengthens its sensory memory representation, which then expresses as an attentional bias. Here we directly tested this hypothesis by comparing attentional biases induced by VWM representations which were the target of a future action, to those induced by VWM representations that were equally task-relevant, but not the direct target of action. We predicted that the first condition would result in a more prioritized memory state and hence stronger attentional biases. Specifically, participants memorized a geometric shape for a subsequent memory test. At test, in case of a match, participants either had to perform a grip movement on the matching object (action condition), or perform the same movement, but on an unrelated object (control condition). To assess any attentional biases, during the delay period between memorandum and test, participants performed a visual selection task in which either the target was surrounded by the memorized shape (congruent trials) or a distractor (incongruent trials). Eye movements were measured as a proxy for attentional priority. We found a significant interaction for saccade latencies between action condition and shape congruency, reflecting more pronounced VWM-based attentional biases in the action condition. Our results are consistent with the idea that action plans prioritize sensory representations in VWM.

In this paper, we investigate the optimal structure of deep convolutional neural networks for classifying human hand gestures using Doppler radar. When hand motions are captured by Doppler radar, the unique micro-Doppler signatures can be observed in the spectrogram. If the signature is distinguishable by a classifier, then the hand gesture can be used for controlling electronics and as an input modality for a human-computer interface. To classiiy signatures in the spectrogram, we propose the use of a deep convolutional neural network (DCNN) as a classifier. DCNN is a powerful classifier that extracts features as well as class boundaries through a training process. We measured seven hand gestures performed in front of Doppler radar while generating spectrograms. To identify an optimal structure, we trained several DCNNs by changing hyperparameters, such as the number of convolutional layers, the number of filters, and the filter size. The classification accuracy obtained from the optimal DCNN structure was approximately 87%.