Neural dynamics of visual working memory representation during sensory distraction

Recent studies have provided evidence for the concurrent encoding of sensory percepts and visual working memory (VWM) contents across visual areas; however, it has remained unclear how these two types of representations are concurrently present. Here, we reanalyzed an open-access fMRI dataset where participants memorized a sensory stimulus while simultaneously being presented with sensory distractors. First, we found that the VWM code in several visual regions did not fully generalize between different time points, suggesting a dynamic code. A more detailed analysis revealed that this was due to shifts in coding spaces across time. Second, we collapsed neural signals across time to assess the degree of interference between VWM contents and sensory distractors, specifically by testing the alignment of their encoding spaces. We find that VWM and feature-matching sensory distractors are encoded in coding spaces that do not fully overlap, but the separation decreases when distractors negatively impact behavioral performance in recalling the target. Together, these results indicate a role of dynamic coding and temporally stable coding spaces in helping multiplex perception and VWM within visual areas.

Visual working memory (VWM) is used to maintain visual information available for subsequent goal-directed behavior. The content of VWM has been shown to affect the behavioral response to concurrent visual input, suggesting that visual representations originating from VWM and from sensory input draw upon a shared neural substrate (i.e., a sensory recruitment stance on VWM storage). Here, we hypothesized that visual information maintained in VWM would enhance the neural response to concurrent visual input that matches the content of VWM. To test this hypothesis, we measured fMRI BOLD responses to task-irrelevant stimuli acquired from 15 human participants (three males) performing a concurrent delayed match-to-sample task. In this task, observers were sequentially presented with two shape stimuli and a retro-cue indicating which of the two shapes should be memorized for subsequent recognition. During the retention interval, a task-irrelevant shape (the probe) was briefly presented in the peripheral visual field, which could either match or mismatch the shape category of the memorized stimulus. We show that this probe stimulus elicited a stronger BOLD response, and allowed for increased shape-classification performance, when it matched rather than mismatched the concurrently memorized content, despite identical visual stimulation. Our results demonstrate that VWM enhances the neural response to concurrent visual input in a content-specific way. This finding is consistent with the view that neural populations involved in sensory processing are recruited for VWM storage, and it provides a common explanation for a plethora of behavioral studies in which VWM-matching visual input elicits a stronger behavioral and perceptual response.SIGNIFICANCE STATEMENT Humans heavily rely on visual information to interact with their environment and frequently must memorize such information for later use. Visual working memory allows for maintaining such visual information in the mind's eye after termination of its retinal input. It is hypothesized that information maintained in visual working memory relies on the same neural populations that process visual input. Accordingly, the content of visual working memory is known to affect our conscious perception of concurrent visual input. Here, we demonstrate for the first time that visual input elicits an enhanced neural response when it matches the content of visual working memory, both in terms of signal strength and information content.

The contents of visual working memory delay the perceived offset of matching visual stimuli

Recent studies have revealed that the deployment of attention can be biased by the content of visual working memory (VWM), but that stored memories do not always interact with attention. This has led to a model which proposes a division within VWM between a single active template that interacts with perception and multiple accessory representations that do not. The present study was designed to study whether multiple memory representations are able to bias attention. Participants performed a visual search task while maintaining a variable number of colors in VWM. Consistent with earlier findings, we observed increased attentional capture by memory related distractors when VWM was filled with a single item. However, memory related capture was no longer present for memory loads beyond a single item. The absence of memory related capture at higher VWM loads was independent of individual VWM capacity, nor was it attributable to weaker encoding, forgetting, or reduced precision of memory representations. When analyses were limited to those trials in which participants had a relatively precise memory, there was still no sign of attentional guidance at higher loads. However, when observers were cued toward a specific memory item after encoding, interference with search returned. These results are consistent with a distinction within VWM between representations that interact with perception and those that do not, and show that only a single VWM representation at a time can interact with visual attention.

Visual working memory (VWM) and oculomotor control exert bi-directional influences on each other: eye movements during memory delays can influence VWM performance, and VWM content can bias the execution of eye movements. Oculomotor capture has been shown to extend to automatic, corrective saccades that can be biased towards distractors whose feature matches VWM content. We address the question of bi-directional influences between VWM and the oculomotor system, here with a focus on corrective saccade biases. Twenty participants had to memorize a color hue and later discriminate it from another hue from a different (Easy condition) or the same (Difficult condition) color category. Between encoding and test, they performed a gaze-contingent saccade task where, mid-saccade, the position of items shifted, introducing an artificial saccade error, and the color of a distractor neighboring the target changed to a color (mis)matching the memorized one. VWM performance was higher in the Easy compared to the Difficult experimental block, reflecting the successful manipulation of task demands. Corrective saccade biases towards the distractor occurred more frequently when it matched memory content compared to when it did not, replicating previous findings. Importantly, they also occurred more in the Difficult experimental block, which required more precise VWM representations than the Easy condition. VWM performance, however, was not affected by saccade corrections to memory (mis)matching distractors. Our results provided no evidence of saccade biases having a role in the updating of VWM content; however, they indicate an influence of task demands and VWM engagement on automatic oculomotor biases.

Visual working memory (VWM) sustains information online as integrated object representations. Neuronal mechanisms supporting the maintenance of feature-specific information have remained unidentified. Synchronized oscillations in the gamma band (30-120 Hz) characterize VWM retention and predict task performance, but whether these oscillations are specific to memorized features and VWM contents or underlie general executive VWM functions is not known. In the present study, we investigated whether gamma oscillations reflect the maintenance of feature-specific information in VWM. Concurrent magneto- and electroencephalography was recorded while subjects memorized different object features or feature conjunctions in identical VWM experiments. Using a data-driven source analysis approach, we show that the strength, load-dependence, and source topographies of gamma oscillations in the visual cortex differentiate these memorized features. Load-dependence of gamma oscillations in feature-specific visual and prefrontal areas also predicts VWM accuracy. Furthermore, corroborating the hypothesis that gamma oscillations support the perceptual binding of feature-specific neuronal assemblies, we also show that VWM for color-location conjunctions is associated with stronger gamma oscillations than that for these features separately. Gamma oscillations hence support the maintenance of feature-specific information and reflect VWM contents. The results also suggest that gamma oscillations contribute to feature binding in the formation of memory representations.

The serial and spatially extended nature of many real-world visual tasks suggests the need for control over the content of visual working memory (VWM). We examined the management of VWM in a task that required participants to prioritize individual objects for retention during scene viewing. There were 5 principal findings: (a) Strategic retention of task-relevant objects was effective and was dissociable from the current locus of visual attention; (b) strategic retention was implemented by protection from interference rather than by preferential encoding; (c) this prioritization was flexibly transferred to a new object as task demands changed; (d) no-longer-relevant items were efficiently eliminated from VWM; and (e) despite this level of control, attended and fixated objects were consolidated into VWM regardless of task relevance. These results are consistent with a model of VWM control in which each fixated object is automatically encoded into VWM, replacing a portion of the content in VWM. However, task-relevant objects can be selectively protected from replacement.

The content of visual working memory (VWM) guides attention, but whether this interaction is limited to a single VWM representation or functional for multiple VWM representations is under debate. To test this issue, we developed a gaze-contingent search paradigm to directly manipulate selection history and examine the competition between multiple cue-matching saccade target objects. Participants first saw a dual-color cue followed by two pairs of colored objects presented sequentially. For each pair, participants selectively fixated an object that matched one of the cued colors. Critically, for the second pair, the cued color from the first pair was presented either with a new distractor color or with the second cued color. In the latter case, if two cued colors in VWM interact with selection simultaneously, we expected the second cued color object to generate substantial competition for selection, even though the first cued color was used to guide attention in the immediately previous pair. Indeed, in the second pair, selection probability of the first cued color was substantially reduced in the presence of the second cued color. This competition between cue-matching objects provides strong evidence that both VWM representations interacted simultaneously with selection. (PsycINFO Database Record

Previous research indicates that visual attention can be automatically captured by sensory inputs that match the contents of visual working memory. However, Woodman and Luck (2007) showed that information in working memory can be used flexibly as a template for either selection or rejection according to task demands. We report two experiments that extend their work. Participants performed a visual search task while maintaining items in visual working memory. Memory items were presented for either a short or long exposure duration immediately prior to the search task. Memory was tested by a change-detection task immediately afterwards. On a random half of trials items in memory matched either one distractor in the search task (Experiment 1) or three (Experiment 2). The main result was that matching distractors speeded or slowed target detection depending on whether memory items were presented for a long or short duration. These effects were more in evidence with three matching distractors than one. We conclude that the influence of visual working memory on visual search is indeed flexible but is not solely a function of task demands. Our results suggest that attentional capture by perceptual inputs matching information in visual working memory involves a fast automatic process that can be overridden by a slower top-down process of attentional avoidance.

SUMMARYNeuronal oscillations are suggested to play an important role in auditory working memory (WM), but their contribution to content-specific representations has remained unclear. Here, we measure magnetoencephalography during a retro-cueing task with parametric ripple-sound stimuli, which are spectrotemporally similar to speech but resist non-auditory memory strategies. Using machine learning analyses, with rigorous between-subject cross-validation and non-parametric permutation testing, we show that memorized sound content is strongly represented in phase-synchronization patterns between subregions of auditory and frontoparietal cortices. These phase-synchronization patterns predict the memorized sound content steadily across the studied maintenance period. In addition to connectivity-based representations, there are indices of more local, “activity silent” representations in auditory cortices, where the decoding accuracy of WM content significantly increases after task-irrelevant “impulse stimuli.” Our results demonstrate that synchronization patterns across auditory sensory and association areas orchestrate neuronal coding of auditory WM content. This connectivity-based coding scheme could also extend beyond the auditory domain.

During scene viewing, visual working memory (VWM) is used to retain information from recently attended and fixated objects. In the present study, I examined whether and how people can strategically control the content of VWM during scene viewing, prioritizing task-relevant objects for retention even as the eyes are directed to subsequent objects. Participants viewed a set of real-world objects presented serially within a 3-D rendered scene. One object in the sequence was cued by a tone as to-beremembered. At the end of the sequence, memory for the visual form of one object was tested. Participants exhibited tight control over the content of VWM, implementing prioritization after the encoding of an object into VWM, protecting that item from subsequent interference. Participants also successfully reallocated protection to subsequent objects, regardless of the duration of prioritization of the original item. Such strategic maintenance of objects in VWM is likely to play an important role in real-world visual behavior, especially when object information must be maintained across shifts of attention and the eyes to other objects (such as when comparing two spatially separated objects). Abstract Approved:________________________________________________________ Thesis Supervisor ________________________________________________________ Title and Department ________________________________________________________ DateApproved:________________________________________________________ Thesis Supervisor ________________________________________________________ Title and Department ________________________________________________________ Date STRATEGIC CONTROL OF VISUAL WORKING MEMORY DURING SCENE VIEWING by Ashleigh Monette Richard A thesis submitted in partial fulfillment of the requirements for the Doctor of Philosophy degree in Psychology in the Graduate College of The University of Iowa

How do verbal descriptions affect visual memory over the short and long term? Here we show for the first time that verbal labeling can boost visual memories, but the source of this benefit depends on whether representations are maintained over the short term in visual working memory or over the long term in visual long-term memory. Across three experiments, we contrasted color memory of randomly colored objects when participants labeled (a) the color, (b) the object, or (c) the color-object binding, to memory under an articulatory suppression condition inhibiting labeling. Memory was tested at two time points: after three objects (visual working memory) and at the end of the experiment (visual long-term memory). In Experiment 1, color labeling improved, whereas object labeling impaired, visual working memory in comparison to suppression. Visual long-term memory remained unchanged across conditions. Experiment 2 tested whether this was attributable to poor overall long-term learning by repeating the colored objects over three successive working memory trials. This increased performance over the short and long term, yet labeling did not change learning rate over repetitions or delayed memory performance, showing no long-term memory benefit. In Experiment 3, a labeling benefit was observed when the color-object binding was labeled both over the short and long term. Mixture modeling indicated that color-labeling benefits in visual working memory resulted from an increase of detailed visual memory, whereas long-term memory benefits accrued from categorical representations. Our findings point to dissociations on the role of language in visual working memory and visual long-term memory. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

In this study, we investigated whether the ability to learn probability information is affected by the type of representation held in visual working memory. Across 4 experiments, participants detected changes to displays of coloured shapes. While participants detected changes in 1 dimension (e.g., colour), a feature from a second, nonchanging dimension (e.g., shape) predicted which object was most likely to change. In Experiments 1 and 3, items could be grouped by similarity in the changing dimension across items (e.g., colours and shapes were repeated in the display), while in Experiments 2 and 4 items could not be grouped by similarity (all features were unique). Probability information from the predictive dimension was learned and used to increase performance, but only when all of the features within a display were unique (Experiments 2 and 4). When it was possible to group by feature similarity in the changing dimension (e.g., 2 blue objects appeared within an array), participants were unable to learn probability information and use it to improve performance (Experiments 1 and 3). The results suggest that probability information can be learned in a dimension that is not explicitly task-relevant, but only when the probability information is represented with the changing dimension in visual working memory.

Although it is well accepted that the formation of visual working memory (VWM) representations from simple static features is a rapid and effortless process that completes within several hundred milliseconds, the storage of motion information in VWM within that time scale can be challenging due to the limited processing capacity of the visual system. Memory formation can also be demanding especially when motion stimuli are visually complex. Here, we investigated whether the formation of VWM representations of motion direction is more gradual than that of static orientation and examined the effects of stimulus complexity on that process. To address these issues, we examined how the number and the precision of stored items in VWM develop over time by using a continuous report procedure. Results showed that while a viewing duration of several seconds was required for the successful storage of multiple motion directions in VWM regardless of motion complexity, the accumulation of memory precision was much slower when the motion stimulus was visually complex (Experiments 1 & 2). Additional experiments showed that the difference in memory performance for simple and complex motion stimuli cannot be explained by differences in signal-to-noise levels of the stimulus (Experiment 3). These results demonstrate remarkable temporal limitations in the formation of VWM representations for dynamic objects, and further show how this process is affected by stimulus properties such as visual complexity and signal-to-noise levels.

Cochlear implantation is a well-established method for restoring hearing sensation in individuals with severe to profound hearing loss. It significantly improves verbal communication for many users, despite substantial variability in patients' reports and performance on speech perception tests and quality-of-life outcome measures. Such variability in outcome measures remains several years after implantation and could reflect difficulties in attentional regulation. The current study assessed the ability to use a cue to guide attention internally toward visual or auditory working memory (i.e., reflective attention) in cochlear implant (CI) users. Participants completed a cognitive task called the delayed match-to-sample task in which a visual gradient was presented on a computer screen and a piano tone was presented through speakers simultaneously. A visual cue (i.e., letter A or V) instructed participants to focus attention on the item held in auditory or visual working memory. After a delay following the cue presentation, participants were presented with a probe item and indicated by pressing a button whether it matched the cued item in working memory. CI users and age-matched normal hearing adults showed comparable benefit from having an informative cue relative to an uninformative cue (i.e., letter X). Although CI users have had a history of severe deafness and experience coarse sound information, they were able to retrospectively orient their attention to an item in auditory or visual working memory. These findings suggest that CI users with at least one year of CI experience can successfully regulate attention to a level that is comparable to that of normal hearing individuals.