Look-Back: Implicit Visual Re-focusing in MLLM Reasoning

OBJECTIVESTo investigate the roles of visual and tactile information in a dyspraxic patient with corticobasal degeneration (CBD) who showed dramatic facilitation in miming the use of a tool or object...

Visual working memory (VWM) is used to maintain visual information available for subsequent goal-directed behavior. The content of VWM has been shown to affect the behavioral response to concurrent visual input, suggesting that visual representations originating from VWM and from sensory input draw upon a shared neural substrate (i.e., a sensory recruitment stance on VWM storage). Here, we hypothesized that visual information maintained in VWM would enhance the neural response to concurrent visual input that matches the content of VWM. To test this hypothesis, we measured fMRI BOLD responses to task-irrelevant stimuli acquired from 15 human participants (three males) performing a concurrent delayed match-to-sample task. In this task, observers were sequentially presented with two shape stimuli and a retro-cue indicating which of the two shapes should be memorized for subsequent recognition. During the retention interval, a task-irrelevant shape (the probe) was briefly presented in the peripheral visual field, which could either match or mismatch the shape category of the memorized stimulus. We show that this probe stimulus elicited a stronger BOLD response, and allowed for increased shape-classification performance, when it matched rather than mismatched the concurrently memorized content, despite identical visual stimulation. Our results demonstrate that VWM enhances the neural response to concurrent visual input in a content-specific way. This finding is consistent with the view that neural populations involved in sensory processing are recruited for VWM storage, and it provides a common explanation for a plethora of behavioral studies in which VWM-matching visual input elicits a stronger behavioral and perceptual response.SIGNIFICANCE STATEMENT Humans heavily rely on visual information to interact with their environment and frequently must memorize such information for later use. Visual working memory allows for maintaining such visual information in the mind's eye after termination of its retinal input. It is hypothesized that information maintained in visual working memory relies on the same neural populations that process visual input. Accordingly, the content of visual working memory is known to affect our conscious perception of concurrent visual input. Here, we demonstrate for the first time that visual input elicits an enhanced neural response when it matches the content of visual working memory, both in terms of signal strength and information content.

In this paper the implementation of Li-Fi technology in classrooms has been investigated. It is well known that the visual information is better perceived by the brain as compared with the textual information. Using visuals to explain a concept helps student get a better perception about the topic. To ensure the best of the visual experience, there is a big need of an efficient way of communication between the devices used by students and the teacher.The traditional methods like Wi-Fi or wired connections are not much efficient due to the speed limitation and other factors. Light fidelity, also known as Li-Fi is a type of data transmission technology which uses light rays to transmit data. By implementation of this technology, high volume of data can be transmitted at extremely high speed as it gives higher bandwidth. It also ensures efficiency, security, as well as the mobility of the device. Hence, we are looking to implement this technology to enhance the learning experience of the students. Every student gets a visual input on what, when, where and how anything happens and the concepts are understood easily and thoroughly. Also, it is cheap and safe to the human body.

The basis for individual differences in the degree to which visual speech input enhances comprehension of acoustically degraded speech is largely unknown. Previous research indicates that fine facial detail is not critical for visual enhancement when auditory information is available; however, these studies did not examine individual differences in ability to make use of fine facial detail in relation to audiovisual speech perception ability. Here, we compare participants based on their ability to benefit from visual speech information in the presence of an auditory signal degraded with noise, modulating the resolution of the visual signal through low-pass spatial frequency filtering and monitoring gaze behavior. Participants who benefited most from the addition of visual information (high visual gain) were more adversely affected by the removal of high spatial frequency information, compared to participants with low visual gain, for materials with both poor and rich contextual cues (i.e., words and sentences, respectively). Differences as a function of gaze behavior between participants with the highest and lowest visual gains were observed only for words, with participants with the highest visual gain fixating longer on the mouth region. Our results indicate that the individual variance in audiovisual speech in noise performance can be accounted for, in part, by better use of fine facial detail information extracted from the visual signal and increased fixation on mouth regions for short stimuli. Thus, for some, audiovisual speech perception may suffer when the visual input (in addition to the auditory signal) is less than perfect.

Recognizing speech from silent lip movement, which is called lip reading, is a challenging task due to 1) the inherent information insufficiency of lip movement to fully represent the speech, and 2) the existence of homophenes that have similar lip movement with different pronunciations. In this paper, we try to alleviate the aforementioned two challenges in lip reading by proposing a Multi-head Visual-audio Memory (MVM). Firstly, MVM is trained with audio-visual datasets and remembers audio representations by modelling the inter-relationships of paired audio-visual representations. At the inference stage, visual input alone can extract the saved audio representation from the memory by examining the learned inter-relationships. Therefore, the lip reading model can complement the insufficient visual information with the extracted audio representations. Secondly, MVM is composed of multi-head key memories for saving visual features and one value memory for saving audio knowledge, which is designed to distinguish the homophenes. With the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input lip movement. This also can be viewed as an explicit implementation of the one-to-many mapping of viseme-to-phoneme. Moreover, MVM is employed in multi-temporal levels to consider the context when retrieving the memory and distinguish the homophenes. Extensive experimental results verify the effectiveness of the proposed method in lip reading and in distinguishing the homophenes.

Addressing the gap in understanding visual comprehension in Large Language Models (LLMs), we designed a challenge-response study, subjecting Google Bard to 64 visual tasks, spanning categories like "Visual Situational Reasoning" and "Next Scene Prediction." Previous models, such as GPT4, leaned heavily on optical character recognition tools like Tesseract, whereas Bard, akin to Google Lens and Visual API, employs deep learning techniques for visual text recognition. However, our findings spotlight Bard's limitations: while proficient in solving visual CAPTCHAs that stump ChatGPT, it falters in recreating visual elements like ASCII art or analyzing Tic Tac Toe grids, suggesting an over-reliance on educated visual guesses. The prediction problem based on visual inputs appears particularly challenging with no commonsense guesses for next scene forecasting based on current “next- token” multimodal models. This study provides experimental insights into the current capacities and areas for improvement in multimodal LLMs.

An image is worth a thousand words, conveying information that goes beyond the mere visual content therein. In this paper, we study the intent behind social media images with an aim to analyze how visual information can facilitate recognition of human intent. Towards this goal, we introduce an intent dataset, Intentonomy, comprising 14K images covering a wide range of everyday scenes. These images are manually annotated with 28 intent categories derived from a social psychology taxonomy. We then systematically study whether, and to what extent, commonly used visual information, i.e., object and context, contribute to human motive understanding. Based on our findings, we conduct further study to quantify the effect of attending to object and context classes as well as textual information in the form of hashtags when training an intent classifier. Our results quantitatively and qualitatively shed light on how visual and textual information can produce observable effects when predicting intent. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Recent years have seen a rapid growth in the number of fake news that are posted online. Fake news detection is very challenging since they are usually created to contain a mixture of false and real information and images that have been manipulated that confuses the readers. In this paper, we propose a multimodal system with the aim to differentiate between fake and real posts. Our system is based on a neural network and combines textual, visual and semantic information. The textual information is extracted from the content of the post, the visual one from the image that is associated with the post and the semantic refers to the similarity between the image and the text of the post. We conduct our experiments on three standard real world collections and we show the importance of those features on detecting fake news.

As mobile and chatbot-based shopping platforms continue to reshape online retail experiences, consumers are increasingly exposed to product information in sequential rather than simultaneous formats. Drawing on information foraging theory, we examine the marginal value consumers derive when textual and visual product information are presented sequentially. We conducted an online experiment (n = 86) with student sample to test whether different information presentation sequence (text first vs. image first) would lead to different perceived diagnosticity, product attitude, and purchase intention. We also examined whether this sequence effect differed between search (notebook and hanger) and experience (sofa and portable closet) products. Findings from mixed-effects regressions show that presenting textual information before visual information results in greater marginal gains in all three outcomes, particularly for experience products. These findings offer new insights into the cognitive dynamics of sequential information processing and practical implications for optimizing product presentation in online retail design.

Different sensory systems (e.g. proprioception and vision) have a combined influence on the perception of body orientation, but the timescale over which they can be integrated remains unknown. Here we examined how visual information and neck proprioception interact in perception of the "subjective straight ahead" (SSA), as a function of time since initial stimulation. In complete darkness, healthy subjects directed a laser spot to the point felt subjectively to be exactly straight ahead of the trunk. As previously observed, left neck muscle vibration led to a disparity between subjective perception and objective position of the body midline, with SSA misplaced to the left. We found that this displacement was sustained throughout 28 min of continuous proprioceptive stimulation, provided there was no visual input. Moreover, prolonged vibration of neck muscles leads to a continuing disparity between subjective and objective body orientation even after offset of the vibration; the longer the preceding vibration, the more persistent the illusory deviation of body orientation. To examine the role of vision, one group of subjects fixated a central visual target at the start of each block of continuous neck vibration, with SSA then measured at successive intervals in darkness. The illusory deviation of SSA was eliminated whenever visual input was provided, but returned as a linear function of time when visual information was eliminated. These results reveal: the persistent effects of neck proprioception on the SSA, both during and after vibration; the influence of vision; and integration between incoming proprioceptive information and working memory traces of visual information.

Decision letter: Visual and motor signatures of locomotion dynamically shape a population code for feature detection in Drosophila

Head direction (HD) neurons, signalling facing direction, generate a signal that is primarily anchored to the outside world by visual inputs. We investigated the route for visual landmark information into the HD system in rats. There are two candidates: an evolutionarily older, larger subcortical retino-tectal pathway and a more recently evolved, smaller cortical retino-geniculo-striate pathway. We disrupted the cortical pathway by lesioning the dorsal lateral geniculate thalamic nuclei bilaterally, and recorded HD cells in the postsubicular cortex as rats foraged in a visual-cue-controlled enclosure. In lesioned rats we found the expected number of postsubicular HD cells. Although directional tuning curves were broader across a trial, this was attributable to the increased instability of otherwise normal-width tuning curves. Tuning curves were also poorly responsive to polarizing visual landmarks and did not distinguish cues based on their visual pattern. Thus, the retino-geniculo-striate pathway is not crucial for the generation of an underlying, tightly tuned directional signal but does provide the main route for vision-based anchoring of the signal to the outside world, even when visual cues are high in contrast and low in detail. KEY POINTS: Head direction (HD) cells indicate the facing direction of the head, using visual landmarks to distinguish directions. In rats, we investigated whether this visual information is routed through the thalamus to the visual cortex or arrives via the superior colliculus, which is a phylogenetically older and (in rodents) larger pathway. We lesioned the thalamic dorsal lateral geniculate nucleus (dLGN) in rats and recorded the responsiveness of cortical HD cells to visual cues. We found that cortical HD cells had normal tuning curves, but these were slightly more unstable during a trial. Most notably, HD cells in dLGN-lesioned animals showed little ability to distinguish highly distinct cues and none to distinguish more similar cues. These results suggest that directional processing of visual landmarks in mammals requires the geniculo-cortical pathway, which raises questions about when and how visual directional landmark processing appeared during evolution.

CE-DCVSI: Multimodal relational extraction based on collaborative enhancement of dual-channel visual semantic information

The goal of this study was to determine what visual information is used to navigate around barriers in a cluttered terrain. Twelve traffic pylons were arranged randomly in a 4.55 x 3.15 m travel area: there were 20 different arrangements. For each arrangement, individuals (N = 6) were positioned in 1 of 3 locations on the outside border with their eyes closed: on verbal command they were instructed to open their eyes and quickly go to 1 of 2 specified goals (2 vertical posts defining a door) located on one edge of the travel area. The movement of the body was tracked using the OPTOTRAK system, with the IREDS placed on a collar worn by the subjects. Experimental data of travel path chosen were compared with those predicted by models that incorporated different types of visual information to control path trajectory. The 6 models basically use 2 different strategies for route selection: reactive control based on visual input about the obstacle encountered in the line-of-sight travel path (Model # 1) and path planning based on different visual information (Model # 2, 3, 4, 5, and 6). The models that involve path planning are grouped into 2 categories: models 2, 3, 4, and 5 need detailed geometrical configuration of the obstacles to plan a route while model 6 plans a route based on identifying and avoiding a cluster of obstacles in the travel path. Two measures were used to compare model performance with the actual travel path: the difference in area between predicted and actual travel path and the number of trials that accurately predicted the number of turns during travel. The results suggest that route selection is not based on reactive control, but does involve path planning. The model that best predicts the travel paths taken by the individuals uses visual information about cluster of obstacles and identification of safe corridors to plan a route.

In natural language processing (NLP) and computer vision (CV), the successful application of foundation models across diverse tasks has demonstrated their remarkable potential.However, despite the rich structural and textual information embedded in knowledge graphs (KGs), existing research of foundation model for KG has primarily focused on their structural aspects, with most efforts restricted to in-KG tasks (e.g., knowledge graph completion, KGC).This limitation has hindered progress in addressing more challenging out-of-KG tasks.In this paper, we introduce MERRY, a foundation model for general knowledge graph reasoning, and investigate its performance across two task categories: in-KG reasoning tasks (e.g., KGC) and out-of-KG tasks (e.g., KG question answering, KGQA).We not only utilize the structural information, but also the textual information in KGs.Specifically, we propose a multi-perspective Conditional Message Passing (CMP) encoding architecture to bridge the gap between textual and structural modalities, enabling their seamless integration.Additionally, we introduce a dynamic residual fusion module to selectively retain relevant textual information and a flexible edge scoring mechanism to adapt to diverse downstream tasks.Comprehensive evaluations on 28 datasets demonstrate that MERRY outperforms existing baselines in most scenarios, showcasing strong reasoning capabilities within KGs and excellent generalization to out-of-KG tasks such as KGQA.