Remembering the Past to See the Future.

Computer visualization can effectively deliver instructions to a user whose task requires understanding of a real world scene. Consider the example of surgical telementoring, where a general surgeon performs an emergency surgery under the guidance of a remote mentor. The mentor guidance includes annotations of the operating field, which conventionally are displayed to the surgeon on a nearby monitor. However, this conventional visualization of mentor guidance requires the surgeon to look back and forth between the monitor and the operating field, which can lead to cognitive load, delays, or even medical errors. Another example is 3D acquisition of a real-world scene, where an operator must acquire multiple images of the scene from specific viewpoints to ensure appropriate scene coverage and thus achieve quality 3D reconstruction. The conventional approach is for the operator to plan the acquisition locations using conventional visualization tools, and then to try to execute the plan from memory, or with the help of a static map. Such approaches lead to incomplete coverage during acquisition, resulting in an inaccurate reconstruction of the 3D scene which can only be addressed at the high and sometimes prohibitive cost of repeating acquisition.Augmented reality (AR) promises to overcome the limitations of conventional out-of-context visualization of real world scenes by delivering visual guidance directly into the user's field of view, guidance that remains in-context throughout the completion of the task. In this thesis, we propose and validate several AR visual interfaces that provide effective visual guidance for task completion in the context of surgical telementoring and 3D scene acquisition.A first AR interface provides a mentee surgeon with visual guidance from a remote mentor using a simulated transparent display. A computer tablet suspended above the patient captures the operating field with its on-board video camera, the live video is sent to the mentor who annotates it, and the annotations are sent back to the mentee where they are displayed on the tablet, integrating the mentor-created annotations directly into the mentee's view of the operating field. We show through user studies that surgical task performance improves when using the AR surgical telementoring interface compared to when using the conventional visualization of the annotated operating field on a nearby monitor. A second AR surgical telementoring interface provides the mentee surgeon with visual guidance through an AR head-mounted display (AR HMD). We validate this approach in user studies with medical professionals in the context of practice cricothyrotomy and lower-limb fasciotomy procedures, and show improved performance over conventional surgical guidance. A comparison between our simulated transparent display and our AR HMD surgical telementoring interfaces reveals that the HMD has the advantages of reduced workspace encumbrance and of correct depth perception of annotations, whereas the transparent display has the advantage of reduced surgeon head and neck encumbrance and of annotation visualization quality. A third AR interface provides operator guidance for effective image-based modeling and rendering of real-world scenes. During the modeling phase, the AR interface builds and dynamically updates a map of the scene that is displayed to the user through an AR HMD, which leads to the efficient acquisition of a five-degree-of-freedom image-based model of large, complex indoor environments. During rendering, the interface guides the user towards the highest-density parts of the image-based model which result in the highest output image quality. We show through a study that first-time users of our interface can acquire a quality image-based model of a 13m $\times$ 10m indoor environment in 7 minutes.A fourth AR interface provides operator guidance for effective capture of a 3D scene in the context of photogrammetric reconstruction. The interface relies on an AR HMD with a tracked hand-held camera rig to construct a sufficient set of six-degrees-of-freedom camera acquisition poses and then to steer the user to align the camera with the prescribed poses quickly and accurately. We show through a study that first-time users of our interface are significantly more likely to achieve complete 3D reconstructions compared to conventional freehand acquisition. We then investigated the design space of AR HMD interfaces for mid-air pose alignment with an added ergonomics concern, which resulted in five candidate interfaces that sample this design space. A user study identified the aspects of the AR interface design that influence the ergonomics during extended use, informing AR HMD interface design for the important task of mid-air pose alignment.

The ability to process visual stimuli rich with motion represents an essential skill for animal survival and is largely already present at the onset of vision. Although the exact mechanisms underlying its maturation remain elusive, spontaneous activity patterns in the retina, known as retinal waves, have been shown to contribute to this developmental process. Retinal waves exhibit complex spatio-temporal statistics and contribute to the establishment of circuit connectivity and function in the visual system, including the formation of retinotopic maps and the refinement of receptive fields in downstream areas such as the thalamus and visual cortex. Recent work in mice has shown that retinal waves have statistical features matching those of natural visual stimuli, such as optic flow, suggesting that they could prime the visual system for motion processing upon vision onset. Motivated by these findings, we examined whether artificial neural network (ANN) models trained on natural movies show improved performance if pre-trained with retinal waves. We employed the spatio-temporally complex task of next-frame prediction, in which the ANN was trained to predict the next frame based on preceding input frames of a movie. We found that pre-training ANNs with retinal waves enhances the processing of real-world visual stimuli and accelerates learning. Strikingly, when we merely replaced the initial training epochs on naturalistic stimuli with retinal waves, keeping the total training time the same, we still found that an ANN trained on retinal waves temporarily outperforms one trained solely on natural movies. Similar to observations made in biological systems, we also found that pre-training with spontaneous activity refines the receptive field of ANN neurons. Overall, our work sheds light on the functional role of spatio-temporally patterned spontaneous activity in the processing of motion in natural scenes, suggesting it acts as a training signal to prepare the developing visual system for adult visual processing.

Motor intention in the posterior parietal cortex

The cholinergic potentiation of visual conditioning enhances visual acuity and discrimination of the trained stimulus. To determine if this also induces long-term plastic changes on cortical maps and connectivity in the visual cortex and higher associative areas, mesoscopic calcium imaging was performed in head-fixed awake GCaMP6s adult mice before and after conditioning. The conditioned stimulus (0.03 cpd, 30°, 100% contrast, 1 Hz-drifting gratings) was presented 10 min daily for a week. Saline or Donepezil (DPZ, 0.3 mg/kg, s.c.), a cholinesterase inhibitor that potentiates cholinergic transmission, were injected prior to each conditioning session and compared to a sham-conditioned group. Cortical maps of resting state and evoked response to the monocular presentation of conditioned or non-conditioned stimulus (30°, 50 and 75% contrast; 90°, 50, 75, and 100% contrast) were established. Amplitude, duration, and latency of the peak response, as well as size of activation were measured in the primary visual cortex (V1), secondary visual areas (AL, A, AM, PM, LM, RL), retrosplenial cortex (RSC), and higher cortical areas. Visual stimulation increased calcium signaling in all primary and secondary visual areas, the RSC, but no other cortices. There were no significant effects of sham-conditioning or conditioning alone, but DPZ treatment during conditioning significantly decreased the integrated neuronal activity of superficial layers evoked by the conditioned stimulus in V1, AL, PM, and LM. The activity of downstream cortical areas was not changed. The size of the activated area was decreased in V1 and PM, and the signal-to-noise ratio was decreased in AL and PM. Interestingly, signal correlation was seen only between V1, the ventral visual pathway, and the RSC, and was decreased by DPZ administration. The resting state activity was slightly correlated and rarely affected by treatments, except between binocular and monocular V1 in both hemispheres. In conclusion, cholinergic potentiation of visual conditioning induced change in visual processing in the superficial cortical layers. This effect might be a key mechanism in the establishment of the fine cortical tuning in response to the conditioned visual stimulus.

In the early stages of development, correlated activity known as retinal waves causes periodic depolarizations of retinal ganglion cells (RGCs). The β2KO mouse, which lacks the β2 subunit of the nicotinic acetylcholine receptor, serves as a model for understanding the role of the cholinergic waves. β2KO mice have disruptions in several developmental processes of the visual system, including reduced retinotopic and eye-specific refinement of RGC axonal projections to their primary brain targets and an impact on the retinal circuits underlying direction selectivity. However, the effects of this mutation on gene expression in individual functional RGC types remain unclear. Here, we performed single-cell RNA sequencing on RGCs isolated at the end of the first postnatal week from wild-type and β2KO mice of either sex. We found that in β2KO, the programs governing RGC differentiation were not impacted and the magnitude of transcriptional changes was modest compared with those observed during 2 d of normal postnatal maturation. This contrasts with the substantial transcriptomic changes seen in downstream visual system areas under wave disruption in recent studies. Overall, we identified ∼238 genes whose expression was altered in a type-specific manner. We confirmed this result via in situ hybridization and whole-cell recording by focusing on one of the downregulated genes in αRGCs, Kcnk9, which encodes the two-pore domain leak potassium channel TASK3. Our study reveals a limited transcriptomic impact of cholinergic signaling in the retina, and instead of affecting all RGCs uniformly, these waves show subtle modulation of molecular programs in a type-specific manner.

Inspired by the human vision system, neuromorphic vision systems simulating the mechanism of signal processing in retina are extensively investigated. This paper describes a silicon neuronal network with bio-inspired structure detects the edge information in image. The digital spiking silicon neuron (DSSN) and silicon synapse models are used to reproduce the neuron, synapse and their dynamic behaviors. The biological receptive fields are realized by the combination of excitatory and inhibitory synapses. Our silicon neuronal network is optimized for the hardware implementation. It is expected to have low hardware consumption and high running speed. The simulation results show that edge detection is successful in our network.

An efficient approach toward a no-reference ringing metric intrinsically exists of two steps: first detecting regions in an image where ringing might occur, and second quantifying the ringing annoyance in these regions. This paper presents a novel approach toward the first step: the automatic detection of regions visually impaired by ringing artifacts in compressed images. It is a no-reference approach, taking into account the specific physical structure of ringing artifacts combined with properties of the human visual system (HVS). To maintain low complexity for real-time applications, the proposed approach adopts a perceptually relevant edge detector to capture regions in the image susceptible to ringing, and a simple yet efficient model of visual masking to determine ringing visibility. The approach is validated with the results of a psychovisual experiment, and its performance is compared to existing alternatives in literature for ringing region detection. Experimental results show that our method is promising in terms of both reliability and computational efficiency.

In this paper, I observed the properties of HVS for Ring Detection. The automatic detection of regions visually impaired by ringing artifacts in compressed images, it is a no-reference approach, taking into account the specific physical structure of ringing artifacts combined with properties of the human visual system (HVS). The approach is validated with the results of a psycho visual experiment, and its performance is compared to existing alternatives in literature for ringing region detection. Experimental results show that our method is appropriate in terms of both reliability and computational efficiency.

Anisometropic amblyopia is a neurodevelopmental disorder of the visual system. There is evidence that the neural deficits spread across visual areas, from the primary cortex up to higher brain areas, including motion coding structures such as MT. Here, we used bistable plaid motion to investigate changes in the underlying mechanisms of motion integration and segmentation and, thus, help us to unravel in more detail deficits in the amblyopic visual motion system. Our results showed that (1) amblyopes globally exhibited normal bistable perception in all viewing conditions compared to the control group and (2) decreased contrast led to a stronger increase in percept switches and decreased percept durations in the control group, while the amblyopic group exhibited no such changes. There were few differences in outcomes dependent upon the use of the weak eye, the strong eye, or both eyes for viewing the stimuli, but this was a general effect present across all subjects, not specific to the amblyopic group. To understand the role of noise and adaptation in such cases of bistable perception, we analyzed predictions from a model and found that contrast does indeed affect percept switches and durations as observed in the control group, in line with the hypothesis that lower stimulus contrast enhances internal noise effects. The combination of experimental and computational results presented here suggests a different motion coding mechanism in the amblyopic visual system, with relatively little effect of stimulus contrast on amblyopes' bistable motion perception.

Unimpaired performance during cognitive and visual manipulations in persons with anterior cruciate ligament reconstruction compared to healthy adults

Extracting statistical regularities from the environment is crucial for survival. It allows us to learn cues for where and when future events will occur. Can we learn these associations even when the cues are not consciously perceived? Can these unconscious processes integrate information over long periods of time? We show that human visual system can track the probability of location contingency between an unconscious prime and a conscious target over a period of time of minutes. In a series of psychophysical experiments, we adopted an exogenous priming paradigm and manipulated the location contingency between a masked prime and a visible target (i.e., how likely the prime location predicted the target location). The prime’s invisibility was verified both subjectively and objectively. Although the participants were unaware of both the existence of the prime and the prime-target contingency, our results showed that the probability of location contingency was tracked and manifested in the subsequent priming effect. When participants were first entrained into the fully predictive prime-target probability, they exhibited faster responses to the more predictive location. On the contrary, when no contingency existed between the prime and target initially, participants later showed faster responses to the less predictive location. These results were replicated in two more experiments with increased statistical power and a fine-grained delineation of prime awareness. Together, we report that the human visual system is capable of tracking unconscious probability over a period of time, demonstrating how implicit and uncertain regularity guides behavior.

During natural vision, humans make frequent eye movements but perceive a stable visual world. It is therefore likely that the human visual system contains representations of the visual world that are invariant to eye movements. Here we present an experiment designed to identify visual areas that might contain eye-movement-invariant representations. We used functional MRI to record brain activity from four human subjects who watched natural movies. In one condition subjects were required to fixate steadily, and in the other they were allowed to freely make voluntary eye movements. The movies used in each condition were identical. We reasoned that the brain activity recorded in a visual area that is invariant to eye movement should be similar under fixation and free viewing conditions. In contrast, activity in a visual area that is sensitive to eye movement should differ between fixation and free viewing. We therefore measured the similarity of brain activity across repeated presentations of the same movie within the fixation condition, and separately between the fixation and free viewing conditions. The ratio of these measures was used to determine which brain areas are most likely to contain eye movement-invariant representations. We found that voxels located in early visual areas are strongly affected by eye movements, while voxels in ventral temporal areas are only weakly affected by eye movements. These results suggest that the ventral temporal visual areas contain a stable representation of the visual world that is invariant to eye movements made during natural vision.

Deep neural networks are sensitive to input perturbations, such as lighting changes, undermining reliability in real-world applications. This paper introduces interval-analysis enhanced MobileNet, which is a robust image classification framework that integrates interval bound propagation into the lightweight MobileNet architecture. By modeling intensity variations as bounded intervals, the method computes guaranteed output bounds that ensure prediction stability across all perturbed inputs within the defined range. The approach reveals high accuracy without any problem in computational efficiency, making it suitable for edge deployment. The network has been trained using a hybrid loss that jointly optimizes standard accuracy and robustness margins under interval uncertainty. The experiments on CIFAR-10 and ImageNet-100 demonstrate extreme performance improvements across all intensity perturbations based on natural accuracy, far exceeding adversarial training, data augmentation, and randomized smoothing. The results show that IA-MobileNet achieves a high level of accuracy while also providing formal robustness. This bridges the gap between efficiency and reliability in vision systems that perform under unpredictable environmental conditions.

This article presents an efficient visual landmark extraction and recognition method that can autonomously and rapidly detect visual features such as objects or groups of small objects, and that can be applied to visual object recognition based SLAM and navigation in indoor/large environments using a monocular/omnidirection vision system. Our method consists of two-stage: (1) we autonomously extract object regions with modified fuzzy object segmentation. We generate a saliency map of the scene based on Modified Phase spectrum of Fourier Transform (mPFT) and extract the final salient object landmark with weighted combination of candidate of objects and saliency map. (2) Using these result, we register current objects as visual landmark and then recognize the current image the scale invariant feature transform (SIFT) - based recognition with probabilistic voting. In experiments results in real indoor and large hall environments, the proposed method was simpler and 10~15% better performance in computation efficiency and successfully extracted salient object landmark in complex environments with high recognition rates. The proposed algorithm can be easily implemented in real-time by reducing the number of objects considered.

In order to have an efficient use of digital image processing and pattern recognition techniques, it is necessary to understand the human visual system behaviour. The human visual system comprises the human eye and some areas of the human brain which performs neurological processing information. Human eye and brain together convert the optical information in a visual scene perception, the eye functions as the human visual system camera, and his function is to transform light electromagnetic waves to small electrical signals carrying the information to the brain to achieve data analysis and build a structured high resolution image. For a machine vision, this process represents a very complex task to implement; adequate elements have to exist like: environment perception, information processing and primitive base knowledge generation to achieve specific actions for decision making based on interpretation of such information. Artificial vision then might be defined as: machine implementation with capabilities of visual perception of the surrounding environment, extraction of region of interest, analysis, scene interpretation and decision making. In this regard some authors like Haralick & Shapiro have defined machine vision as the science which study and develop the theoretical bases and algorithms to obtain information of the real world from one or some images (Haralick&Shapiro, 1992). Another notable definition is proposed by Pajares et al. that define it as a machine capability to see its rounded world, to understand the structure and properties of a 3D world based in the analysis of one or more 2D images. The system described in this article resembles the above capability in automated industrial applications using manufacturing machinery or robots, which in some moment of the industrial process need to obtain its position and location within their working space, so to accomplish their labour tasks efficiently. A real example is locating and positioning working pieces within a manufacture area during assembly, painting, sorting or storage operations. The goal is to identify the working piece and make reference to a visual scene, then with a geometrical model, the working piece position is calculated and so the necessary path to reach the pieces using a robot-arm as described by de Lope (de Lope, et al., 1997) typically in an eye-in-hand configuration. A frequent requirement that we have found during manipulative tasks using mobile robots or multiple collaborative robot arms is to accurately locate objects within the work space or