Applying artificial vision models to human scene understanding.

The occipital place area represents the local elements of scenes

Human adults flawlessly and effortlessly navigate boundaries and obstacles in the immediately visible environment, a process we refer to as guided navigation. Neuroimaging work in adults suggests this ability involves the occipital place area (OPA) [1, 2]-a scene-selective region in the dorsal stream that selectively represents information necessary for visually guided navigation [3-9]. Despite progress in understanding the neural basis of visually guided navigation, however, little is known about how this system develops. Is navigationally relevant information processing present in the first few years of life? Or does this information processing only develop after many years of experience? Although a handful of studies have found selective responses to scenes (relative to objects) in OPA in childhood [10-13], no study has explored how more specific navigationally relevant information processing emerges in this region. Here, we do just that by measuring OPA responses to first-person perspective motion information-a proxy for the visual experience of actually navigating the immediate environment-using fMRI in 5- and 8-year-old children. We found that, although OPA already responded more to scenes than objects by age 5, responses to first-person perspective motion were not yet detectable at this same age and rather only emerged by age 8. This protracted development was specific to first-person perspective motion through scenes, not motion on faces or objects, and was not found in other scene-selective regions (the parahippocampal place area or retrosplenial complex) or a motion-selective region (MT). These findings therefore suggest that navigationally relevant information processing in OPA undergoes prolonged development across childhood.

Perceived egocentric distance sensitivity and invariance across scene-selective cortex

The parahippocampal place area (PPA) is one of several brain regions that respond more strongly to scenes than to non-scene items such as objects and faces. The mechanism underlying this scene-preferential response remains unclear. One possibility is that the PPA is tuned to low-level stimulus features that are found more often in scenes than in less-preferred stimuli. Supporting this view, Nasr et al. (2014) recently observed that some of the stimuli that are known to strongly activate the PPA contain a large number of rectilinear edges. They further demonstrated that PPA response is modulated by rectilinearity for a range of non-scene images. Motivated by these results, we tested whether rectilinearity suffices to explain PPA selectivity for scenes. In the first experiment, we replicated the previous finding of modulation by rectilinearity in the PPA for arrays of 2-d shapes. However, two further experiments failed to find a rectilinearity effect for faces or scenes: high-rectilinearity faces and scenes did not activate the PPA any more strongly than low-rectilinearity faces and scenes. Moreover, the categorical advantage for scenes vs. faces was maintained in the PPA and two other scene-selective regions—the retrosplenial complex (RSC) and occipital place area (OPA)—when rectilinearity was matched between stimulus sets. We conclude that selectivity for scenes in the PPA cannot be explained by a preference for low-level rectilinear edges.

The perception and imagery of landmarks activate similar content‐dependent brain areas, including occipital and temporo‐medial brain regions. However, how these areas interact during visual perception and imagery of scenes, especially when recollecting their spatial location, remains unknown. Here, we combined functional magnetic resonance imaging (fMRI), resting‐state functional connectivity (rs‐fc), and effective connectivity to assess spontaneous fluctuations and task‐induced modulation of signals among regions entailing scene‐processing, the primary visual area and the hippocampus (HC), responsible for the retrieval of stored information. First, we functionally defined the scene‐selective regions, that is, the occipital place area (OPA), the retrosplenial complex (RSC) and the parahippocampal place area (PPA), by using the face/scene localizer, observing that two portions of the PPA—anterior and posterior PPA—were consistently activated in all subjects. Second, the rs‐fc analysis (n = 77) revealed a connectivity pathway similar to the one described in macaques, showing separate connectivity routes linking the anterior PPA with RSC and HC, and the posterior PPA with OPA. Third, we used dynamic causal modelling to evaluate whether the dynamic couplings among these regions differ between perception and imagery of familiar landmarks during a fMRI task (n = 16). We found a positive effect of HC on RSC during the retrieval of imagined places and an effect of occipital regions on both RSC and pPPA during the perception of scenes. Overall, we propose that under similar functional architecture at rest, different neural interactions take place between regions in the occipito‐temporal higher‐level visual cortex and the HC, subserving scene perception and imagery.

The occipital place area (OPA) is a scene-selective region on the lateral surface of human occipitotemporal cortex that spatially overlaps multiple visual field maps, as well as portions of cortex that are not currently defined as retinotopic. Here we combined population receptive field modeling and responses to scenes in a representational similarity analysis (RSA) framework to test the prediction that the OPA's visual field map divisions contribute uniquely to the overall pattern of scene selectivity within the OPA. Consistent with this prediction, the patterns of response to a set of complex scenes were heterogeneous between maps. To explain this heterogeneity, we tested the explanatory power of seven candidate models using RSA. These models spanned different scene dimensions (Content, Expanse, Distance), low- and high-level visual features, and navigational affordances. None of the tested models could account for the variation in scene response observed between the OPA's visual field maps. However, the heterogeneity in scene response was correlated with the differences in retinotopic profiles across maps. These data highlight the need to carefully examine the relationship between regions defined as category-selective and the underlying retinotopy, and they suggest that, in the case of the OPA, it may not be appropriate to conceptualize it as a single scene-selective region.

Previous work has shown that different cortical regions are sensitive to objects of differing real-world sizes (Konkle & Oliva, 2012; Mullally & Maguire, 2011; Bainbridge et al, 2012). Whereas a region in the occipito-temporal sulcus (OTS) shows small object selectivity, scene-selective regions (parahippocampal place area, PPA; retrosplenial cortex, RSC; and occipital place area, OPA) show systematically more activity for objects that are large in the world than smaller objects. What is the nature of this scene-like representation of objects? Here, we study an affordance-based object property: "interaction space". Interaction space is the area around objects that users traverse when interacting with the object. This interaction may include calculations of grasping points and surface area of contact, and is not necessarily conjoined with size. To fully deconvolve object interaction from physical size, we conducted a 2 x 2 factor design fMRI study with four conditions: 1) small one-handed objects (i.e., hairbrush), 2) large one-handed objects (i.e., umbrella), 3) small two-handed objects (i.e., hamburger), and 4) large two-handed objects (i.e., guitar). Stimuli were matched for ground truth size (based on physical sizes and weights), subjective size from an online survey, color, luminance, spectral energy, and retinal size. Results from a 2-factor ANOVA on beta values in functionally-defined ROIs show PPA, OPA, and RSC are sensitive to interaction space regardless of size, while PPA, OPA, and OTS are also sensitive to size regardless of interaction space. Importantly, effect sizes for interaction space in the PPA, OPA, and RSC were greater than those for object size, while the reverse was true for the OTS. These results show that interaction space is a perceptual object property that is represented in scene-selective regions, and suggest an overlapping interaction-based representation of objects and scenes in the brain. Meeting abstract presented at VSS 2013

The Neurocognitive Basis of Spatial Reorientation

Biologically inspired deep convolutional neural networks (CNNs), trained for computer vision tasks, have been found to predict cortical responses with remarkable accuracy. However, the internal operations of these models remain poorly understood, and the factors that account for their success are unknown. Here we develop a set of techniques for using CNNs to gain insights into the computational mechanisms underlying cortical responses. We focused on responses in the occipital place area (OPA), a scene-selective region of dorsal occipitoparietal cortex. In a previous study, we showed that fMRI activation patterns in the OPA contain information about the navigational affordances of scenes; that is, information about where one can and cannot move within the immediate environment. We hypothesized that this affordance information could be extracted using a set of purely feedforward computations. To test this idea, we examined a deep CNN with a feedforward architecture that had been previously trained for scene classification. We found that responses in the CNN to scene images were highly predictive of fMRI responses in the OPA. Moreover the CNN accounted for the portion of OPA variance relating to the navigational affordances of scenes. The CNN could thus serve as an image-computable candidate model of affordance-related responses in the OPA. We then ran a series of in silico experiments on this model to gain insights into its internal operations. These analyses showed that the computation of affordance-related features relied heavily on visual information at high-spatial frequencies and cardinal orientations, both of which have previously been identified as low-level stimulus preferences of scene-selective visual cortex. These computations also exhibited a strong preference for information in the lower visual field, which is consistent with known retinotopic biases in the OPA. Visualizations of feature selectivity within the CNN suggested that affordance-based responses encoded features that define the layout of the spatial environment, such as boundary-defining junctions and large extended surfaces. Together, these results map the sensory functions of the OPA onto a fully quantitative model that provides insights into its visual computations. More broadly, they advance integrative techniques for understanding visual cortex across multiple level of analysis: from the identification of cortical sensory functions to the modeling of their underlying algorithms.

This study evaluates the generalization capabilities of computer vision (CV) models for detecting cow locations in indoor free-stall barns, with a focus on quantifying the resources required for effective deployment across different farms where camera angles and lighting conditions vary. Given the increasing reliance of precision livestock farming (PLF) on object localization techniques to monitor livestock health and optimize resource management, understanding how to deploy CV models with minimal resources while maintaining high performance is critical. To assess generalization, we investigated three key hypotheses: (1) Model generalization is equally influenced by changes in lighting conditions and camera angles; (2) Higher model complexity guarantees better generalization performance; and (3) Fine-tuning with custom initial weights trained on related tasks enhances final performance compared to using pre-trained weights. The study utilized the newly released COws LOcalization (COLO) dataset, comprising 1,254 images and 11,818 cow instances captured from five distinct imaging settings with varying camera angles and lighting conditions. The first hypothesis was tested by training models on images from specific camera angles and evaluating performance on new angles. The second hypothesis was examined by comparing the performance of five different model sizes (e.g., YOLO11n with 2.6M parameters vs. YOLO11x with 56.9M parameters) on the same dataset. The third hypothesis was evaluated by pre-training a model on a subset of images from a specific barn and fine-tuning it on another subset from a similar setting to simulate transferring the training outcome to a different location within the same barn. Results indicate that changes in lighting conditions have a negligible impact on model generalization, whereas shifting from a top-view to a side-view perspective significantly reduces performance (P < 0.001). Model complexity does not always correlate with improved generalization, as optimal complexity depends on the dataset and task. Additionally, fine-tuning with custom initial weights benefits small, simple models but provides no significant (P > 0.05) advantage for larger models compared to training from scratch. As artificial intelligence adoption grows, alongside rising concerns about its environmental impact, this study provides practical guidelines for PLF researchers to optimize CV model deployment with minimal resource consumption. It also highlights key generalization challenges and leverages the COLO dataset to drive further research in resource-efficient livestock monitoring.

The papers in this special section address the programs and services supported by graphical models in computer vision. This section explores the main challenges in this framework—modeling novel priors, learning, inference—and presents innovative solutions. The papers cover the aspects of modeling novel priors, inference algorithms and parameter learning methods in the context of higher order graphical models.

DeepSolar++: Understanding residential solar adoption trajectories with computer vision and technology diffusion models

Acoustic returns scattered from either surfaces or objects are typically interpreted using image processing techniques such as beamforming and computer vision models such as convolutional neural networks. However, image representations can be burdensome to generate, creating delayed responses in real-time scenarios, and difficult to parse from the scene-understanding perspective. This work investigates direct exploitation of the information content from streaming, near-raw sonar time series data for the purpose of scene characterization. Summary statistics drawn from the sonar performance estimation community and communities that study aural-based perception are used to identify scene characteristics such as flat sand, ripples, gravel beds, or sea grass, in side-looking synthetic aperture sonar (SAS) data. An empirical multi-class classification study is presented that explores the performance of seafloor characterization methods on time series versus imagery.

MP07-16 COMPUTER VISION ANALYSIS OF UPPER TRACT UROTHELIAL CARCINOMA TO PREDICT HIGH VS LOW GRADE PATHOLOGY

The line structure is an efficient form of representation and modeling for LiDAR point clouds, while the Line Structure Construction (LSC) method aims to extract complete and coherent line structures from complex 3D point clouds, thereby providing a foundation for geometric modeling, scene understanding, and downstream applications. However, traditional LSC methods often fall short in preserving both the geometric integrity and topological connectivity of line structures derived from such datasets. To address this issue, we propose the Geometry and Topology Preservable Line Structure Construction (GTP-LSC) method, based on the Encoding and Extracting Framework (EEF). First, in the encoding phase, point cloud features related to line structures are mapped into a high-dimensional feature space. A 3D U-Net is then employed to compute Subsets with Structure feature of Line (SSL) from the dense, unstructured, and noisy indoor LiDAR point cloud data. Next, in the extraction phase, the SSL is transformed into a 3D field enriched with line features. Initially extracted line structures are then constructed based on Morse theory, effectively preserving the topological relationships. In the final step, these line structures are optimized using RANdom SAmple Consensus (RANSAC) and Constructive Solid Geometry (CSG) to ensure geometric completeness. This step also facilitates the generation of complex entities, enabling an accurate and comprehensive representation of both geometric and topological aspects of the line structures. Experiments were conducted using the Indoor Laser Scanning Dataset, focusing on the parking garage (D1), the corridor (D2), and the multi-room structure (D3). The results demonstrated that the proposed GTP-LSC method outperformed existing approaches in terms of both geometric integrity and topological connectivity. To evaluate the performance of different LSC methods, the IoU Buffer Ratio (IBR) was used to measure the overlap between the actual and constructed line structures. The proposed method achieved IBR scores of 92.5% (D1), 94.2% (D2), and 90.8% (D3) for these scenes. Additionally, Precision, Recall, and F-Score were calculated to further assess the LSC results. The F-Score of the proposed method was 0.89 (D1), 0.92 (D2), and 0.89 (D3), demonstrating superior performance in both visual analysis and quantitative results compared to other methods.