TSAVD-CLIP: One visual encoder learn both temporal and spatial feat

Research on graphical perception of time series visualisations has focused on visual representation, and not on interaction. Even for visual representation, there has been limited study of the impact on users of visual encodings and the strengths and weaknesses of Cartesian and Polar coordinate systems. In order to address this research gap, we performed a comprehensive graphical perception study that measured the effectiveness of time series visualisations with different interactions, visual encodings and coordinate systems for several tasks. Our results show that, while positional and colour visual encodings were better for most tasks, area visual encoding performed better for data comparison. Most importantly, we identified that introducing interactivity within time series visualisations considerably enhances the user experience, without any loss of efficiency or accuracy. We believe that our findings can greatly improve the development of visual analytics tools using time series visualisations in a variety of domains.

Visualisations are often the entry point to information that supports stakeholders’ decision- and policy-making processes. Visual displays can employ either static, dynamic or interactive formats as well as various types of representations and visual encodings, which differently affect the attention, recognition and working memory of users. Despite being well-suited for expert audiences, current climate data visualisations need to be further improved to make communication of climate information more inclusive for broader audiences, including people with disabilities. However, the lack of evidence-based guidelines and tools makes the creation of accessible visualisations challenging, potentially leading to misunderstanding and misuse of climate information by users. Taking stock of visualisation challenges identified in a workshop by climate service providers, we review good practices commonly applied by other visualisation-related disciplines strongly based on users’ needs that could be applied to the climate services context. We show how lessons learned in the fields of user experience, data visualisation, graphic design and psychology make useful recommendations for the development of more effective climate service visualisations. This includes applying a user-centred design approach, using interaction in a suitable way in visualisations, paying attention to information architecture or selecting the right type of representation and visual encoding. The recommendations proposed here can help climate service providers reduce users’ cognitive load and improve their overall experience when using a service. These recommendations can be useful for the development of the next generation of climate services, increasing their usability while ensuring that their visual components are inclusive and do not leave anyone behind.

In order to obtain the accurate spatial position reconstruction of the drilling bit wear surface, a drilling bit posture measurement system based on visual encoder is proposed in this paper. A tool clamping device with adjustable posture is designed for 3D digital microscope to hold the drilling bit with outer diameter within 6-8mm. The adjustment of the spatial posture of the drill bit is completed by the cooperation of two rotating discs whose axes are perpendicular to each other. The shaft of the vertical rotating disk is consistent with that of the encoder, and its rotation angle is used as the reference and measured accurately. The experimental results show that the angle detection accuracy is 0.1 degree. The accurate angle information provides the basis for the machine.

Abstract. When two speeded response tasks are performed in close succession, performance on the second task is usually impaired. Recently, an impairment has also been observed when the second task required only the visual identi1cation of a stimulus. Thus, visual encoding seems to suffer from the need to share limited processing capacities with some processes in the 1rst task. Yet, it is still unclear which particular processes in the 1rst task compete with stimulus encoding in the second task. The present experiments further examined the in2uence of response planning and execution on visual encoding and found both content-nonspeci1c and content-speci1c interference. An event-coding account is proposed that posits structural and procedural overlap of responseplanning and stimulus-encoding mechanisms.

Gone full circle: A radial approach to visualize event-based networks in digital humanities

Two experiments tested the hypothesis that visual encoding of television messages is a relatively automatic process, whereas verbal encoding is a relatively controlled process. Subjects viewed 30 messages crossed on Production Pacing (slow, medium, fast) and Arousing Content (calming, arousing). It was argued that as pacing and arousal increased, the resources required to process the messages would increase, which would interfere with the controlled process of verbal encoding but not with the automatic process of visual encoding. As expected, visual recognition was not affected by the increased resource requirements, but verbal recognition declined.

This article proposes a novel multitask learning framework that integrates contrastive learning and natural language generation (NLG) to enhance medical image classification and report generation. The goal is to improve disease classification accuracy and interpretability in medical diagnostics. The model architecture consists of a Vision Transformer (ViT) as a visual encoder, a transformer-based text encoder, and a multimodal decoder. The visual encoder processes medical images, while the text encoder handles disease-related text prompts. These components are trained jointly using image-text contrastive loss and language generation loss. Evaluations on the MIMICCXR and Chexpert datasets show that the model with NLG (Plain + NLG) outperforms the baseline contrastive learning model (Plain) in disease classification. For example, in the MIMICCXR dataset, the accuracy for Atelectasis increased from 17.44%(Plain) to 41.5% (Plain + NLG), and for Cardiomegaly, it improved from 19.25% to 47.4%. In Chexpert, the accuracy for Atelectasis increased from 12.5% to 58.5%, and for Pleural Effusion, from 61.10% to 64.0%. The model also demonstrated improvements in F1 scores, particularly for complex diseases like Cardiomegaly and Consolidation. The proposed multitask framework effectively combines contrastive learning with NLG, leading to improved disease classification and medical report generation. This approach has potential clinical applications by enhancing AI’s interpretability and accuracy in medical decision-making.

Evaluation criteria like expressiveness and effectiveness favor optimal use of space and visual encoding channels in a single visualization. However, individually optimized views may be inconsistent with one another when presented as a set in rec-ommender systems and narrative visualizations. For example, two visualizations might use very similar color palettes for different data fields, or they might render the same field but in different scales. These inconsistencies in visualization sets can cause interpretation errors and increase the cognitive load on viewers trying to analyze a set of visualizations. We propose two high-level principles for evaluating visualization set consistency: (1) the same fields should be presented in the same way, (2) different fields should be presented differently. These two principles are operationalized as a set of constraints for common visual encoding channels (x, y, color, size, and shape) to enable automated visualization set evaluation. To balance global (visualization set) consistency and local (single visualization) effectiveness, trade-offs in space and visual encodings have to be made. We devise an effectiveness preservation score to guide the selection of which conflicts to surface and potentially revise for sets of quantitative and ordinal encodings and a palette resource allocation mechanism for nominal encodings.

Most visualization recommendation systems predominantly rely on graphical previews to describe alternative visual encodings. However, since InfoVis novices are not familiar with visual representations (e.g., interpretation barriers [GTS10]), novices might have difficulty understanding and choosing recommended visual encodings. As an initial step toward understanding effective representation methods for visualization recommendations, we investigate the effectiveness of three representation methods (i.e., previews, animated transitions, and textual descriptions) under scatterplot construction tasks. Our results show how different representations individually and cooperatively help users understand and choose recommended visualizations, for example, by supporting their expect‐and‐confirm process. Based on our study results, we discuss design implications for visualization recommendation interfaces.

Recent brain imaging studies have shown that estrogens alter brain activation patterns upon working memory tasks in postmenopausal women. Estrogens, however, have many systemic side effects. We investigated the effect of the Selective estrogen receptor modulator (SERM) raloxifene on brain activation patterns during a memory task in postmenopausal women with functional magnetic resonance imaging (fMRI). Twenty postmenopausal and right handed women (mean age 65.7 years; SD 3.0) were included in this double blind, placebo controlled and randomized study. Whole brain fMRI was performed before and after three months of daily treatment with raloxifene 60 mg or placebo. Each scanning session consisted of a visual encoding task, a recognition test and a simple photic stimulation test. Data analyses was performed with SPM99b software. Specific regions of interest for the tasks were defined based on previous experiments. Visual encoding activated the ventral route, posterior medial temporal lobe and frontal cortex in both groups. Treatment interactions for raloxifene compared to placebo were a decrease in activation in the left parahippocampal gyrus and left lingual gyrus, and an increase in activation in the right superior frontal gyrus. The mean recognition test and the simple photic stimulation test showed no treatment interactions. Our results show that raloxifene affects brain activation patterns upon visual encoding in postmenopausal women.

Existing few-shot learning models leverage vision-language pre-trained models to alleviate the data scarcity problem. However, such models usually process visual and text information separately, which causes still inherent disparities between cross-modal features. Therefore, we propose the ProFusion model, which leverages multimodal pre-trained models and prototypical networks to construct multiple prototypes. Specifically, ProFusion generates image and text prototypes symmetrically using the visual encoder and text encoder, while integrating visual and text information through the fusion module to create more expressive multimodal feature fusion prototypes. Additionally, we introduce the alignment module to ensure consistency between image and text prototypes. During inference, ProFusion calculates the similarity of test images to the three types of prototypes separately and applies a weighted sum to generate the final prediction. Experiments demonstrate that ProFusion performs outstanding classification tasks on 15 benchmark datasets.

Decision-making is a central yet under-defined goal in visualization research. While existing task models address decision processes, they often neglect the conditions framing a decision. To better support decision-making tasks, we propose a characterization scheme that describes decision problems through key properties of the data, users, and task context. This scheme helps visualization researchers specify decision-support claims more precisely and informs the design of appropriate visual encodings and interactions. We demonstrate the utility of our approach by applying it to characterize decision tasks targeted by existing design studies, highlighting opportunities for future research in decision-centric visualization.

To address the issues of modeling the relationships between multiple local region objects in images and enhancing local region features, as well as mapping global image semantics to global text semantics and local region image semantics to local text semantics, a novel image captioning method based on CLIP and integrating local feature enhancement and multi-scale semantic guidance is proposed. The model employs ViT as the global visual encoder, Faster R-CNN as the local region visual encoder, BERT as the text encoder, and GPT-2 as the text decoder. By constructing a KNN graph of local image features, the model models the relationships between local region objects and then enhances the local region features using a graph attention network. Additionally, a multi-scale semantic guidance method is utilized to calculate the global and local semantic weights, thereby improving the accuracy of scene description and attribute detail description generated by the GPT-2 decoder. Evaluated on MSCOCO and Flickr30k datasets, the model achieves a significant improvement in the core metric CIDEr over established strong baselines, with 4.7% higher CIDEr than OFA on MSCOCO, and 16.6% higher CIDEr than Unified VLP on Flickr30k. Ablation studies and qualitative analysis validate the effectiveness of each proposed module.

In a delayed-response task, response selection marks an important transition from sensory to motor processing. Using event-related functional magnetic resonance imaging, we imaged the human brain during performance of a novel delayed-saccade task that isolated response selection from visual encoding and motor execution. The frontal eye fields (FEFs) and intraparietal sulcus (IPS) both showed robust contra-lateralized activity time-locked to response selection. Moreover, response selection affected delay-period activity differently in these regions; it persisted throughout the memory delay period following response selection in the FEF but not IPS. Our results indicate that the FEF and IPS both make important but distinct contributions to spatial working memory. The mechanism that the FEF uses to support spatial working memory is tied to the selection and prospective coding of saccade goals, whereas the role of the IPS may be more tied to retrospective coding of sensory representations.

Spiking neural networks are in theory more computationally powerful than rate-based neural networks often used in deep learning architectures. However, unlike rate-based neural networks, it is yet unclear how to train spiking networks to solve complex problems. There are still no standard algorithms and it is preventing roboticists to use spiking networks, yielding a lack of Neurorobotics applications. The contribution of this paper is twofold. First, we present a modular framework to evaluate neural self-driving vehicle applications. It provides a visual encoder from camera images to spikes inspired by the silicon retina (DVS), and a steering wheel decoder based on an agonist antagonist muscle model. Secondly, using this framework, we demonstrate a spiking neural network which controls a vehicle end-to-end for lane following behavior. The network is feed-forward and relies on hand-crafted feature detectors. In future work, this framework could be used to design more complex networks and use the evaluation metrics for learning.