Visual Learning of Affordance Based Cues

This work is about the relevance of Gibson's concept of affordances [1] for visual perception in interactive and autonomous robotic systems. In extension to existing functional views on visual feature representations, we identify the importance of learning in perceptual cueing for the anticipation of opportunities for interaction of robotic agents. We investigate how the originally defined representational concept for the perception of affordances - in terms of using either optical flow or heuristically determined 3D features of perceptual entities - should be generalized to using arbitrary visual feature representations. In this context we demonstrate the learning of causal relationships between visual cues and predictable interactions, and emphasize on a novel framework for cueing and hypothesis verification of affordances that could play an important role in future robot control architectures. We argue that affordance based perception should enable systems to react to environment stimuli both more efficient and autonomous, and provide a potential to plan on the basis of responses to more complex perceptual configurations. We verify the concept with a concrete implementation applying state-of-the-art visual descriptors and regions of interest within a simulated robot scenario and prove that these features were successfully selected for predicting opportunities of robot interaction.

Recently, the aspect of visual perception has been explored in the context of Gibson's concept of affordances [1] in various ways [4-9]. In extension to existing functional views on visual feature representations, we focus on the importance of learning in perceptual cueing for the anticipation of opportunities for interaction of robotic agents. Furthermore, we propose that the originally defined representational concept for the perception of affordances - in terms of using either optical flow or heuristically determined 3D features of perceptual entities - should be generalized towards using arbitrary visual feature representations. In this context we demonstrate the learning of causal relationships between visual cues and associated anticipated interactions, using visual information within the framework of Markov Decision Processes (MDPs). We emphasize a new framework for cueing and recognition of affordance-like visual entities that could play an important role in future robot control architectures. Affordance-like perception should enable systems to react to environment stimuli both more efficiently and autonomously, and provide a potential to plan on the basis of relevant responses to more complex perceptual configurations. We verify the concept with a concrete implementation of learning visual cues by reinforcement, applying state-of-the-art visual descriptors and regions of interest that were extracted from a simulated robot scenario and prove that these features were successfully selected for their relevance in predicting opportunities of robot interaction.

Exploiting the relevant speech information that is embedded in facial images has been a significant research topic in recent years, because it has provided complementary information to acoustic signals for a wide range of automatic speech recognition (ASR) tasks. Visual information is particularly important in many real applications where acoustic signals are corrupted by environmental noises. This chapter reviews the most recent advances in feature extraction and representation for Visual Speech Recognition (VSR). In comparison with other surveys published in the past decade, this chapter presents a more up-to-date survey and highlights the strengths of two newly developed approaches (i.e., graph-based learning and deep learning) for VSR. In particular, we summarise the methods of using these two techniques to overcome one of the most challenging difficulties in this area-that is, how to automatically learn good visual feature representations from facial images to replace the widely used handcrafted features. This chapter concludes by discussing potential visual feature representation solutions that may overcome the remaining challenges in this domain.

The ever-increasing numbers of medical images in digital format are generated in clinical practice every day. These images include high-resolution and temporal data of various modalities and which constitute an important source of anatomical and functional information for diagnosis of diseases, research and education. So how to manage these large data and utilize them effectively and efficiently possess significant technical challenges. Thus, the technique of Content-based Medical Image Retrieval (CBMIR) emerges as the times require. However, current CBMIR is not sufficient to capture the semantic content of an image and difficult to provide good results according to the predefined categories in the medical domain for less using the medical knowledge. Accordingly, in this paper a mapping modelling of visual feature and knowledge representation is proposed to approach for medical image retrieval. Firstly, the low-level fusion visual features are extracted based on intensity, texture, and their extended versions. Secondly, a set of disjoint semantic tokens with appearance in lung CT images is selected to define a vocabulary based on medical knowledge representation. Finally, a mapping modeling is investigated to associate low-level visual image features with their high-level semantic. Experiments are conducted with a medical image database consisting of 220 lung medical images obtained from the Heilongjiang Province Hospital. And the comparison of the retrieval results shows that the approach proposed in this paper is effective.

Human motion transfer aims to animate the pose of a human in a source image driven by the poses of a human in a target video. To warp (transfer) human poses, most of the existing methods are based on optical flow or affine transformations as an intermediate representation followed by a generator module to perform the motion transfer. Existing methods perform well in terms of reconstruction quality. However, the quality of the human pose transfer has received less attention although it is an important part of the motion transfer process. Therefore, in this paper, we propose a method focusing on both the reconstruction quality as well as pose consistency. In contrast to existing methods, performing warping procedures in 2D- or 3D-space, we introduce a strategy to combine the warped features in both 2D- and 3D-space to alleviate the self-occlusion problem. In this way, our method benefits from 2D (robustness) and 3D (steering) information to guide the generation process. To reduce the pose error caused by inaccurate 3D estimation, a method is proposed to maintain semantic consistency between predictions and target images at arm and leg regions. Experiments conducted on large scale datasets show that the proposed method outperforms existing methods. Ablation studies clarify the benefits of using feature fusion and semantic consistency.

Content-based image retrieval (CBIR) has become one of the most active research areas in the past few years. Many visual feature representations have been explored and many systems built. While these research efforts establish the basis of CBIR, the usefulness of the proposed approaches is limited. Specifically, these efforts have relatively ignored two distinct characteristics of CBIR systems: (1) the gap between high level concepts and low level features; (2) subjectivity of human perception of visual content. This paper proposes a relevance feedback based interactive retrieval approach, which effectively takes into account the above two characteristics in CBIR. During the retrieval process, the user's high level query and perception subjectivity are captured by dynamically updated weights based on the user's relevance feedback. The experimental results show that the proposed approach greatly reduces the user's effort of composing a query and captures the user's information need more precisely.

It's well known that visual information such as lip shape and its movement can indicate what the speaker is talking about. In this paper, we present an automatic lipreading system solely using visual information for recognizing isolated English digits from 0 to 9. A parameter set of a 14-point ASM lip model is used to describe the outer lip contour. The inner mouth information such as the teeth region and the mouth opening are also extracted. With appropriate normalization, the feature vectors containing the normalized outer lip features, inner mouth features and also their first order derivatives are obtained for training the HMM word models. Experiments have been carried out to investigate the recognition performance using our visual feature set compared with other traditional visual feature representations. An accuracy of 93% for speaker dependent recognition and 84% for speaker independent recognition is achieved using our visual feature representation. A real-time automatic lipreading system has been successfully implemented on a 1.9-GHz PC.

Color histogram is one of the most widely used visual feature representations the JPEG/MPEG compressed domains, the DC coefficients are used commonly to form the color histograms without fully decompressing the image or the video bit stream. In this paper, we address the issues arising from the adaptation of DC based color histograms to the arbitrarily shaped MPEG-4 video objects. More specifically, we discuss the color space selection, quantization, and histogram computation with the consideration of the specific characteristics of the MPEG-4 video objects. We also propose a method for reducing the chroma keying artifacts that may occur at the boundaries of the objects. The experimental results show that great retrieval performance improvements are achieved by employing the proposed method in the presence of such artifacts.KeywordsColor SpaceColor HistogramRetrieval PerformanceRetrieval ResultBoundary BlockThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Interactive Machine Learning considers that a robot is learning with and/or from a human. In this paper, we investigate the impact of human social traits on the robot learning. We explore social traits such as age (children vs. adult) and pathology (typical developing children vs. children with autistic spectrum disorders). In particular, we consider learning to recognize both postures and identity of a human partner. A human-robot posture imitation learning, based on a neural network architecture, is used to develop a multi-task learning framework. This architecture exploits three learning levels : 1) visual feature representation, 2) posture classification and 3) human partner identification. During the experiment the robot interacts with children with autism spectrum disorders (ASD), typical developing children (TD) and healthy adults. Previous works assessed the impact on learning of these social traits at the group level. In this paper, we focus on the analysis of individuals separately. The results show that the robot is impacted by the social traits of these different groups' individuals. First, the architecture needs to learn more visual features when interacting with a child with ASD (compared to a TD child) or with a TD child (compared to an adult). However, this surplus in the number of neurons helped the robot to improve the TD children's posture recognition but not that of children with ASD. Second, preliminary results show that this need of a neurons surplus while interacting with children with ASD is also generalizable to the identity recognition task.

Content-Based Image Retrieval (CBIR) allows to automatically extracting target images according to objective visual contents of the image itself. Representation of visual features and similarity match are important issues in CBIR. Color, texture and shape information have been the primitive image descriptors in content-based image retrieval systems. This paper presents a fast and efficient image indexing and search system based on color and texture features. The color features are represented by combines 2-D histogram and statistical moments and texture features are represented by combines the direction measure and a gray level co-occurrence matrix (GLCM). The detailed experimental analysis is carried out using precision and recall ondataset Wang. The time analysis is also performed to compare processing speeds of the proposed method with the existing similar best. The experimental results demonstrated that proposed method achieved a better image recognition, and the accuracy of classification based on this method has been significantly improved. The performance is measured in terms of recall and precision; also the obtained performances are compared with several state-of-the-art algorithms and showed that our algorithm is simple, fast, and efficient in terms of results and memory. Keywords: Co-occurrence Matrix, Direction Measure, 2-D Histogram, Statistical Moments, CBIR, GLCM

In heavy ion radiation therapy for cancer, tracking the region of interest (ROI) is important in order to monitor the position of the treatment target, which varies with respect to respiration, during therapy. We previously proposed a method to compute the optical flow, which is a 2-D velocity field, and the locus of the ROI in the internal organs using the optical flow and its reliability. In the present work, this locus is assumed to be prior knowledge and an on-line tracking method of the ROI is proposed. In this method, the ROI position is predicted using the estimated locus, and the template matching technique using the respiratory information is applied. The locus is then updated using the template matching result obtained via the Kalman filter. The effectiveness and reliability of this method are confirmed through real image experiments.

The brain represents visual objects with topographic cortical patterns. To address how distributed visual representations enable object categorization, we established predictive encoding models based on a deep residual network, and trained them to predict cortical responses to natural movies. Using this predictive model, we mapped human cortical representations to 64,000 visual objects from 80 categories with high throughput and accuracy. Such representations covered both the ventral and dorsal pathways, reflected multiple levels of object features, and preserved semantic relationships between categories. In the entire visual cortex, object representations were organized into three clusters of categories: biological objects, non-biological objects, and background scenes. In a finer scale specific to each cluster, object representations revealed sub-clusters for further categorization. Such hierarchical clustering of category representations was mostly contributed by cortical representations of object features from middle to high levels. In summary, this study demonstrates a useful computational strategy to characterize the cortical organization and representations of visual features for rapid categorization.

A Collaborative Process for Developing Map Symbol Standards

The purpose of this study was to evaluate a recently developed hardware and software system for CT scanning that generates images in real time and switches to helical CT scanning by either a visual cue or a region of interest (ROI) amplitude threshold. We randomly and prospectively divided 120 abdominal CT examinations into three groups. Two groups received 75 ml of contrast agent at 1.5 ml/sec. Helical CT scanning began after visualization of the contrast bolus arrival in the hepatic veins (visual cue triggering) (39 patients) or after reaching an ROI threshold (automated ROI threshold triggering) (39 patients). A third group served as a control group (42 patients) and received 150 ml of contrast agent at 1 ml/sec. Quality of hepatic enhancement was assessed objectively and subjectively. Comparisons were made after stratifying each group into three weight classes. Errors occurred in 12 (31%) of 39 examinations in the group with automated ROI threshold triggering. In that group, we found a significantly (p < .04) lower mean hepatic enhancement in two of three weight categories, and a significantly (p < .04) lower mean subjective scan quality in one of three weight categories, than we found in the group with visual cue triggering. Optimizing portal venous phase helical CT of the liver after a low-volume bolus of contrast agent and an injection rate of 1.5 ml/sec is best achieved by initiating helical CT scanning after visualizing the contrast bolus arrival within the liver rather than after reaching a preset attenuation threshold.

Differing from common 2D images, a texture map, since it is used to project onto a 3D model in 3D space, not only contains 2D texture information, but also implicitly associates certain 3D geometric information. Related to this, an effective 3D geometry-dependent texture map compression method with hybrid region of interest (ROI) coding is proposed in this paper. We regard the visually important area of the texture map as the ROI. To acquire the visually important areas of the texture map, we take into account information from both the 3D geometry and 2D texture maps, depicting the saliency of the textured model, distortion of the texture mapping, and boundary of the texture atlas. These visually important areas are expressed as a visual importance map. According to the particularity of the texture map, a hybrid ROI coding method that utilizes Max-Shift and an improved post compression rate distortion (PCRD) technique is presented, guided by this visual importance map. To find the exact wavelet coefficients pertaining to these ROIs before carrying out the hybrid ROI coding, this paper proposes a stochastic coefficient priority mask map computational method. Experimental results show that the visually important areas of the texture image have a better visual effect and that a good rendering result can be obtained from the texture mapping.