Development of Visual Data Acquisition Systems of Household Objects

Object extraction is one of the important and chal-lenging tasks in the computer vision and/or robotics ? elds.This task is to extract the object from the scene using anypossible cues. The scenario discussed in this study was the objectextraction which considering the Space of Interest (SOI), i.e.,the three dimensional area where the object probably existed.To complete such task, the object extraction method based onthe probabilistic maps of multiple cues was proposed. Thanksto the Kinect V2 sensor, multiple cues such as color, depth, andnear-infrared information can be acquired simultaneously. TheSOI was modeled by a simple probabilistic model by consideringthe geometry of the possible objects and the reachability of thesystem acquired from depth information. To model the color andnear-infrared information, a Gaussian mixture models (GMM)was used. All of the models were combined to generate theprobabilistic maps that were used to extract the object fromthe scene. To validate the proposed object extraction, severalexperiments were conducted to investigate the best combinationof the cues used in this study.

In this paper, we propose a background subtraction (BGS) method based on the Gaussian mixture models using color and depth information. For combining color and depth information, we used the probabilistic model based on Gaussian distribution. In particular, we focused on solving color camouflage problem and depth denoising. For evaluating our method, we built a new dataset containing normal, color camouflage and depth camouflage situations. The dataset files consist of color, depth and ground truth image sequences. With these files, we compared the proposed algorithm with the conventional color-based BGS techniques in terms of precision, recall and F-measure. As a result, our method showed the best performance. Thus, this technique will help to robustly detect regions of interest as pre-processing in high-level image processing stages.

Reliable and real-time crowd counting is one of the most important tasks in intelligent visual surveillance systems. Most previous works only count passing people based on color information. Owing to the restrictions of color information influences themselves for multimedia processing, they will be affected inevitably by the unpredictable complex environments (e.g. illumination, occlusion, and shadow). To overcome this bottleneck, we propose a new algorithm by multimodal joint information processing for crowd counting. In our method, we use color and depth information together with a ordinary depth camera (e.g. Microsoft Kinect). Specifically, we first detect each head of the passing or still person in the surveillance region with adaptive modulation ability to varying scenes on depth information. Then, we track and count each detected head on color information. The characteristic advantage of our algorithm is that it is scene adaptive, which means the algorithm can be applied into all kinds of different scenes directly without additional conditions. Based on the proposed approach, we have built a practical system for robust and fast crowd counting facing complicated scenes. Extensive experimental results show the effectiveness of our proposed method.

Drawing inspiration from haptic exploration of objects by humans, the current work proposes a novel framework for robotic tactile object recognition, where visual information in the form of a set of visually interesting points is employed to guide the process of tactile data acquisition. Neuroscience research confirms the integration of cutaneous data as a response to surface changes sensed by humans with data from joints, muscles, and bones (kinesthetic cues) for object recognition. On the other hand, psychological studies demonstrate that humans tend to follow object contours to perceive their global shape, which leads to object recognition. In compliance with these findings, a series of contours are determined around a set of 24 virtual objects from which bimodal tactile data (kinesthetic and cutaneous) are obtained sequentially and by adaptively changing the size of the sensor surface according to the object geometry for each object. A virtual Force Sensing Resistor array (FSR) is employed to capture cutaneous cues. Two different methods for sequential data classification are then implemented using Convolutional Neural Networks (CNN) and conventional classifiers, including support vector machines and k-nearest neighbors. In the case of conventional classifiers, we exploit contourlet transformation to extract features from tactile images. In the case of CNN, two networks are trained for cutaneous and kinesthetic data and a novel hybrid decision-making strategy is proposed for object recognition. The proposed framework is tested both for contours determined blindly (randomly determined contours of objects) and contours determined using a model of visual attention. Trained classifiers are tested on 4560 new sequential tactile data and the CNN trained over tactile data from object contours selected by the model of visual attention yields an accuracy of 98.97% which is the highest accuracy among other implemented approaches.

Evidence for a specific role for muscarinic receptors in crossmodal object recognition in rats

This paper presents our approach to using a stereo camera to obtain 3-D image data to be used to improve existing lip boundary detection techniques. We show that depth information as provided by our approach can be used to significantly improve boundary detection systems. Our system detects the face and mouth area in the image by using color, geometric location, and additional depth information for the face. Initially, color and depth information can be used to localize the face. Then we can determine the lip region from the intensity information and the detected eye locations. The system has successfully been used to extract approximate lip regions using RGB color information of the mouth area. Merely using color information is not robust because the quality of the results may vary depending on light conditions, background, and the human race. To overcome this problem, we used a stereo camera to obtain 3-D facial images. 3-D data constructed from the depth information along with color information can provide more accurate lip boundary detection results as compared to color only based techniques.

The ghost imaging is a new field in the research of the quantum nature of light. The long-distance reflective ghost imaging experiment with thermal light is suit for remote sensing with a high practicability. A real-time data acquisition (DAQ) system is presented for the experiment. In the setup, the light from source is separated into signal beam and reference beam. The signal beam which illuminates the target is collected by a bucket detector. The reference beam is directly captured by an area detector. The DAQ system employs a photo-multiplier tube as the bucket detector and a charge-coupled device camera as the area detector. The two detectors separately detect intensity fluctuations of two light beams. Meanwhile, the DAQ system implements high-speed and high-precision data acquisition and data processing to reconstruct the image of the target. The results of electronic tests and experiments show that the DAQ system satisfies the requirements of the experiment.

Visual information, in the form of images and video, comes from the interaction of light with objects. Illumination is a fundamental element of visual information. Detecting and interpreting illumination effects is part of our everyday life visual experience. Shading for instance allows us to perceive the three-dimensional nature of objects. Shadows are particularly salient cues for inferring depth information. However, we do not make any conscious or unconscious effort to avoid them as if they were an obstacle when we walk around. Moreover, when humans are asked to describe a picture, they generally omit the presence of illumination effects, such as shadows, shading, and highlights, to give a list of objects and their relative position in the scene. Processing visual information in a way that is close to what the human visual system does, thus being aware of illumination effects, represents a challenging task for computer vision systems. Illumination phenomena interfere in fact with fundamental tasks in image analysis and interpretation applications, such as object extraction and description. On the other hand, illumination conditions are an important element to be considered when creating new and richer visual content that combines objects from different sources, both natural and synthetic. When taken into account, illumination effects can play an important role in achieving realism. Among illumination effects, shadows are often integral part of natural scenes and one of the elements contributing to naturalness of synthetic scenes. In this thesis, the problem of extracting shadows from digital images is discussed. A new analysis method for the segmentation of cast shadows in still and moving images without the need of human supervision is proposed. The problem of separating moving cast shadows from moving objects in image sequences is particularly relevant for an always wider range of applications, ranging from video analysis to video coding, and from video manipulation to interactive environments. Therefore, particular attention has been dedicated to the segmentation of shadows in video. The validity of the proposed approach is however also demonstrated through its application to the detection of cast shadows in still color images. Shadows are a difficult phenomenon to model. Their appearance changes with changes in the appearance of the surface they are cast upon. It is therefore important to exploit multiple constraints derived from the analysis of the spectral, geometric and temporal properties of shadows to develop effective techniques for their extraction. The proposed method combines an analysis of color information and of photometric invariant features to a spatio-temporal verification process. With regards to the use of color information for shadow analysis, a complete picture of the existing solutions is provided, which points out the fundamental assumptions, the adopted color models and the link with research problems such as computational color constancy and color invariance. The proposed spatial verification does not make any assumption about scene geometry nor about object shape. The temporal analysis is based on a novel shadow tracking technique. On the basis of the tracking results, a temporal reliability estimation of shadows is proposed which allows to discard shadows which do not present time coherence. The proposed approach is general and can be applied to a wide class of applications and input data. The proposed cast shadow segmentation method has been evaluated on a number of different video data representing indoor and outdoor real-world environments. The obtained results have confirmed the validity of the approach, in particular its ability to deal with different types of content and its robustness to different physically important independent variables, and have demonstrated the improvement with respect to the state of the art. Examples of application of the proposed shadow segmentation tool to the enhancement of video object segmentation, tracking and description operations, and to video composition, have demonstrated the advantages of a shadow-aware video processing.

Computerized data acquisition systems are often the principle method of recording experimental results. This book takes the reader step-by-step through the process of data acquisition and analysis, explaining how to set up the systems and then obtain useful information from the data recorded. In an easy-to-read style, it guides researchers through the basics of data acquisition systems, explains the important underlying concepts, and gives numerous examples of how to analyze the recorded information. While aimed at researchers in the life sciences, the topics covered are general and will be valuable to anyone interested in learning to use data acquisition systems. The principles can be applied to the collection of data from respiratory apparatus to behavioral science experiments, and a host of other situations. Many illustrations and worked examples accompany the text, and the mathematics are kept simple. This book is an invaluable tool for the non-engineer who is collecting and analyzing experimental data using data acquisition systems.

This paper presents the application of 3D object reconstruction in orbital proximity operations. This promising novel technology is proposed to improve both Human Machine Interfaces (HMI) and autonomous algorithms for Guidance, Navigation and Control (GNC) in terms of situation awareness, docking efficiency and resource consumption. During this study a software framework was developed which implements a flexible real-time-capable toolchain to perform all necessary tasks for 3D object reconstruction. A driver module reads and filters the data stream from a given optical sensor (e.g. stereo camera or combined visual camera and infrared time-of-flight sensor). Image maps and depth information are then provided to computer vision algorithms for Simultaneous Localization and Mapping (SLAM) and algorithms for 3D reconstruction. As an output these algorithms generate a 3D point cloud and a 3D mesh that can be displayed to the human operator, fed into GNC algorithms or further processed to generate adequate surface models for visualization and inspection. This concept was verified in the Robotic Actuation and On-Orbit Navigation Laboratory (RACOON-Lab), a simulation environment for end-to-end technology development and evaluation for close-range proximity operations. A sub-scale hardware mock-up of a geostationary target satellite attached to the RACOON-Lab facility was successfully reconstructed using the described setup. During the simulated maneuver a rotating target satellite was observed by the sensors attached to the simulated chasing satellite. The software was executed on the embedded computer which is part of the facility. The cameras Kinect v2 and ZED produced adequate 3D reconstructions in intervals of less than 10 seconds. The Kinect v2 generates more accurate structures and includes more details, whereas the ZED results in a better color fidelity. Both cameras were sensitive to changes of lighting conditions. For longer acquisition times, drift caused by uncertainties in the pose estimation decreases the quality of the reconstruction significantly.

Kinect is a motion sensing input device which is widely used in computer vision and other related fields. However, there are many inaccurate depth data in Kinect depth images even Kinect v2. In this paper, an algorithm is proposed to enhance Kinect v2 depth images. According to the principle of its depth measuring, the foreground and the background are considered separately. As to the background, the holes are filled according to the depth data in the neighborhood. And as to the foreground, a filling algorithm, based on the color image concerning about both space and color information, is proposed. An adaptive joint bilateral filtering method is used to reduce noise. Experimental results show that the processed depth images have clean background and clear edges. The results are better than ones of traditional Strategies. It can be applied in 3D reconstruction fields to pretreat depth image in real time and obtain accurate results.

Visual information is processed in the brain primarily through two distinct pathways, the dorsal and the ventral visual streams. The present functional magnetic resonance imaging study investigated the specialization and integration of dorsal and ventral streams using tasks of object recognition and location detection. The study included 22 healthy adult volunteers who viewed stimuli consisting of grayscale photographs of common household objects presented in blocked design. Participants were asked to either recognize an object or to locate its position. While the location detection task elicited greater activation in the dorsal visual stream, recognizing objects showed greater activation in the middle occipital gyri, left inferior temporal gyrus, and in the left inferior frontal gyrus. The integration between dorsal and ventral brain areas was stronger during location detection than during object recognition. In addition, a principal components analysis found preliminary evidence for a group of regions, such as frontal and parietal cortex, working together in this task. Overall, the results of this study indicate the existence of specialized modules for object recognition and location detection, and possible interactions between areas beyond the visual cortex that may play a role in such tasks.

The location and intensity of non-resolved space object can be obtained in space-based surveillance, however, the material, size, and status of the object are lost. Spectrum represents the inherent property difference of the object, which can be used as an important means for feature extraction and recognition of non-resolved space object. According to the influence factors, including material, structure, background and orbit, a mathematical model for spectral properties of space object is established. Based on the model, inverse calculation method for feature extraction and recognition of space object is proposed. Taking the HuanJing-1 satellite scale model as an example, experimental verification for feature extraction and recognition of space object in typical parameters is made. Experimental results demonstrate the validity of the modeling method.

Simultaneous localization and mapping (SLAM) is becoming one of the most attractive research focuses of robot control and visual processing. In this paper, robot performs a SLAM mission in an unknown and structured indoor environment with the Microsoft XBOX Kinect obtaining visual and depth information. The line features of the object as mapping elements are extracted from visual images, and according to the extraction result, the distance between line features of the object and the robot can be obtained to portray the object edge onto the map in 2D, and the distance is provided by depth data from the Kinect. Meanwhile the robot motion model is created for the trajectory plan. A method using the extended Kalman filter (EKF) is applied to provide a pose estimate for the robot motion trajectory. The SLAM strategy is demonstrated in simulation and experimental environment.

This paper presents a detailed framework for Gaussian mixture model (GMM)-based articulatory inversion equipped with special postprocessing smoothers, and with the capability to perform audio-visual information fusion. The effects of different acoustic features on the GMM inversion performance are investigated and it is shown that the integration of various types of acoustic (and visual) features improves the performance of the articulatory inversion process. Dynamic Kalman smoothers are proposed to adapt the cutoff frequency of the smoother to data and noise characteristics; Kalman smoothers also enable the incorporation of auxiliary information such as phonetic transcriptions to improve articulatory estimation. Two types of dynamic Kalman smoothers are introduced: global Kalman (GK) and phoneme-based Kalman (PBK). The same dynamic model is used for all phonemes in the GK smoother; it is shown that GK improves the performance of articulatory inversion better than the conventional low-pass (LP) smoother. However, the PBK smoother, which uses one dynamic model for each phoneme, gives significantly better results than the GK smoother. Different methodologies to fuse the audio and visual information are examined. A novel modified late fusion algorithm, designed to consider the observability degree of the articulators, is shown to give better results than either the early or the late fusion methods. Extensive experimental studies are conducted with the MOCHA database to illustrate the performance gains obtained by the proposed algorithms. The average RMS error and correlation coefficient between the true (measured) and the estimated articulatory trajectories are 1.227 mm and 0.868 using audiovisual information fusion and GK smoothing, and 1.199 mm and 0.876 using audiovisual information fusion together with PBK smoothing based on a phonetic transcription of the utterance.