Spatio-temporal segmentation and object tracking

Visual information, in the form of images and video, comes from the interaction of light with objects. Illumination is a fundamental element of visual information. Detecting and interpreting illumination effects is part of our everyday life visual experience. Shading for instance allows us to perceive the three-dimensional nature of objects. Shadows are particularly salient cues for inferring depth information. However, we do not make any conscious or unconscious effort to avoid them as if they were an obstacle when we walk around. Moreover, when humans are asked to describe a picture, they generally omit the presence of illumination effects, such as shadows, shading, and highlights, to give a list of objects and their relative position in the scene. Processing visual information in a way that is close to what the human visual system does, thus being aware of illumination effects, represents a challenging task for computer vision systems. Illumination phenomena interfere in fact with fundamental tasks in image analysis and interpretation applications, such as object extraction and description. On the other hand, illumination conditions are an important element to be considered when creating new and richer visual content that combines objects from different sources, both natural and synthetic. When taken into account, illumination effects can play an important role in achieving realism. Among illumination effects, shadows are often integral part of natural scenes and one of the elements contributing to naturalness of synthetic scenes. In this thesis, the problem of extracting shadows from digital images is discussed. A new analysis method for the segmentation of cast shadows in still and moving images without the need of human supervision is proposed. The problem of separating moving cast shadows from moving objects in image sequences is particularly relevant for an always wider range of applications, ranging from video analysis to video coding, and from video manipulation to interactive environments. Therefore, particular attention has been dedicated to the segmentation of shadows in video. The validity of the proposed approach is however also demonstrated through its application to the detection of cast shadows in still color images. Shadows are a difficult phenomenon to model. Their appearance changes with changes in the appearance of the surface they are cast upon. It is therefore important to exploit multiple constraints derived from the analysis of the spectral, geometric and temporal properties of shadows to develop effective techniques for their extraction. The proposed method combines an analysis of color information and of photometric invariant features to a spatio-temporal verification process. With regards to the use of color information for shadow analysis, a complete picture of the existing solutions is provided, which points out the fundamental assumptions, the adopted color models and the link with research problems such as computational color constancy and color invariance. The proposed spatial verification does not make any assumption about scene geometry nor about object shape. The temporal analysis is based on a novel shadow tracking technique. On the basis of the tracking results, a temporal reliability estimation of shadows is proposed which allows to discard shadows which do not present time coherence. The proposed approach is general and can be applied to a wide class of applications and input data. The proposed cast shadow segmentation method has been evaluated on a number of different video data representing indoor and outdoor real-world environments. The obtained results have confirmed the validity of the approach, in particular its ability to deal with different types of content and its robustness to different physically important independent variables, and have demonstrated the improvement with respect to the state of the art. Examples of application of the proposed shadow segmentation tool to the enhancement of video object segmentation, tracking and description operations, and to video composition, have demonstrated the advantages of a shadow-aware video processing.

The growing popularity of high resolution video and the continuously increasing demands for high quality video on mobile devices are producing stronger needs for more efficient video encoder. Concerning these desires, HEVC, a newest video coding standard, has been developed by a joint team formed by ISO/IEO MPEG and ITU/T VCEG. Its design goal is to achieve a 50% compression gain over its predecessor H.264 with an equal or even higher perceptual video quality. Motion Estimation (ME) being as one of the most critical module in video coding contributes almost 50%-70% of computational complexity in the video encoder. This high consumption of the computational resources puts a limit on the performance of encoders, especially for full HD or ultra HD videos, in terms of coding speed, bit-rate and video quality. Thus the major part of this work concentrates on the computational complexity reduction and improvement of timing performance of motion estimation algorithms for HEVC standard. First, a new strategy to calculate the SAD (Sum of Absolute Difference) for motion estimation is designed based on the statistics on property of pixel data of video sequences. This statistics demonstrates the size relationship between the sum of two sets of pixels has a determined connection with the distribution of the size relationship between individual pixels from the two sets. Taking the advantage of this observation, only a small proportion of pixels is necessary to be involved in the SAD calculation. Simulations show that the amount of computations required in the full search algorithm is reduced by about 58% on average and up to 70% in the best case. Secondly, from the scope of parallelization an enhanced TZ search for HEVC is proposed using novel schemes of multiple MVPs (motion vector predictor) and shared MVP. Specifically, resorting to multiple MVPs the initial search process is performed in parallel at multiple search centers, and the ME processing engine for PUs within one CU are parallelized based on the MVP sharing scheme on CU (coding unit) level. Moreover, the SAD module for ME engine is also parallelly implemented for PU size of 32×32. Experiments indicate it achieves an appreciable improvement on the throughput and coding efficiency of the HEVC video encoder. In addition, the other part of this thesis is contributed to the VLSI architecture design for finding the first W maximum/minimum values targeting towards high speed and low hardware cost. The architecture based on the novel bit-wise AND scheme has only half of the area of the best reference solution and its critical path delay is comparable with other implementations. While the FPCG (full parallel comparison grid) architecture, which utilizes the optimized comparator-based structure, achieves 3.6 times faster on average on the speed and even 5.2 times faster at best comparing with the reference architectures. Finally the architecture using the partial sorting strategy reaches a good balance on the timing performance and area, which has a slightly lower or comparable speed with FPCG architecture and a acceptable hardware cost

Visual illusions occur around us all the time, which present in nature and appear when you least expect it. It is well known that visual illusions can have a dramatic effect upon our visual perception of an object 's size and position. However, whether visual illusions have a similar influence on visual guided action or not, it remains the subject of much debate. The purpose of this study is to explore whether the visual information used to identify and categorize an object would be similar to the visual information used to plan and control visually guided movements directed toward that object. The outcome of this study will be helpful for designers to understand our visual information system and apply that knowledge to the design practice. In order to achieve the purpose of this study mentioned above, three visual illusions as the experiment's apparatus are applied to examine the relative effect of visual illusions on both the visual perceived center of an object and the position of a grasp on that object when a balanced lift is required. The results of the experiment show that (1) Illusions influence both the perceived and the grasped estimates of the center position, but the grasp center is more veridical; (2) Judd-illusion, one of the three visual illusions, has a bigger degree of effect than two other illusions on both the perceived and the grasped estimates of the center position; (3) By providing the participants with a mean to guide their visual perception of these illusions, the errors from the estimation of the center position can be significantly reduced.

Multidimensional Signal, Image, and Video Processing and Coding

Distributed Video Coding (DVC) is a new paradigm for video compression based on the information theoretical results of Slepian and Wolf (SW), and Wyner and Ziv (WZ). While conventional coding has a rigid complexity allocation as most of the complex tasks are performed at the encoder side, DVC enables a flexible complexity allocation between the encoder and the decoder. The most novel and interesting case is low complexity encoding and complex decoding, which is the opposite of conventional coding. While the latter is suitable for applications where the cost of the decoder is more critical than the encoder's one, DVC opens the door for a new range of applications where low complexity encoding is required and the decoder's complexity is not critical. This is interesting with the deployment of small and battery-powered multimedia mobile devices all around in our daily life. Further, since DVC operates as a reversed-complexity scheme when compared to conventional coding, DVC also enables the interesting scenario of low complexity encoding and decoding between two ends by transcoding between DVC and conventional coding. More specifically, low complexity encoding is possible by DVC at one end. Then, the resulting stream is decoded and conventionally re-encoded to enable low complexity decoding at the other end. Multiview video is attractive for a wide range of applications such as free viewpoint television, which is a system that allows viewing the scene from a viewpoint chosen by the viewer. Moreover, multiview can be beneficial for monitoring purposes in video surveillance. The increased use of multiview video systems is mainly due to the improvements in video technology and the reduced cost of cameras. While a multiview conventional codec will try to exploit the correlation among the different cameras at the encoder side, DVC allows for separate encoding of correlated video sources. Therefore, DVC requires no communication between the cameras in a multiview scenario. This is an advantage since communication is time consuming (i.e. more delay) and requires complex networking. Another appealing feature of DVC is the fact that it is based on a statistical framework. Moreover, DVC behaves as a natural joint source-channel coding solution. This results in an improved error resilience performance when compared to conventional coding. Further, DVC-based scalable codecs do not require a deterministic knowledge of the lower layers. In other words, the enhancement layers are completely independent from the base layer codec. This is called the codec-independent scalability feature, which offers a high flexibility in the way the various layers are distributed in a network. This thesis addresses the following topics: First, the theoretical foundations of DVC as well as the practical DVC scheme used in this research are presented. The potential applications for DVC are also outlined. DVC-based schemes use conventional coding to compress parts of the data, while the rest is compressed in a distributed fashion. Thus, different conventional codecs are studied in this research as they are compared in terms of compression efficiency for a rich set of sequences. This includes fine tuning the compression parameters such that the best performance is achieved for each codec. Further, DVC tools for improved Side Information (SI) and Error Concealment (EC) are introduced for monoview DVC using a partially decoded frame. The improved SI results in a significant gain in reconstruction quality for video with high activity and motion. This is done by re-estimating the erroneous motion vectors using the partially decoded frame to improve the SI quality. The latter is then used to enhance the reconstruction of the finally decoded frame. Further, the introduced spatio-temporal EC improves the quality of decoded video in the case of erroneously received packets, outperforming both spatial and temporal EC. Moreover, it also outperforms error-concealed conventional coding in different modes. Then, multiview DVC is studied in terms of SI generation, which differentiates it from the monoview case. More specifically, different multiview prediction techniques for SI generation are described and compared in terms of prediction quality, complexity and compression efficiency. Further, a technique for iterative multiview SI is introduced, where the final SI is used in an enhanced reconstruction process. The iterative SI outperforms the other SI generation techniques, especially for high motion video content. Finally, fusion techniques of temporal and inter-view side informations are introduced as well, which improves the performance of multiview DVC over monoview coding. DVC is also used to enable scalability for image and video coding. Since DVC is based on a statistical framework, the base and enhancement layers are completely independent, which is an interesting property called codec-independent scalability. Moreover, the introduced DVC scalable schemes show a good robustness to errors as the quality of decoded video steadily decreases with error rate increase. On the other hand, conventional coding exhibits a cliff effect as the performance drops dramatically after a certain error rate value. Further, the issue of privacy protection is addressed for DVC by transform domain scrambling, which is used to alter regions of interest in video such that the scene is still understood and privacy is preserved as well. The proposed scrambling techniques are shown to provide a good level of security without impairing the performance of the DVC scheme when compared to the one without scrambling. This is particularly attractive for video surveillance scenarios, which is one of the most promising applications for DVC. Finally, a practical DVC demonstrator built during this research is described, where the main requirements as well as the observed limitations are presented. Furthermore, it is defined in a setup being as close as possible to a complete real application scenario. This shows that it is actually possible to implement a complete end-to-end practical DVC system relying only on realistic assumptions. Even though DVC is inferior in terms of compression efficiency to the state of the art conventional coding for the moment, strengths of DVC reside in its good error resilience properties and the codec-independent scalability feature. Therefore, DVC offers promising possibilities for video compression with transmission over error-prone environments requirement as it significantly outperforms conventional coding in this case.

The computational complexity issue is critical for present and future video applications implemented by relatively new video coding standards, such as the H.264/AVC (Advanced Video Coding), which has a large number of coding modes. One of the main reasons for the importance of providing an efficient complexity control in video coding applications is a strong need to decrease the encoding/decoding computational complexity, especially when the encoding and/or decoding devices are resource-limited, such as portable devices. In turn, efficient complexity control enables reducing the video coding processing time and enables saving power resources during the encoding and/or decoding process. Since the recent dramatic progress in the development of multimedia technologies has made portable devices widespread everywhere, especially in order to provide or receive real-time video contents, the need to enhance the computational complexity control in video coding applications is expected to be further significantly increased as a function of the dramatic increase in the mobile/portable device penetration into the every-day life environment. In this chapter, the authors perform a comprehensive review of the recent advances in computational complexity techniques for video coding applications. This chapter will not only summarize the recent advances in this field, but will also provide explicit directions for the design of the future complexity-aware video coding applications.

Analyzing students' classroom behavior is an important part of evaluating classroom teaching effects in the education field, and the behavior of using mobile phones is an important manifestation of students' learning status. Therefore, the status of using mobile phones in the classroom can reflect the effect of classroom teaching to a certain extent. This article establishes a video-based classroom student behavior data set, and divides the behavior categories in the data into two categories: mobile phone playing and other; analysis of students playing mobile phone and other behaviors reveals that there are subtle movements in student behavior and a certain visual tempo. The resolution is low, and the occlusion is serious. In response to the above problems, this paper proposes a multi-scale mobile phone playing behavior recognition method based on temporal information enhancement and interaction. First, use the motion enhancement module to enhance the motion information between two frames to improve the recognition ability of subtle actions; secondly, add the temporal pyramid to extract the multi-scale features of the action, and then obtain the visual tempo information of the video; finally add the temporal information interaction module to enhance the temporal dimension information interaction , To further model the temporal information. The experimental results on the self-made student action dataset StudentAction show that compared with the existing methods, the algorithm has significantly improved recognition accuracy and better solves the problem of low accuracy in the recognition of subtle actions. Good performance have shown on the public datasets HMDB51 and UCF101, indicating that the method has strong generalization ability and can adapt to the recognition problems of different scene actions.

The recognition of daily actions, such as walking, sitting or standing, in the home is informative for assisted living, smart homes and general health care. A variety of actions in complex scenes can be recognised using visual information. However cameras succumb to privacy concerns. In this paper, we present a home action recognition system using an 8×8 infared sensor array. This low spatial resolution retains user visual privacy, but is still a powerful representation of actions in a scene. Actions are recognised using a 3D convolutional neural network, extracting not only spatial but temporal information from video sequences. Experimental results obtained from a publicly available dataset Infra-ADL2018 demonstrate a better performance of the proposed approach compared to the state-of-the-art. We show that the sensor is considered better at detecting the occurrence of falls and actions of daily living. Our method achieves an overall accuracy of 97.22% across 7 actions with a fall detection sensitivity of 100% and specificity of 99.31%.

This dissertation reports on rate-distortion optimized and computation-constrained video coding algorithms which yield effective and efficient block-based video encoders. We first propose a new cost function for motion estimation which can achieve good tradeoffs among motion vector bit rate, compensation distortion, and number of computations. Based on the new cost function, we propose a predictive motion estimation algorithm which can effectively determine the size of the search region according to the statistics of the motion field. The proposed motion estimation algorithm allows control of the motion vector bit rate and the computational cost, simultaneously. We also develop two efficient rate-distortion optimized coding mode selection algorithms, based on thresholding and finite state machine (FSM) methods, respectively. The thresholding method can eliminate consideration of the computationally expensive modes. The FSM method determines the test orders of the modes and associated parameters by exploiting the local statistics of the input video sequence. The proposed two methods can reduce substantially the computation requirements as compared to the full search mode selection method. Based on the proposed motion estimation and mode selection algorithms, we implement an H.263-based video coder and an MPEG-2, compliant video coder for very low bit rate video applications, and high bit rate interlaced video applications, respectively. The resulting video coders achieve excellent tradeoffs among bit rate, quality, and computational complexity. In fact, experimental results show that our video coders outperform the best known video coders in terms of compression performance and encoding speed.

The purpose of this experiment was to study the effects of limiting visual and/or locomotor access to a part of the environment in the building up of a spatial representation of the whole space. During five sessions, rats were allowed to explore separately and successively the two halves (subspaces) of a circular open field containing four objects. During exploration of each half, continuous or discontinuous locomotor and/or visual access to the other half was provided by using opaque or transparent partitions, with or without doors. Once habituation was complete, the partition was removed for some subjects but remained for others. The locomotor and exploratory reactions to this removal were recorded. Whatever their locomotor experience (continuous or discontinuous), rats that had a discontinuous visual experience between the subspaces displayed a renewal of exploratory activity, whereas the rats that had received a continuous visual experience did not re-explore the objects. This result suggests that continuous visual access to the whole space is necessary for the construction of an overall representation. Furthermore, continuous locomotor activity does not seem to compensate for the discontinuity of visual information.

In this study, we investigated the role of visual and phonological information in lexical access of Chinese characters. Homophonic English words have been the main source of stimuli for word recognition research. However, since these stimuli also often look alike, visual and phonological information may be confounded in reported experiments. In contrast, many homophonic Chinese characters are visually distinct. In addition, visually similar characters often have very different pronunciations. These characteristics allow a more controlled investigation of the roles of visual and phonological information in activation of meaning. In the present study, two types of Chinese characters were used in a semantic categorization paradigm: integrated characters, which contain strokes that are not separable; and compound characters, which contain at least two clearly identifiable components. The results show that the recognition of a Chinese integrated character depends primarily on visual information, whereas the recognition of a Chinese compound character relies on visual, phonological, and semantic information. It is concluded that visual information plays a greater role in Chinese character recognition than has previously been documented.

The wireless environment is built everywhere, especially IEEE 802.11 WLANs. More users use wireless network for searching information in Internets, TV program streaming, and video/voice conference. The packets include not only traditional data but also interactive multimedia data. There is an extension to IEEE 802.11 standards called IEEE 802.11e, improving the QoS capabilities on WLANs. This protocol enhances QoS data as above applications and features. The contribution, Cross-Layer Adaptive Video Prioritization (CAVP), in this thesis paper is divided into two parts. First, it provides MAC-Layer adaptive prioritization (MAP) to prioritize the packet which has the information of video frames at the Medium Access Control (MAC) Layer of IEEE 802.11e. In IEEE 802.11e, the priority of packets is very important, and it affects the order of received packets and the probability of successfully-received packet. The method to prioritize packets is estimating the access time of each access category (AC) for the arrived packet and putting the packet into the AC which has the minimal access time. Second, Video Frame Prioritization (VFP) prioritizes the video packet. Because the importance of video frames is different, we set the initial priority at Application Layer by choosing the maximal PSNR-difference frame as the highest priority frame. The maximal PSNR-difference frame critically influences the overall video sequences. These two parts of our contribution can greatly improve the quality of video streaming. The cross-layer design, CAVP, involves MAC Layer and Application Layer. The information of a packet arrived at MAC Layer is gotten from Application Layer. The benefit of cross-layer design is effectively using the information of different layers and combining their feature. In simulations, we use the popular simulator ns-2 as the experiment platform. We design several scenarios to show that our cross-layer design improves the quality of video streaming. The result shows that our cross-layer design proposal not only greatly increases PSNR of video but also improves the video in vision after the wireless transmission.

Given a video sequence decompressed from a compressed video sequence associated with motion vector information for each image frame, this short note presents an efficient predictive watershed-based video segmentation algorithm using motion vectors. The proposed algorithm has much computation-saving merit since the next frame can inherit the segmented results of the current frame according to the motion vector information of the next frame. Experimental results demonstrate that with little perceptual segmentation degradation, the proposed algorithm has better execution-time performance when compared to the previous Chien et al.'s video segmentation algorithm whose input video sequence is assumed to have no motion vector information. Further, the alleviation of the over-segmentation problem existed in both algorithms is investigated.

Many locomotor tasks involve interactions with moving objects. When observer (i.e., self-)motion is accompanied by object motion, the optic flow field includes a component due to self-motion and a component due to object motion. For moving observers to perceive the movement of other objects relative to the stationary environment, the visual system could recover the object-motion component – that is, it could factor out the influence of self-motion. In principle, this could be achieved using visual self-motion information, non-visual self-motion information, or a combination of both. In this study, we report evidence that visual information about the speed (Experiment 1) and direction (Experiment 2) of self-motion plays a role in recovering the object-motion component even when non-visual self-motion information is also available. However, the magnitude of the effect was less than one would expect if subjects relied entirely on visual self-motion information. Taken together with previous studies, we conclude that when self-motion is real and actively generated, both visual and non-visual self-motion information contribute to the perception of object motion. We also consider the possible role of this process in visually guided interception and avoidance of moving objects.

Abstract: In team sports, attention must be divided to multiple players in order to monitor their movements and to initiate the correct motor response at the right time. To solve such tasks, experienced athletes seem to make use of their peripheral vision by using a task-dependent gaze behavior (Vater, Williams, & Hossner, 2019). In a series of three experimental studies, this monitoring functionality has been investigated to propose ways for sport-specific investigations on peripheral vision. For the experimental studies, we chose the Multiple-Object Tracking (MOT) task, a task with high experimental control in which we could expect that peripheral vision would naturally be used for target monitoring (i.e., without the instruction to locate gaze on a fixation cross). The task requires one to monitor 4 initially highlighted targets moving together with 6 distractors (10 square objects in total). In this task, gaze seems predominantly not be located on individual targets, but rather on the virtual center of mass (“centroid” of the targets). Thus, it can be expected that peripheral vision is used for target monitoring. In Study 1, the validity of the MOT paradigm was tested to make sure that peripheral vision was indeed used for target monitoring as well as to examine the motion sensitivity of peripheral vision. In Study 2, we tested if this motion sensitivity holds true in dual task situations with a concurrent monitoring and detection tasks. In Study 3, we systematically manipulated visual and attentional demands to analyze the influence of these factors on gaze behavior during monitoring and change-detection performance. Considering the results of all three studies, the functionality of peripheral vision allows for the monitoring of multiple moving objects, the (simultaneous) detection of motion changes and the avoidance of saccade-related costs (because saccades would interrupt visual information processing). It could further be shown that capabilities of the visual and the attentional systems affect the gaze anchoring location and peripheral monitoring and detection performance. In a theoretical paper, we link our fundamental results to sports situations, for example, the monitoring of multiple players in soccer and the peripheral detection of a crucial player’s movement, and discuss how a sport-specific functionality could be tested in future experiments. In field- and laboratory experiments, we already tested this proposed functionality and found, for example in martial arts, that national athletes anchor their gaze on the opponent’s chest to detect attacks from arms and legs. In a beach-volleyball defense situations, experts seem to anchor their gaze between the ball and the attacking player (i.e., in free space and not on an actual visual cue). This suggests that peripheral vision is used to monitor both movements. In current studies, we make use of virtual reality applications to examine the functionality of peripheral vision with a higher degree of experimental control but at the same time a high ecological validity. Our final aim is to train expert’s peripheral vision in sport specific settings to improve visual information pick-up outside the fovea.