Spatiotemporal Learning for Motion Estimation and Visual Recognition

We propose MoBox, a low-cost solution for semi-supervised video object segmentation that requires only bounding boxes as manual annotations for training. Built upon a mature semi-supervised video object segmentation network, we redesign the training losses and employ a more stringent training strategy. Specifically, we introduce a well-designed constraint term that enhances traditional spatial projection by simultaneously leveraging the projections of both the ground-truth box and the predicted mask across two axes, rather than evaluating discrepancies along the x-axis and y-axis independently. To harness the intrinsic properties of videos, considering the underlying correspondence between motion represented by optical flow and the original image, we incorporate motion coherence information into the color consistency loss as supplementary information and propose a motion discrepancy loss to obtain accurate boundaries. Additionally, to mitigate the ambiguity of weak supervision, we further introduce the pseudo strict constraint during training, which significantly improves model performance. Our approach yields competitive scores on popular benchmarks, achieving a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {J}\&amp; \mathcal {F}$ </tex-math></inline-formula> score of 78.6 on the DAVIS 2017 validation set and an Overall score of 78.0 on the YouTube-VOS 2018 validation set. These results highlight the efficacy of MoBox, demonstrating that the semi-supervised video object segmentation model can be effectively trained using only motion-augmented box supervision and intrinsic information of videos.

Segmentation, recognition and 3D reconstruction of objects have been cutting-edge research topics, which have many applications ranging from environmental and medical to geographical applications as well as intelligent transportation. In this dissertation, I focus on the study of segmentation, recognition and 3D reconstruction of objects using LiDAR data/MRI. Three main works are that (I). Feature extraction algorithm based on sparse LiDAR data. A novel method has been proposed for feature extraction from sparse LiDAR data. The algorithm and the related principles have been described. Also, I have tested and discussed the choices and roles of parameters. By using correlation of neighboring points directly, statistic distribution of normal vectors at each point has been effectively used to determine the category of the selected point. (II). Segmentation and 3D reconstruction of objects based on LiDAR/MRI. The proposed method includes that the 3D LiDAR data are layered, that different categories are segmented, and that 3D canopy surfaces of individual tree crowns and clusters of trees are reconstructed from LiDAR point data based on a region active contour model. The proposed method allows for delineations of 3D forest canopy naturally from the contours of raw LiDAR point clouds. The proposed model is suitable not only for a series of ideal cone shapes, but also for other kinds of 3D shapes as well as other kinds dataset such as MRI. (III). Novel algorithms for recognition of objects based on LiDAR/MRI. Aimed to the sparse LiDAR data, the feature extraction algorithm has been proposed and applied to classify the building and trees. More importantly, the novel algorithms based on level set methods have been provided and employed to recognize not only the buildings and trees, the different trees (e.g. Oak trees and Douglas firs), but also the subthalamus nuclei (STNs). By using the novel algorithms based on level set method, a 3D model of the subthalamus nuclei (STNs) in the brain has been successfully reconstructed based on the statistical data of previous investigations of an anatomy atlas as reference. The 3D rendering of the subthalamic nuclei and the skull directly from MR imaging is also utilized to determine the 3D coordinates of the STNs in the brain. In summary, the novel methods and algorithms of segmentation, recognition and 3D reconstruction of objects have been proposed. The related experiments have been done to test and confirm the validation of the proposed methods. The experimental results also demonstrate the accuracy, efficiency and effectiveness of the proposed methods. A framework for segmentation, recognition and 3D reconstruction of objects has been established, which has been applied to many research areas.

To address the issues of low discrimination accuracy of similar objects and error accumulation caused by segmentation errors in video object segmentation, we propose a novel semi-supervised video object segmentation method with global feature enhancement and mask correction. Firstly, we employ a global context-aware module to enhance features by modeling the global dependencies of features via two global memory units, capturing global contextual information within and between video frames and thus improving the models’ ability to distinguish between similar distractors. Secondly, we propose a detail-aware decoding which fuses encoded features through skip connections in the early decoding stage to learn detail-enhanced decoding features. Finally, we design a mask correction module at the later decoding stage to estimate uncertain areas in coarse segmentation masks and correct blurred object boundaries and other segmentation errors, producing accurate video object segmentation results. Extensive validation experiments on challenging DAVIS and YouTube-VOS benchmark datasets demonstrate that our method is significantly superior to the comparative methods presented in this paper. Specifically, on the YouTube-VOS 2019 validation set, its performance score <italic>G</italic> outperforms the STCN and GSFM methods by 1.6 and 0.3, respectively.

Due to the variety of video types and different quality on the Internet, it brings more challenges to video processing algorithms such as video object segmentation. Most existing video object segmentation methods rely on modules in other fields as an additional structure of the segmentation model. The combination of modules can improve the accuracy of the model, but it will also reduce the algorithm speed. This paper proposes a semi-supervised video object segmentation method based on local and global consistency learning, which does not rely on additional structures to achieve fast segmentation. First, we extract the embedding features of the image based on GhostNet which is the lightweight network. By using the embedded features of pixels, the graph model is established based on the similarity between pixels. Second, we adopt the local-global consistency learning framework to construct the label conduction model. Third, to optimize the memory occupation and inference speed of the model, we propose a sampling strategy for reference frames by considering local and global information. Finally, we establish a high-speed monitoring video dataset to verify the practical application effect of the method. Our method achieves a result of 69.5% $J\& F$ mean with 46 FPS on DAVIS 2017 dataset. At the same time, this paper constructed a high-speed monitoring video dataset. The algorithm obtained 68.2% $J\& F$ on this dataset, indicating that the method has good generalization and robust performance in practical applications.

Semi-supervised video object segmentation (VOS) aims to track the designated objects present in the initial frame of a video at the pixel level. To fully exploit the appearance information of an object, pixel-level feature matching is widely used in VOS. Conventional feature matching runs in a surjective manner, i.e., only the best matches from the query frame to the reference frame are considered. Each location in the query frame refers to the optimal location in the reference frame regardless of how often each reference frame location is referenced. This works well in most cases and is robust against rapid appearance variations, but may cause critical errors when the query frame contains background distractors that look similar to the target object. To mitigate this concern, we introduce a bijective matching mechanism to find the best matches from the query frame to the reference frame and vice versa. Before finding the best matches for the query frame pixels, the optimal matches for the reference frame pixels are first considered to prevent each reference frame pixel from being overly referenced. As this mechanism operates in a strict manner, i.e., pixels are connected if and only if they are the sure matches for each other, it can effectively eliminate background distractors. In addition, we propose a mask embedding module to improve the existing mask propagation method. By embedding multiple historic masks with coordinate information, it can effectively capture the position information of a target object.

In the booming video era, video segmentation attracts increasing research attention in the multimedia community. Semi-supervised video object segmentation (VOS) aims at segmenting objects in all target frames of a video, given annotated object masks of reference frames. Most existing methods build pixel-wise reference-target correlations and then perform pixel-wise tracking to obtain target masks. Due to neglecting object-level cues, pixel-level approaches make the tracking vulnerable to perturbations, and even indiscriminate among similar objects. Towards robust VOS, the key insight is to calibrate the representation and mask of each specific object to be expressive and discriminative. Accordingly, we propose a new deep network, which can adaptively construct object representations and calibrate object masks to achieve stronger robustness. First, we construct the object representations by applying an adaptive object proxy (AOP) aggregation method, where the proxies represent arbitrary-shaped segments at multi-levels for reference. Then, prototype masks are initially generated from the reference-target correlations based on AOP. Afterwards, such proto-masks are further calibrated through network modulation, conditioning on the object proxy representations. We consolidate this conditional mask calibration process in a progressive manner, where the object representations and proto-masks evolve to be discriminative iteratively. Extensive experiments are conducted on the standard VOS benchmarks, YouTube-VOS-18/19 and DAVIS-17. Our model achieves the state-of-the-art performance among existing published works, and also exhibits superior robustness against perturbations.

Object recognition and segmentation in videos by connecting heterogeneous visual features

Semi-supervised video object segmentation is to segment the target objects given the ground truth annotation of the first frame. Previous successful methods mostly rely on online learning or static image pre-train to improve accuracy. However, online learning methods require huge time costs at inference time, thus restrict their practical use. Methods with static image pre-train require heavy data augmentation that is complicated and time-consuming. This paper presents a fast Hierarchical Embedding Guided Network (HEGNet) which is only trained on Video Object Segmentation (VOS) datasets and does not utilize online learning. Our HEGNet integrates propagation-based and matching-based methods. It propagates the predicted mask of the previous frame as a soft cue and extracts hierarchical embedding at both deep and shallow layers to do feature matching. The produced label map of the deep layer is also used to guide the matching of the shallow layer. We evaluated our method on the DAVIS-2016 and DAVIS-2017 validation sets and achieved overall scores of 84.9% and 71.9% respectively. Our method surpasses the methods without online learning and static image pre-train and runs at 0.08 seconds per frame.

The constrained two-bit transform (C2BT) is recently proposed for low complexity motion estimation (ME) to reduce the high computational complexity of the typical two-bit transform (2BT) while maintaining the ME accuracy. And the bit-inverted Gray-coded bit-plane matching (BGCBPM) is also proposed to enhance the ME accuracy. In this paper, we propose a fast ME algorithm exploiting these two bit-plane matching (BPM) criteria. Using the low complexity image transformations of typical images into bit-planes and their corresponding matching criteria and another hybrid matching criterion, we can efficiently determine multiple candidate motion vectors and increase the ME accuracy substantially. Experimental results show that the peak-to-peak signal-to-noise ratio (PSNR) difference between the proposed algorithm and the typical sum of absolute differences (SAD) based full search algorithm is only 0.03 dB on average with negligible computational complexity increase.

A number of behavioural phenomena distinguish the recognition of faces and objects, even when members of a set of objects are highly similar. Because faces have the same parts in approximately the same relations, individuation of faces typically requires specification of the metric variation in a holistic and integral representation of the facial surface. The direct mapping of a hypercolumn-like pattern of activation onto a representation layer that preserves relative spatial filter values in a two-dimensional (2D) coordinate space, as proposed by C. von der Malsburg and his associates, may account for many of the phenomena associated with face recognition. An additional refinement, in which each column of filters (termed a 'jet') is centred on a particular facial feature (or fiducial point), allows selectivity of the input into the holistic representation to avoid incorporation of occluding or nearby surfaces. The initial hypercolumn representation also characterizes the first stage of object perception, but the image variation for objects at a given location in a 2D coordinate space may be too great to yield sufficient predictability directly from the output of spatial kernels. Consequently, objects can be represented by a structural description specifying qualitative (typically, non-accidental) characterizations of an object's parts, the attributes of the parts, and the relations among the parts, largely based on orientation and depth discontinuities (as shown by Hummel & Biederman). A series of experiments on the name priming or physical matching of complementary images (in the Fourier domain) of objects and faces documents that whereas face recognition is strongly dependent on the original spatial filter values, evidence from object recognition indicates strong invariance to these values, even when distinguishing among objects that are as similar as faces.

Conventional motion estimation algorithms in video coding consider only translations, which can be readily characterized by displacement motion vectors. We might be able to achieve more accurate motion estimation by enlarging the search windows, at the expense of drastically increased computational complexity. Alternatively, we could also increase the accuracy of motion estimation by taking into account potential non-translation motions of objects in a video scene, without increasing the search window size. In this paper, we conduct a comparative study on the computational complexity of these two methods. We demonstrate that our motion estimation method based on the Lie operators can significantly increase the accuracy of motion estimation, while requiring much lower complexity than the method of enlarged search windows.

Motion estimation is a vital function of video coders that use motion compensated prediction to exploit the temporal redundancy of video signals. The reference images used in the motion estimation process can be either original images (open‐loop architecture) or reconstructed images (closed‐loop architecture). While the closed‐loop architecture is intuitively superior to the open‐loop architecture, closed‐loop motion estimation is more difficult to implement in a real‐time video encoder. A compromise is to perform open‐loop integer‐pel accurate motion estimation followed by closed‐loop half‐pel accurate estimation. In this article we present a comparative study of the performance of half‐pel closed‐loop versus open‐loop motion estimation for coding of high definition television with the MPEG algorithm. Simulation results indicate that on average, closed‐loop half‐pel accurate motion estimation provides slightly better performance than open‐loop half‐pel accurate estimation. This improvement depends on the bit rate of the coded video biststream. For relatively high bit rates, the closed‐loop implementation often provides no improvement, and sometimes an actual degradation of performance. An analysis of why a closed‐loop implementation does not guarantee a better performance is presented.©1994 John Wiley &amp; Sons Inc

Object recognition is a technology based on computer vision techniques to identify the object in images or video. A previous study revealed that object recognition is useful to improve the learning outcome in augmented reality applications. However, there is yet to be a comprehensive augmented reality (AR) framework on object recognition in the learning application domain setting. This study aims to review the existing AR frameworks in object recognition and highlight the important components needed in AR framework. This Systematic Literature Review (SLR) was developed based on the PRISMA method. A preliminary search resulted in a total of 70 articles. After the removal of duplicates and a more rigorous screening based on the titles and abstracts, the number of articles was reduced to 26 articles. Then, for final review, only 9 articles were chosen. Based on the SLR, the components needed in the AR framework for object recognition are tracking, image classification, object localization, object detection, object segmentation, interface and interaction. Some of the limitations identified from the existing AR framework for object recognition include the lower-resolution camera that uses compressed images and due to the low detection accuracy and the limited of the tools. The proposed framework suggests the initial for AR framework object recognition.

The problem of searching for a model-based scene interpretation is analyzed within a probabilistic framework. Object models are formulated as generative models for range data of the scene. A new statistical criterion, the truncated object probability, is introduced to infer an optimal sequence of object hypotheses to be evaluated for their match to the data. The truncated probability is partly determined by prior knowledge of the objects and partly learned from data. Some experiments on sequence quality and object segmentation and recognition from stereo data are presented. The article recovers classic concepts from object recognition (grouping, geometric hashing, alignment) from the probabilistic perspective and adds insight into the optimal ordering of object hypotheses for evaluation. Moreover, it introduces point-relation densities, a key component of the truncated probability, as statistical models of local surface shape.

PurposeArtificial intelligence-enabled techniques can process large amounts of surgical data and may be utilized for clinical decision support to recognize or forecast adverse events in an actual intraoperative scenario. To develop an image-guided navigation technology that will help in surgical education, we explored the performance of a convolutional neural network (CNN)-based computer vision system in detecting intraoperative objects.MethodsThe surgical videos used for annotation were recorded during surgeries conducted in the Department of Surgery of Tokyo Women’s Medical University from 2019 to 2020. Abdominal endoscopic images were cut out from manually captured surgical videos. An open-source programming framework for CNN was used to design a model that could recognize and segment objects in real time through IBM Visual Insights. The model was used to detect the GI tract, blood, vessels, uterus, forceps, ports, gauze and clips in the surgical images.ResultsThe accuracy, precision and recall of the model were 83%, 80% and 92%, respectively. The mean average precision (mAP), the calculated mean of the precision for each object, was 91%. Among surgical tools, the highest recall and precision of 96.3% and 97.9%, respectively, were achieved for forceps. Among the anatomical structures, the highest recall and precision of 92.9% and 91.3%, respectively, were achieved for the GI tract.ConclusionThe proposed model could detect objects in operative images with high accuracy, highlighting the possibility of using AI-based object recognition techniques for intraoperative navigation. Real-time object recognition will play a major role in navigation surgery and surgical education.Supplementary InformationThe online version contains supplementary material available at 10.1007/s11548-021-02434-w.