Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion

This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movements and object motions in a scene. However, enabling intuitive shot design in modern image-to-video generation systems presents two main challenges: first, effectively capturing user intentions on the motion design, where both camera movements and scene-space object motions must be specified jointly; and second, representing motion information that can be effectively utilized by a video diffusion model to synthesize the image animations. To address these challenges, we introduce MotionCanvas, a method that integrates user-driven controls into image-to-video (I2V) generation models, allowing users to control both object and camera motions in a scene-aware manner. By connecting insights from classical computer graphics and contemporary video generation techniques, we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis without requiring costly 3D-related training data. MotionCanvas enables users to intuitively depict scene-space motion intentions, and translates them into spatiotemporal motion-conditioning signals for video diffusion models. We demonstrate the effectiveness of our method on a wide range of real-world image content and shot-design scenarios, highlighting its potential to enhance the creative workflows in digital content creation and adapt to various image and video editing applications.

Motion analysis plays a key role in the understanding of video imagery. The goal of our research is simultaneous camera motion modeling and object motion tracking in video sequences captured with public access pan-tilt-zoom web cameras. As a first step, we have developed and evaluated two techniques for camera motion modeling. Our first method uses SURF feature detection to detect camera and object motion. Our second method uses watershed region segmentation on sequential video frames, followed by template matching to detect the motion of watershed region over time and quantify camera motion.

A large number of the recently proposed motion segmentation algorithms model the problem as one of clustering the trajectory data to its corresponding affine subspace. While these algorithms achieve near perfect results on benchmark datasets, they fail to address certain very key real-world challenges, including perspective effects and motion degeneracies. In this paper we present a motion segmentation algorithm capable of dealing with the degenerate cases, where camera motion follows that of the moving object. We first generate a set of prospective motion models for the various moving and stationary objects in the video sequence by a RANSAC-like procedure. Then, we incorporate affine and gestalt-inspired motion similarity constraints, into a multi-label Markov Random Field (MRF). It's inference leads to an over-segmentation, where each label belongs to a particular moving object or the background. This is followed by a model selection step where we merge clusters based on a novel motion coherence constraint, we call in-frame shear, that tracks the in-frame change in orientation and distance between the clusters, leading to the final segmentation.We show results on the Hopkins-155 benchmark motion segmentation dataset [16]. In addition we also show results for several on-road scenes where camera and object motion are near identical. Our algorithm is competitive with the state-of-the-art algorithms on [16] and exceeds them on the on-road sequences.

In this paper we propose a system for the analysis of user generated video (UGV). UGV often has a rich camera motion structure that is generated at the time the video is recorded by the person taking the video, i.e., the ?camera person.? We exploit this structure by defining a new concept known as camera view for temporal segmentation of UGV. The segmentation provides a video summary with unique properties that is useful in applications such as video annotation. Camera motion is also a powerful feature for identification of keyframes and regions of interest (ROIs) since it is an indicator of the camera person's interests in the scene and can also attract the viewers' attention. We propose a new location-based saliency map which is generated based on camera motion parameters. This map is combined with other saliency maps generated using features such as color contrast, object motion and face detection to determine the ROIs. In order to evaluate our methods we conducted several user studies. A subjective evaluation indicated that our system produces results that is consistent with viewers' preferences. We also examined the effect of camera motion on human visual attention through an eye tracking experiment. The results showed a high dependency between the distribution of fixation points of the viewers and the direction of camera movement which is consistent with our location-based saliency map.

This paper describes an efficient approach to scene-break detection, which can detect cuts, dissolves, and wipes reliably and effectively by means of temporally linear prediction models. In our algorithm, two linear prediction models are adopted to predict a current frame: one for dissolves, and the other for stationary scenes. The predicted frames, derived based on the two models, are compared with the original frames, and cuts and dissolves are then determined based on Bayesian cost functions. For the detection, our algorithm requires the setting of a single threshold only. In wipe detection, our linear prediction models are employed to detect areas of change between two successive frames. By accumulating the changed areas and the overlap of the changed areas over the successive frames, wipes of an arbitrary shape and direction are detected. Experimental results show that our algorithm can achieve a high level of precision even if a video contains object motion and camera motion. The detection time required to analyze a 38-min video is no more than several seconds.

In recent scenario, video surveillance systems are important to monitor security sensitive areas such as banks, crowded public places and borders. Hence, Object detection in such video surveillance is an important task. Research work to detect an object with steady background is carried out and upto some extent it is easy but with moving background it is still challenging. Object detection is complicated as the camera motion and object motion are mixed. Motion vectors, represents flow of a moving object, are obtained using Lucas - kanade optical flow algorithm for moving object detection with complex background. These flow vectors are quantized using a predefined threshold to decide whether a pixel is a part of an object or a background. We have used bilateral filter as a pre-processing step due to which accurate and fast detection of the object is achieved.

Moving object detection and tracking from video captured by moving camera

Attention-from-motion: A factorization approach for detecting attention objects in motion

Novel view synthesis (NVS) and video prediction (VP) are typically considered disjoint tasks in computer vision. However, they can both be seen as ways to observe the spatial-temporal world: NVS aims to synthesize a scene from a new point of view, while VP aims to see a scene from a new point of time. These two tasks provide complementary signals to obtain a scene representation, as viewpoint changes from spatial observations inform depth, and temporal observations inform the motion of cameras and individual objects. Inspired by these observations, we propose to study the problem of Video Extrapolation in Space and Time (VEST). We propose a model that leverages the self-supervision and the complementary cues from both tasks, while existing methods can only solve one of them. Experiments show that our method achieves performance better than or comparable to several state-of-the-art NVS and VP methods on indoor and outdoor real-world datasets. (Project page: https://cs.stanford.edu/~yzzhang/projects/vest/.)

With the increasing popularity of digital camera, organizing and managing the large collection of digital photos effectively become popular. Most researchers focus on clustering and image retrieval, but never on photo album compression. Some set redundancy methods for compressing similar image sets cannot handle photo album compression well because they don't account for any camera and object motion information between images. In this paper, we analyzed the characteristics of camera and object motion between similar photos and designed a novel lossy coding scheme for similar image sets. In our proposed coding scheme, we applied Global Motion Compensation (GMC), Local Motion Compensation (LMC) and Intra prediction inspired from video coding technology. Our method provides a compact and reasonable format for people to store and transmit their large number of digital photos. Experiments prove that our algorithm is efficient and effective for digital photo processing.

Monocular spherical depth estimation with explicitly connected weak layout cues

Visual change detection, aiming at segmentation of video frames into foreground and background regions, is one of the elementary tasks in computer vision and video analytics. The applications of change detection include anomaly detection, object tracking, traffic monitoring, human machine interaction, behavior analysis, action recognition, and visual surveillance. Some of the challenges in change detection include background fluctuations, illumination variation, weather changes, intermittent object motion, shadow, fast/slow object motion, camera motion, heterogeneous object shapes and real-time processing. Traditionally, this problem has been solved using hand-crafted features and background modelling techniques. In recent years, deep learning frameworks have been successfully adopted for robust change detection. This article aims to provide an empirical review of the state-of-the-art deep learning methods for change detection. More specifically, we present a detailed analysis of the technical characteristics of different model designs and experimental frameworks. We provide model design based categorization of the existing approaches, including the 2D-CNN, 3D-CNN, ConvLSTM, multi-scale features, residual connections, autoencoders and GAN based methods. Moreover, an empirical analysis of the evaluation settings adopted by the existing deep learning methods is presented. To the best of our knowledge, this is a first attempt to comparatively analyze the different evaluation frameworks used in the existing deep change detection methods. Finally, we point out the research needs, future directions and draw our own conclusions.

In the modes of both object motion and camera motion, an enhanced Camshift algorithm, which is based on suppressing similar color features of background and on joint color probability density distribution image, is proposed to real-time track head in dynamic complex environment. The system consists of face detection module, head tracking module and camera control module. When tracking fails, a self-recovery mechanism is introduced. At first the Adaboost face detector based on Haar-like features is implemented to find frontal faces, the false positive is filtered according to the skin color criterion, and the true face is used to initialize the tracking module. In hue saturation value (HSV) colorspace, the hue-saturation (H-S) histogram of face skin and the saturation-value (S-V) histogram of hair are built to produce the joint color probability density distribution image, and this is intended to realize the head tracking with arbitrary pose. During tracking, region of interest (ROI) is introduced, and the color probability density distribution of a specified background area outside the ROI is learned, similar color features in the head are suppressed according to the learning result. The background suppression step is intended to resolve the problem that the tracker maybe fails when the head is distracted by backgrounds having similar colors with the head. A closed loop control model based on speed regulation is applied to drive an active camera to center the head. Once tracking drift or failure is detected, the system stops tracking and returns to the face detection module. Our experimental results show that the presented system is well suitable for tracking head with arbitrary pose in dynamic complex environments, also the active camera can track moving head smoothly and stably. The system is computationally efficient and can run in real-time completely.

Typical result of an automatic shot change detection algorithm expectedly includes a certain number of undetected shots as well as falsely detected shots. Even though automatic shot change detection algorithms are continuing to improve, the ultimate goal of automatically detecting all shot changes without false alarms may never be achieved. Thus, allowing a human operator to intervene—to review and verify the result of a shot change detection algorithm, to delete falsely detected shots as well as to find undetected shots—may be the most viable approach currently available for increasing the accuracy of the overall shot detection process. For this exact purpose, we propose a shot verifier based on the visual rhythm. The visual rhythm, an abstraction of the video, is a single image, a sub-sampled version of a full video in which the sampling is performed in a pre-determined and in a systematic fashion. It is a representation of the video, which includes the overall content of the video. But most importantly, the visual rhythm contains patterns or visual features that allow the viewer/operator to distinguish and classify many different types of video effects (edits and otherwise): cuts, wipes, dissolves, fades, camera motions, object motions, flashlights, zooms, etc. The different video effects manifest themselves as different patterns on the visual rhythm. Using the visual rhythm, it becomes possible, without sequentially playing the entire video, to find false positive shots as well as undetected shots. Thus, inclusion of the visual rhythm in the shot verification process will aid the operator to verify detected shots as well as to find undetected shots fast and efficiently. Our newly developed shot verifier based on the visual rhythm has been designed for operator efficiency. The design of our shot verifier presented and the usefulness of the visual rhythm during the shot verification process will be demonstrated.

Multi-Object Tracking (MOT) in Unmanned Aerial Vehicle (UAV) scenarios is characterized by frequent and abrupt camera motion, which presents two unique challenges: nonlinear motion and appearance degradation. Traditional motion models, designed for smooth and consistent motion, struggle to capture the complex background motion patterns caused by UAV movement; while appearance-based methods are easily disrupted by occlusion and blur, leading to unreliable associations. Even though dense optical flow is widely utilized to model complex motion patterns, the entanglement of background and object motion often introduces interference, limiting its effectiveness. To this end, we propose MoDe-Track, a unified framework that explicitly decouples scene motion into background and object components, and serves as an elegant integration of three robust components. Specifically, the Scene Motion Decomposition (SMD) module decouples the motion into the background and object components based on robust principal component analysis, serving as the foundation for motion compensation and feature propagation. Afterwards, the Background Motion Compensation (BMC) uses the decomposed background flow to estimate and compensate for camera motion, mitigating the effects of nonlinear motion. Finally, the Foreground-guided Feature Propagation (FFP) module uses the decoupled object flow to guide feature propagation across frames, achieving temporal consistency and enhancing robustness against occlusion and motion blur. Extensive experimental results on two benchmarks, VisDrone2019 and UAVDT, demonstrate that MoDe-Track consistently outperforms current multi-object tracking methods. We achieve 56.0% MOTA on VisDrone2019 and 56.2% MOTA on UAVDT, reaching the state-of-the-art among existing methods.