Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation

Video object extraction is an important issue in video content analysis. In this paper, a novel approach to motion trajectory extraction is proposed, which may effectively facilitate video object extraction. A motion trajectory in this approach is defined as a trail of image regions with coherent velocities in an image sequence. The regions with color and texture coherence are extracted first. Then, the region velocity is estimated through the relationships between itself and its adjacent regions in the consecutive frames. This process is called velocity voting. After that, the genetic algorithm is applied to establish the 2D+t motion trajectories based on the region velocities. Finally, some short motion trajectories are concatenated along the time axis. In the spatial plane, the regions are merged by grouping the similar motion trajectories. Experimental results show that the proposed approach effectively suppresses some challenging issues in video object extraction, such as over-segmentation and losing track of objects.

A new semantic steganographic algorithm for hiding data in the motion trajectories of Video Objects (VOs) is presented. First, a set of primitives (objects bounding-box coordinates and its centroid) are used to dene a VO. The centroid of VO is tracked in each frame and its coordinate is stored in a trajectory vector. The proposed algorithm embeds data in the VO object trajectory by presenting a relationship between the hidden data and the trajectory data in the form of a motion drift or perturbation. The perturbation introduced by the embedding algorithm is implemented by motion compensating the pixels in the VO bounding box. Assessment of the steganographic algorithm is done by both subjective judgments based on the perceived quality of the perturbed video sequence and by some commonly used statistical steganalysis metrics. Both subjective assessments and statistical measures indicate visual and statistical invisibility.

Generating temporally smooth and high-resolution videos is a crucial objective in talking face generation tasks. Diffusion-based generative models have emerged as a prime choice for these tasks due to their ability to produce high-quality outputs. To mitigate the impact of stochasticity in the diffusion process, recent research has predominantly utilized self-attention layers to extract temporal features, ensuring temporal consistency in the generated videos. However, self-attention mechanisms have computational complexity that scales quadratically with video length, leading to high computational costs. This limitation poses significant challenges when attempting to generate longer video sequences using diffusion models. To address this challenge, we propose Talking-DiSSM, an end-to-end method for generating audio-driven talking face videos using State-Space Models (SSMs). This novel framework for conditional video diffusion modeling integrates Bidirectional State-Space Models (Bi-SSM) as temporal modeling modules with linear complexity, effectively capturing complex sequential temporal information and intra-batch sequential interdependencies in videos. Additionally, we employ a simple yet effective batch-overlapped sampling strategy to process input video clips, constructing inter-batch correlations while incorporating reference face clips and landmarks as conditions to ensure stability in the generation process. Extensive experiments demonstrate that Talking-DiSSM generates temporally consistent, high-quality, and identity-preserving talking face videos synchronized with the driving audio, achieving state-of-the-art results compared to existing models.

In this paper, we exploit a new type of data redundancy in the multisource surveillance video to reduce the huge gap between the growth rate of the data and the video compression rate. Global redundancy caused by correlated appearances of moving objects in multiple videos consists of model similarity, spatial correlation and temporal consistency. Therefore, we propose a global coding scheme of moving objects to eliminate the global redundancy: a model based object reconstruction is initially employed to reconstruct the objects in the video, then a pose-based residual error prediction is developed to compensate the difference between the real video appearance and the initial reconstruction from model. The experiment with two simulated surveillance videos has proved that the proposed coding scheme can achieve better coding performance than the main profile of HEVC and surveillance profile of IEEE 1857-2013.

We present an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method allows generation of single transparent images or of multiple transparent layers. The method learns a "latent transparency" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset with minimal changes to the original latent distribution of the pretrained model. In this way, any latent diffusion model can be converted into a transparent image generator by finetuning it with the adjusted latent space. We train the model with 1M transparent image layer pairs collected using a human-in-the-loop collection scheme. We show that latent transparency can be applied to different open source image generators, or be adapted to various conditional control systems to achieve applications like foreground/background-conditioned layer generation, joint layer generation, structural control of layer contents, etc. A user study finds that in most cases (97%) users prefer our natively generated transparent content over previous ad-hoc solutions such as generating and then matting. Users also report the quality of our generated transparent images is comparable to real commercial transparent assets like Adobe Stock.

Providing high accuracy of the coordinates and trajectories of objects by measurements conducted in navigation systems and complexes is an urgent task, which improves safety and efficiency of different modes of transport. However difficult environmental conditions, where vehicles are commonly used, stipulate influence of different factors on performance of onboard satellite navigation receivers, which are used as basic navigation devices for ground vehicle nowadays. Setting on cars used for common purposes additional navigation devices, which provide better performance, in most cases is economically unreasonable. Economically reasonable ways to improve onboard navigation complexes of vehicles, which are used for common purposes, are examined in this article. Functional diagram and principles of work of navigational complex, which uses the satellite navigation receiver and simplified variant of inertial navigation system is pointed as well. Also, the justification of methods for minimizing the error formats of coordinates and trajectories of moving objects based on information processing in multipositional, in particular satellite-inertial navigation systems and complexes, is presented. The obtained research results give an opportunity to develop an algorithm for coordinate refinement, which can be implemented in the improved on-board navigational complex of vehicle. KEY WORDS: NAVIGATION SYSTEMS AND COMPLEXES, INERTIAL SENSORS, NAVIGATION DEFINITIONS, ACCURACY AND RELIABILITY OF COORDINATES AND TRAJECTORIES OF MOVING OBJECTS, ELLIPS OF ERRORS, PROBABILISTIC-GEOMETRIC METHODS.

Existing methods for video object segmentation (VOS) have achieved significant success by performing semantic guidance, spatial constraint, or temporal consistency. However, VOS still remains highly challenging because it is difficult to collaboratively leverage spatial constraint, temporal consistency, and semantic guidance while reducing redundant information. In this paper, we propose an efficient unified spatio-temporal dynamic routing (STDR) framework to address VOS by achieving a better spatio-temporal balance while avoiding redundancy. Specifically, our unified spatio-temporal modeling contains three paths: 1) short-term spatial path is employed to mine the spatial constraints from the previous frame; 2) long-term semantic path is used to capture semantic cues from the first reference frame with ground-truth labels; 3) memory queue path is designed to efficiently exploit the temporal consistency of middle frames with a compact memory bank of constant size. To enhance the input of each path, we introduce a progressive contextual memory enhancement module to exploit the contextualized memory with growing receptive fields by progressively aggregating spatial contextual information from adjacent frames for each memory frame. Furthermore, we design a dynamic memory-routed module to globally refine the outputs of our three paths for unified modeling. Enhanced by the proposed modules, our STDR achieves state-of-the-art performance with fast speed on the DAVIS 2016, DAVIS 2017 Val/Test, YouTube-VOS 2018/2019, and real-world long-video benchmarks.

A feature point based method is proposed for tracking multiple fish in 3D space. First, a simplified representation of the object is realized through construction of two feature point models based on its appearance characteristics. After feature points are classified into occluded and non-occluded types, matching and association are performed, respectively. Finally, the object's motion trajectory in 3D space is obtained through integrating multi-view tracking results. Experimental results show that the proposed method can simultaneously track 3D motion trajectories for up to 10 fish accurately and robustly.

In recent years, diffusion models have dominated the field of image generation with their outstanding generation quality. However, pre-trained large-scale diffusion models are generally trained using fixed-size images, and fail to maintain their performance at different aspect ratios. Existing methods for generating arbitrary-size images based on diffusion models face several issues, including the requirement for extensive finetuning or training, sluggish sampling speed, and noticeable edge artifacts. This paper presents the InstantAS method for arbitrary-size image generation. This method performs non-overlapping minimum coverage segmentation on the target image, minimizing the generation of redundant information and significantly improving sampling speed. To maintain the consistency of the generated image, we also proposed the Inter-Domain Distribution Bridging method to integrate the distribution of the entire image and suppress the separation of diffusion paths in different regions of the image. Furthermore, we propose the dynamic semantic guided cross-attention method, allowing for the control of different regions using different semantics. Experimental results show that InstantAS has better fusion capabilities compared to previous arbitrary-size image generation methods and is far ahead in sampling speed compared to them.

The data-intensive nature of Diffusion models amplifies the risks of privacy infringements and copyright disputes, particularly when training on extensive unauthorized data scraped from the Internet. Membership Inference Attacks (MIA) aim to determine whether a data sample has been utilized by the target model during training, thereby serving as a pivotal tool for privacy preservation. Current MIA employs the prediction loss to distinguish between training member samples and non-members. These methods assume that, compared to non-members, members, having been encountered by the model during training result in a smaller prediction loss. However, this assumption proves ineffective in diffusion models due to the random noise sampled during the training process. Rather than estimating the loss, our approach examines this random noise and reformulate the MIA as a noise search problem, assuming that members are more feasible to find the noise used in the training process. We formulate this noise search process as an optimization problem and employ the fixed-point iteration to solve it. We analyze current MIA methods through the lens of the noise search framework and reveal that they rely on the first residual as the discriminative metric to differentiate members and non-members. Inspired by this observation, we introduce OMS, which augments existing MIA methods by iterating One More fixed-point Step to include a further residual, i.e., the second residual. We integrate our method into various MIA methods across different diffusion models. The experimental results validate the efficacy of our proposed approach.

Due to its strong ability in distribution representation, the diffusion model has been incorporated into offline reinforcement learning (RL) to cover diverse trajectories of the complex behavior policy. However, this also causes several challenges. Training the diffusion model to imitate behavior from the collected trajectories suffers from limited stitching capability which derives better policies from suboptimal trajectories. Furthermore, the inherent randomness of the diffusion model can lead to unpredictable control and dangerous behavior for the robot. To address these concerns, we propose the Value-learning-based Decision Diffuser(V-DD), which consists of the trajectory diffusion module(TDM) and the trajectory evaluation module(TEM). During the training process, the TDM combines the state-value and classifier-free guidance to bolster the ability to stitch suboptimal trajectories. During the inference process, we design the TEM to select a feasible trajectory generated by the diffusion model. Empirical results demonstrate that our method delivers competitive results on the D4RL benchmark and substantially outperforms current diffusion model-based methods on the real-world robot task.

Denoising diffusion models have demonstrated tremendous success in modeling data distributions and synthesizing high-quality samples. In the 2D image domain, they have become the state-of-the-art and are capable of generating photo-realistic images with high controllability. More recently, researchers have begun to explore how to utilize diffusion models to generate 3D data, as doing so has more potential in real-world applications. This requires careful design choices in two key ways: identifying a suitable 3D representation and determining how to apply the diffusion process. In this survey, we provide the first comprehensive review of diffusion models for manipulating 3D content, including 3D generation, reconstruction, and 3D-aware image synthesis. We classify existing methods into three major categories: 2D space diffusion with pretrained models, 2D space diffusion without pretrained models, and 3D space diffusion. We also summarize popular datasets used for 3D generation with diffusion models. Along with this survey, we maintain a repository https://github.com/cwchenwang/awesome-3d-diffusion to track the latest relevant papers and codebases. Finally, we pose current challenges for diffusion models for 3D generation, and suggest future research directions.

Recent text-to-video diffusion models have achieved impressive progress. In practice, users often desire the ability to control object motion and camera movement independently for customized video creation. However, current methods lack the focus on separately controlling object motion and camera movement in a decoupled manner, which limits the controllability and flexibility of text-to-video models. In this paper, we introduce Direct-a-Video, a system that allows users to independently specify motions for multiple objects as well as camera’s pan and zoom movements, as if directing a video. We propose a simple yet effective strategy for the decoupled control of object motion and camera movement. Object motion is controlled through spatial cross-attention modulation using the model’s inherent priors, requiring no additional optimization. For camera movement, we introduce new temporal cross-attention layers to interpret quantitative camera movement parameters. We further employ an augmentation-based approach to train these layers in a self-supervised manner on a small-scale dataset, eliminating the need for explicit motion annotation. Both components operate independently, allowing individual or combined control, and can generalize to open-domain scenarios. Extensive experiments demonstrate the superiority and effectiveness of our method. Project page and code are available at https://direct-a-video.github.io/.

This paper presents a novel framework integrating the RePaint method with the performance-guided denoising diffusion probabilistic model (DDPM) to complete missing or undefined design components based on any given partial structures while satisfying specified performance targets. RePaint allows a pre-trained diffusion model to synthesize designs under the condition of partial designs, offering flexibility and controllability of generative design results. Using a dataset of parametric ship hull designs, a performance-guided diffusion model is pre-trained to generate designs with targeted total resistance coefficients. Experiment results demonstrate that our method generates new designs from random incomplete ship hull designs with performances close to those generated by the pre-trained models. The study further analyzes factors that might influence the new framework’s performance and reveals that the pre-trained model’s design parameter value distribution has a significant effect on the new method’s output designs. This distribution can be leveraged to guide the synthesis of new designs with desirable performance.

Text-conditioned image generation enables cross-modal comprehension. Recent emergence of many platforms have found applications in diverse domains like assisted designing and video gaming. However, there still exist challenges in existing platforms due to their expensive training and time-consuming generation processes. In this paper, we introduce an efficient text-conditioned image generation platform, termed InstantPainting. Unlike existing platforms based on large-scale pre-trained diffusion models, InstantPainting expands generative adversarial networks (GANs) to achieve efficient generation by using only about three percent pre-training data of other platforms. Compared to existing platforms, InstantPainting achieves the following functions at a very low deployment cost and approximately 4 to 5 times faster generation speeds: (1) Multi-category and multi-size image generation (2) Image stylization and controlled generation (3) Creative generation, including the generation of poetry pictures and counterfactual images. The proposed platform provides web application implementations for PC and mobile, users can create high-quality images directly through the user interface.