Human Behavior Recognition Method based on Image Sequences

To improve traditional video surveillance systems' performance on human behavior recognition, a new system has been built. Not visual camera but depth camera is chosen as sensor. To adapt to the most common forward oblique view of camera, a normalized digital elevation map, whose pixel intensity indicates the elevation of the scene, is built from the depth image. Coordinates and intensity in the digital elevation map represent the position information about the scene. Oriented templates are proposed to match and detect the human head robustly in the elevation map. As for the excellent visibility of human head in the elevation map, we track the human head to get trajectory. By combining the trajectory information of human head with the elevation map, several predefined human behaviors are recognized. Our behavior recognition method is straightforward and robust. This uniqueness has no similar with the traditional machine learning and classification framework about human behavior recognition.

Measuring changes in cardiac motion patterns can assist in diagnosing the onset of arrhythmia and ischaemia and in the follow-up of treatment. This work presents a methodology for measuring such motion changes from MR images. Non-rigid registration is used to track cardiac motion in a sequence of 3D tagged MR images. We use a cylindrical coordinate system to subdivide the myocardium into smaller anatomically meaningful regions and to express motion derived measurements such as displacement and strain for each myocardial region during the cardiac cycle. In the first experiment we have evaluated the proposed methods using synthetic image sequences where the ground truth was available. These images were generated using a cardiac motion simulator for tagged MRI. Normal and abnormal motion fields were produced by modifying parameters in a small region of the myocardium. In the second experiment we have acquired two separate tagged MR image sequences from five healthy volunteers. Both acquisitions have been carried out without moving the volunteer inside the scanner, thus avoiding potential misregistration errors due to subject motion between scans. In addition, one of volunteers was subjected to stress during one of the scans. In the final experiment we acquired tagged MR images from a patient with super-ventricular tachyarrhythmia, before and after radio frequency ablation. The image acquisition and catheter intervention were performed with a combined X-ray and MRI system. Detection results were correct on synthetic data and no region was incorrectly classified as having significant changes in the repetition studies. Significant changes in motion pattern were measured in the stress and ablation studies. Furthermore, results seem to corroborate that the ablation regularised cardiac contraction.

Big data analysis of human behavior can provide the basis and support for the application of various scenarios. Using sensors for human behavior analysis is an effective means of identification method, which is very valuable for research. To address the problems of low recognition accuracy, low recognition efficiency of traditional human behavior recognition (HBR) algorithms in complex scenes, in this paper, we propose an HBR algorithm for Mobile Big data analytics in wireless sensor network using improved transfer learning. First, different wireless sensors are fused to obtain human behavior mobile big data, and then by analyzing the importance of human behavior features (HBF), the dynamic change parameters of HBF extraction threshold are calculated. Second, combined with the dynamic change parameters of threshold, the HBF of complex scenes are extracted. Finally, the best classification function of human behavior in complex scenes is obtained by using the classification function of HBF in complex scenes. Human behavior in complex scenes is classified according to the HBF in the feature set. The HBR algorithm is designed by using the improved transfer learning network to realize the recognition of human behavior in complex scenes. The results show that the proposed algorithm can accurately recognize up to 22 HBF points, and can control the HBR time within 2 s. The human behavior false recognition rate of miscellaneous scenes is less than 10%. The recognition speed is above 10/s, and the recall rate can reach more than 98%, which improves the HBR ability of complex scenes.

In sports, because the movement of the human body is composed of the movements of the human limbs, and the complex and changeable movements of the human limbs lead to various and complicated movement modes of the entire human body, it is not easy to accurately track the human body movement. The recognition of human characteristic behavior belongs to a higher level computer vision topic, which is used to understand and describe the characteristic behavior of people, and there are also many research difficulties. Because the radial basis fuzzy neural network has the characteristics of parallel processing, nonlinearity, fault tolerance, self-adaptation, and self-learning, it has the advantage of high recognition efficiency when it is applied to the recognition of intersecting features and incomplete features. Therefore, this paper applies it to the analysis of the human body information recognition model in sports. The research results show that the human body information recognition model proposed in this paper has a high recognition accuracy and can detect the movement state of people in sports in real time and accurately.

While quantitative precipitation estimation (QPE) using weather radar is widely adopted in operation, precipitation data sets are often highly imbalanced. In particular, extreme precipitation usually lacks representation, which may introduce the bottleneck for radar QPE with machine learning models. Discovering the intrinsic characteristic of extreme precipitation with few samples is challenging. In this letter, we focus on the radar reflectivity data and aim to generate synthetic radar image sequences with respect to extreme precipitation. Considering the relatively long interval between continuous radar images due to radar volume scan, traditional methods in video generation are not suitable. In this letter, we propose Two-stage Generative Adversarial Networks (TsGANs) to address the above-mentioned problem. In general, our TsGAN constructs adversarial process between generators and discriminators: the generator produces samples similar to real data, while the discriminator determines whether or not a sample is eligible. In Stage I, we generate an image sequence containing content and motion features. In Stage II, we design an enhanced net structure to enrich the adversarial processes and further improve the motion features. Experimental testing is performed within the radar coverage in Shenzhen, China, on rainfall events in 2014-2016. Results show that our TsGAN is superior to previous works.

We present a new algorithm for the nonrigid registration of three-dimensional magnetic resonance (MR) intraoperative image sequences showing brain shift. The algorithm tracks key surfaces of objects (cortical surface and the lateral ventricles) in the image sequence using a deformable surface matching algorithm. The volumetric deformation field of the objects is then inferred from the displacements at the boundary surfaces using a linear elastic biomechanical finite-element model. Two experiments on synthetic image sequences are presented, as well as an initial experiment on intraoperative MR images showing brain shift. The results of the registration algorithm show a good correlation of the internal brain structures after deformation, and a good capability of measuring surface as well as subsurface shift. We measured distances between landmarks in the deformed initial image and the corresponding landmarks in the target scan. Cortical surface shifts of up to 10 mm and subsurface shifts of up to 6 mm were recovered with an accuracy of 1 mm or less and 3 mm or less respectively.

The entry process of virus particles into cells is decisive for infection. In this work, we investigate fusion of virus particles with the cell membrane via time-lapse fluorescence microscopy. To automatically identify fusion for single particles based on their intensity over time, we have developed a layered probabilistic approach. The approach decomposes the action of a single particle into three abstractions: the intensity over time, the underlying temporal intensity model, as well as a high level behavior. Each abstraction corresponds to a layer and these layers are represented via stochastic hybrid systems and hidden Markov models. We use a maxbelief strategy to efficiently combine both representations. To compute estimates for the abstractions we use a hybrid particle filter and the Viterbi algorithm. Based on synthetic image sequences, we characterize the performance of the approach as a function of the image noise. We also characterize the performance as a function of the tracking error. We have also successfully applied the approach to real image sequences displaying pseudotyped HIV-1 particles in contact with host cells and compared the experimental results with ground truth obtained by manual analysis.

Falling is a common but fatal human behavior in life. With the rapid growth of the aging population, fall-related human behavior recognition has been extensively investigated using radar. Nevertheless, human behavior recognition frequently exhibits suboptimal generalization capabilities due to the scarcity of labeled data. To address this problem, we propose physics-informed data augmentation (PIDA) for the micro-Doppler signature (m-DS) produced from frequency-modulated continuous wave (FMCW) radar to facilitate domain generalization (DG) of human behavior recognition. Specifically, leveraging classical electromagnetic wave propagation theory and radar signal processing principles, we introduce a PIDA strategy: 1) Distance Data Augmentation (DDA), which simulates electromagnetic attenuation effects across varying detection ranges, and 2) Behavior Pattern Data Augmentation (BPDA), designed to preserve intrinsic kinematic features while diversifying motion styles. Subsequently, combined with the proposed PIDA, a multi-source domain adversarial neural (MSDAN) network based on transfer learning is constructed to capture invariant features of human behaviors. To verify the superiority of PIDA, a series of experiments are conducted on various unseen domain data. The results showed that the PIDA method improves average DG accuracy by 1.52%, 2.48% and 4.88%, respectively, on the own dataset and 3.67%, 6.29% and 9.78%, respectively, on the public dataset, compared with baseline data, traditional optical data augmentation (TODA), and generative data augmentation (GDA) methods. Compared with current fall-related human behavior recognition works, our proposed method has a small DG gap.

The effects of maternal exercise on fetal heart rate and movement patterns.

Human behaviour recognition is one of the most fundamental tasks in Industrial Internet of Behaviour (IIoB) and is crucial for the safe and reliable IIoB. Existing methods lacks adaptability and transferability. In addition, there is a data isolation problem among different users. Therefore, there is an urgent requirement to construct a secure and adaptive human behaviour recognition model in IIoB without violating the privacy of users. Mamba, a structured state space model that integrates a selection mechanism and a scan module, is used for time series modelling tasks. To tackle the aforementioned problems, an Federated Learning-based lightweight human behaviour recognition model with selective state space models is proposed. First, we design a human behaviour recognition model integrating Mamba and residual structure to achieve lightweight and secure human behaviour modelling. In addition, considering data privacy and training efficiency, a decentralised dynamic FL framework is designed to achieve lightweight and secure model collaborative training, including: initial selection of source users, model aggregation strategy based on dynamic weighting, and fine-tuning module based on small-sample data, to improve the training efficiency of the model and the accuracy of human behaviour recognition. Extensive experiments are conducted to prove the superior performance of the proposed method.

Computing optical flow is one of the most fundamental problem to motion image analysis. Many methods have been proposed for computing optical flow, among them gradient-based methods are the most well-known and most used. A new gradient-based method for the computation of optical flow is proposed. In this method, optical flow is computed by minimizing a weighted least-squares error estimator for a constant motion vector model in a local spatial neighborhood, where the weight of each image location in the neighborhood is determined by its multiple constraints. Several experiments on real and synthetic image sequences have been carried out to verify the efficacy and the reliability of the new method.

The motion of the carotid artery wall has been previously estimated from ultrasound image sequences using block matching. In this paper, this conventional method was extended through its combination with Kalman filtering in two distinct scenarios; (a) by renewing the reference block and (b) by updating the estimate of the conventional algorithm. Both procedures were evaluated on synthetic image sequences through the estimation of the warping index. The results showed that incorporation of the Kalman filter in conventional block matching slightly improved the accuracy in arterial wall motion estimation. Updating the estimate of the conventional algorithm using Kalman filtering was the most efficient procedure and could be used to study further the displacements of the arterial wall in an attempt to obtain useful knowledge about arterial biomechanics. © 2010 IEEE.

Spatial segmentation of image sequences is usually computed based on motion between two frames. Some recent approaches extend this to joint segmentation in space-time; the resulting 3D segmentation (in x-y-t space) can be interpreted as a volume carved out by a moving in the image sequence domain, or the so-called object tunnel. We extend this concept to explicit modeling of events in the x-y-t space. In addition to the modeling of evolution, we also model occluded and newly-exposed areas in the background and in the by means of an occlusion volume, a new space-time concept. We propose a variational formulation of the problem that we solve using the multiphase level set method. We show experimental results for synthetic and natural image sequences.

Neuroscience studies have shown that incorporating gaze view with third view perspective has a great influence to correctly infer human behaviors. Given the importance of both first and third person observations for the recognition of human behaviors, we propose a method that incorporates these observations in a technical system to enhance the recognition of human behaviors, thus improving beyond third person observations in a more robust human activity recognition system. First, we present the extension of our proposed semantic reasoning method by including gaze data and external observations as inputs to segment and infer human behaviors in complex real-world scenarios. Then, from the obtained results we demonstrate that the combination of gaze and external input sources greatly enhance the recognition of human behaviors. Our findings have been applied to a humanoid robot to online segment and recognize the observed human activities with better accuracy when using both input sources; for example, the activity recognition increases from 77% to 82% in our proposed pancake-making dataset. To provide completeness of our system, we have evaluated our approach with another dataset with a similar setup as the one proposed in this work, that is, the CMU-MMAC dataset. In this case, we improved the recognition of the activities for the egg scrambling scenario from 54% to 86% by combining the external views with the gaze information, thus showing the benefit of incorporating gaze information to infer human behaviors across different datasets.

Human behavior recognition is an active research field in computer vision and image processing. A novel method is proposed for human behavior recognition in video image sequences. First of all, a video sequence is represented by extracting space-time interest points. Then Human behavior is represented by activities through Motion Decomposition. The activity comprises labeled bags that are composed of unlabeled instances comprising to action. Final labeled activities are used to train a strong classifier which is used to predict the labels of unseen behavior bags. Experimental results show the effectiveness of the proposed method in comparison with other related works in the literature and can also tolerate noise and interference conditions.