Sensor-Based Human Activity and Behavior Computing

To improve traditional video surveillance systems' performance on human behavior recognition, a new system has been built. Not visual camera but depth camera is chosen as sensor. To adapt to the most common forward oblique view of camera, a normalized digital elevation map, whose pixel intensity indicates the elevation of the scene, is built from the depth image. Coordinates and intensity in the digital elevation map represent the position information about the scene. Oriented templates are proposed to match and detect the human head robustly in the elevation map. As for the excellent visibility of human head in the elevation map, we track the human head to get trajectory. By combining the trajectory information of human head with the elevation map, several predefined human behaviors are recognized. Our behavior recognition method is straightforward and robust. This uniqueness has no similar with the traditional machine learning and classification framework about human behavior recognition.

Neuroscience studies have shown that incorporating gaze view with third view perspective has a great influence to correctly infer human behaviors. Given the importance of both first and third person observations for the recognition of human behaviors, we propose a method that incorporates these observations in a technical system to enhance the recognition of human behaviors, thus improving beyond third person observations in a more robust human activity recognition system. First, we present the extension of our proposed semantic reasoning method by including gaze data and external observations as inputs to segment and infer human behaviors in complex real-world scenarios. Then, from the obtained results we demonstrate that the combination of gaze and external input sources greatly enhance the recognition of human behaviors. Our findings have been applied to a humanoid robot to online segment and recognize the observed human activities with better accuracy when using both input sources; for example, the activity recognition increases from 77% to 82% in our proposed pancake-making dataset. To provide completeness of our system, we have evaluated our approach with another dataset with a similar setup as the one proposed in this work, that is, the CMU-MMAC dataset. In this case, we improved the recognition of the activities for the egg scrambling scenario from 54% to 86% by combining the external views with the gaze information, thus showing the benefit of incorporating gaze information to infer human behaviors across different datasets.

The violence detection is mostly achieved through handcrafted feature descriptors, while some researchers have also employed deep learning-based representation models for violent activity recognition. Deep learning-based models have achieved encouraging results for fight activity recognition on benchmark data sets such as hockey and movies. However, these models have limitations in learning discriminating features for violence activity classification with abrupt camera motion. This research work investigated deep representation models using transfer learning for handling the issue of abrupt camera motion. Consequently, a novel deep multi-net (DMN) architecture based on AlexNet and GoogleNet is proposed for violence detection in videos. AlexNet and GoogleNet are top-ranked pre-trained models for image classification with distinct pre-learnt potential features. The fusion of these models can yield superior performance. The proposed DMN unleashed the integrated potential by concurrently coalescing both networks. The results confirmed that DMN outperformed state-of-the-art methods by learning finest discriminating features and achieved 99.82% and 100% accuracy on hockey and movies data sets, respectively. Moreover, DMN has faster learning capability i.e. 1.33 and 2.28 times faster than AlexNet and GoogleNet, which makes it an effective learning architecture on images and videos.

Due to its applications for the betterment of human life, human activity recognition has attracted more researchers in the recent past. Anticipation of intension behind the motion and behaviour recognition are intensive applications for research inside human activity recognition. Gyroscope, accelerometer, and magnetometer sensors are heavily used to obtain the data in time series for every timestep. The selection of temporal features is required for the successful recognition of human motion primitives. Different data pre-processing and feature extraction techniques were used in most past approaches with the constraint of sufficient domain knowledge. These approaches are heavily dependent on the quality of handcrafted features and are also time-consuming and not generalized. In this paper, a single head deep neural network-based approach with the combination of a convolutional neural network, Gated recurrent unit, and Long Short Term memory is proposed. The raw data from wearable sensors are used with minimum pre-processing steps and without the involvement of any feature extraction method. 93.48 % and 98.51% accuracy are obtained on UCI-HAR and WISDM datasets. This single-head deep neural network-based model shows higher classification performance over other architectures under deep neural networks.

Big data analysis of human behavior can provide the basis and support for the application of various scenarios. Using sensors for human behavior analysis is an effective means of identification method, which is very valuable for research. To address the problems of low recognition accuracy, low recognition efficiency of traditional human behavior recognition (HBR) algorithms in complex scenes, in this paper, we propose an HBR algorithm for Mobile Big data analytics in wireless sensor network using improved transfer learning. First, different wireless sensors are fused to obtain human behavior mobile big data, and then by analyzing the importance of human behavior features (HBF), the dynamic change parameters of HBF extraction threshold are calculated. Second, combined with the dynamic change parameters of threshold, the HBF of complex scenes are extracted. Finally, the best classification function of human behavior in complex scenes is obtained by using the classification function of HBF in complex scenes. Human behavior in complex scenes is classified according to the HBF in the feature set. The HBR algorithm is designed by using the improved transfer learning network to realize the recognition of human behavior in complex scenes. The results show that the proposed algorithm can accurately recognize up to 22 HBF points, and can control the HBR time within 2 s. The human behavior false recognition rate of miscellaneous scenes is less than 10%. The recognition speed is above 10/s, and the recall rate can reach more than 98%, which improves the HBR ability of complex scenes.

Fog Computing is an approach involving smart devices. These devices carry out data processing to provide collaborative services to reach a common goal, usually, in the cloud. In the fog computing paradigm, uncertainty and vagueness are inherent to the data processing due to the limitations of computational and communication capabilities of the smart devices. Fuzzy logic and protoforms represent a powerful tool to model and compute imprecise data presented within the fog-computing paradigm. In this paper, we present a fuzzy cloud-fog approach based on fuzzy temporal windows and fuzzy aggregation. The innovations of this paper are: i) to model the uncertainty involved in fog nodes linguistically, ii) to compute and distribute relevant linguistic information (protoforms), and iii) to publish the computed protoforms in the cloud to generate complex protoforms, which reach the common goal. This new fuzzy cloud-fog approach is applied to the problem of activity recognition in smart homes. In this context, the smart devices in a smart home are represented by fog nodes, which cooperate for activity recognition using a fuzzy cloud-fog computing approach to provide solutions in ambient-assisted living. Finally, to demonstrate the effectiveness of the proposal in handling situations/environments where multiple and heterogeneous devices are involved (such as UWB beacons, smart objects and smart wearable devices), a case study of the proposed fuzzy cloud-fog approach is implemented in the smart lab of the University of Jaén. So, the results obtained with the proposed approach in the case study are compared to the results obtained with a non-fuzzy approach with the aim of showing the advantages of the fuzzy methodology.

Falling is a common but fatal human behavior in life. With the rapid growth of the aging population, fall-related human behavior recognition has been extensively investigated using radar. Nevertheless, human behavior recognition frequently exhibits suboptimal generalization capabilities due to the scarcity of labeled data. To address this problem, we propose physics-informed data augmentation (PIDA) for the micro-Doppler signature (m-DS) produced from frequency-modulated continuous wave (FMCW) radar to facilitate domain generalization (DG) of human behavior recognition. Specifically, leveraging classical electromagnetic wave propagation theory and radar signal processing principles, we introduce a PIDA strategy: 1) Distance Data Augmentation (DDA), which simulates electromagnetic attenuation effects across varying detection ranges, and 2) Behavior Pattern Data Augmentation (BPDA), designed to preserve intrinsic kinematic features while diversifying motion styles. Subsequently, combined with the proposed PIDA, a multi-source domain adversarial neural (MSDAN) network based on transfer learning is constructed to capture invariant features of human behaviors. To verify the superiority of PIDA, a series of experiments are conducted on various unseen domain data. The results showed that the PIDA method improves average DG accuracy by 1.52%, 2.48% and 4.88%, respectively, on the own dataset and 3.67%, 6.29% and 9.78%, respectively, on the public dataset, compared with baseline data, traditional optical data augmentation (TODA), and generative data augmentation (GDA) methods. Compared with current fall-related human behavior recognition works, our proposed method has a small DG gap.

Nowadays, the human behavior analysis by computer vision techniques has been an interesting issue for researchers. Automatic recognition of actions in video allows automation of many otherwise manually intensive tasks such as video surveillance. Video surveillance system especially for elderly care and their behavior analysis has an important role to take care of aged, impatient or bedridden persons. In this paper, we propose a high accuracy human action classification and recognition method using hidden Markov model classifier. In our approach, first, we use star skeleton feature extraction method to extract extremities of human body silhouette to produce feature vectors as inputs of hidden Markov model classifier. Then, hidden Markov model, which is learned and used in our proposed surveillance system, classifies the investigated behaviors and detects abnormal actions with high accuracy in comparison by other abnormal detection reported in previous works. The accuracy about 94% resulted from confusion matrix approve the efficiency of the proposed method when compared with its counterparts for abnormal action detection.

With the development of society and the continuous progress of science and technology, it has become the mainstream measure to promote the development of the social economy through science and technology. Therefore, to improve the current situation of tourism consumption, improve the consumer sentiment of tourists, and promote the development of the tourism economy, the convolutional neural network (CNN) technology model is used to analyze the tourist's consumer psychology and behavior. Based on this, the user's consumption situation is analyzed, thus providing support for the intelligent improvement of tourism consumption. First, the basic characteristics of tourism consumption mood and behavior are introduced, and the methods to improve the tourism consumption mood and behavior are briefly introduced. Then, the CNN algorithm is employed to identify consumers' travel consumption behaviors and emotions. To improve the recognition effect, the algorithm is combined with skeleton node behavior recognition and video image behavior recognition. Finally, the performance of the designed algorithm is tested. The accuracy of the human behavior recognition (HBR) algorithm is more than 0.88. Compared with the detection effect of the HBR algorithm, the combined algorithm adopted in this work can reduce the image processing time and improve the detection efficiency. The multithread method can effectively reduce the complexity of the model and improve the recognition accuracy. The test results on different data sets show that the proposed algorithm can better adapt to the changes in identification samples and obtain more accurate recognition results compared with similar algorithms. In summary, this study not only provides technical support for the rational analysis of consumer sentiment and consumer behavior but also contributes to the comprehensive development of the tourism market.

Human behaviour recognition is one of the most fundamental tasks in Industrial Internet of Behaviour (IIoB) and is crucial for the safe and reliable IIoB. Existing methods lacks adaptability and transferability. In addition, there is a data isolation problem among different users. Therefore, there is an urgent requirement to construct a secure and adaptive human behaviour recognition model in IIoB without violating the privacy of users. Mamba, a structured state space model that integrates a selection mechanism and a scan module, is used for time series modelling tasks. To tackle the aforementioned problems, an Federated Learning-based lightweight human behaviour recognition model with selective state space models is proposed. First, we design a human behaviour recognition model integrating Mamba and residual structure to achieve lightweight and secure human behaviour modelling. In addition, considering data privacy and training efficiency, a decentralised dynamic FL framework is designed to achieve lightweight and secure model collaborative training, including: initial selection of source users, model aggregation strategy based on dynamic weighting, and fine-tuning module based on small-sample data, to improve the training efficiency of the model and the accuracy of human behaviour recognition. Extensive experiments are conducted to prove the superior performance of the proposed method.

In recent years, thermal imaging cameras are widely used in the field of intelligent surveillance because of their special imaging characteristics and better privacy protection properties. However, due to the low resolution and fixed location for current thermal imaging cameras, it is difficult to effectively identify human behavior using a single detection method based on skeletal keypoints. Therefore, a self-update learning method is proposed for fixed thermal imaging camera scenes, called the behavioral parameter field (BPF). This method can express the regularity of human behavior patterns concisely and directly. Firstly, the detection accuracy of small targets under low-resolution video is improved by optimizing the YOLOv4 network to obtain a human detection model under thermal imaging video. Secondly, the BPF model is designed to learn the human normal behavior features at each position. Finally, based on the learned BPF model, we propose to use metric modules, such as cosine similarity and intersection over union matching, to accomplish the classification of human abnormal behaviors. In the experimental stage, the living scene of the indoor elderly living alone is applied as our experimental case, and a variety of detection models are compared to the proposed method for verifying the effectiveness and practicability of the proposed behavioral parameter field in the self-collected thermal imaging dataset for the indoor elderly living alone.

Orthogonally constrained matrix factorization for robust unsupervised feature selection with local preserving

The current research on still image recognition has been very successful, but the study of action recognition for video classes is still a challenging topic. In this work, we propose a random projection‐based human action recognition algorithm to address the lack of depth information in color information (RGB video frames) that is not easily affected by environmental factors such as illumination and the lack of ability to recognize actions along the direction of view. A network structure is designed to take the obvious advantage of long‐ and short‐term memory networks for controlling and remembering long sequences of historical information. The network structure in this paper is constituted by multiple memory units. At the same time, this paper constructs the spatial features, temporal features, and depth features of the three recognition stream outputs into a feature matrix, whose feature matrix is divided into multiple temporal segments according to the temporal dimension, then inputs them into the network layer in order, and achieves the fusion of the feature matrix in this paper according to their correlation characteristics on the temporal axis. Here, we proposed the concept of random batch projection operators. This basically uses as much sublimitation information as possible to improve projection accuracy by randomly selecting several subdependencies as projections defined during projection. A compressed sensing design of human motion acceleration data for low‐power body area networks is proposed, and the basic idea and implementation process of compressed sensing theory for human motion data compression and reconstruction in wireless body area networks are introduced in detail.

Human behaviour recognition is an important research hotspot in the field of artificial intelligence. Current behaviour recognition methods have low recognition accuracy under different viewing angles, therefore, this paper proposes a novel restricted Boltzmann machine (RBM)-based temporal-spatial correlation method for student behaviour recognition in depth video. The RBM is used to map the human behaviour from different viewing angles to the high-dimensional space. The time level pooling function is applied in the time series activated by each neuron to realise the encoding of the video time sub-series. Finally, behaviour recognition and classification experiments are conducted on different public datasets and real classroom student behaviour datasets with other methods. The results show that the proposed method improves the accuracy of depth video recognition under different viewing angles and has good generalisation performance. The data analysis of abnormal behaviour in class can play an auxiliary role in dynamic classroom management.

Against a complex background, we can accurately recognize targets with our eyes. In the field of research on robot vision, it has become a hot research topic to acquire interesting information from the surroundings through image sequences and then recognize targets. This paper studies the acquisition and matching of human body behavior data based on binocular stereo vision, extracts sparse feature points through the set calibration method, and extracts more accurate matching points by comparing the point matching method of limit constraint and the matching method of combining point features and regional matching, thus increasing the stability of feature points and improving the accuracy of acquisition. Through experiments, it is proved that the method presented in this paper for the acquisition of human body behavior data is practical and feasible.