Mapping Vicon Motion Tracking to 6-Axis IMU Data for Wearable Activity Recognition

Human activity recognition is a very active research on pervasive computing and mobile health application. Many human activity systems based on inertial measurement unit (IMU) sensor data were proposed in the past few years. These systems mainly use IMU sensor placed on he torso and limbs to collect data and utilize supervised machine learning algorithms on sensor data. One main issue of these systems is that wearing multiple on-body IMU sensors may bring inconvenience to users' daily life. The other issue of these exiting methods is that an activity recognition model that is trained on a specific subject does not work well when being applied to predict another subject's activities since IMU activity data always carry information that is specific to the human subject who conducts the activities. In our work, inspired by the principle of domain adaption, we proposed a new deep-learning activity recognition model based on an adversarial network which can remove the subject-specific information within the IMU activity data and extract subject-independent features shared by the data collected on different subjects. We also for the first time use data collected from insole based IMU sensors on 8 participants for 5 common activities to build a new real world human activity dataset which can minimize the inconvenience for users to wear. We conducted experiments with our new real-world dataset. Results show that our subject independent activity recognition model outperforms state-of-art supervised learning techniques and eliminates the effects of individual differences between subjects successfully. The average recognition accuracy under the leave-one-out (L1O) condition achieves 99.0% which is higher than the performance of traditional human activity recognition system based on CNNs.

There has been a proliferation of smartphones, smart watches, and wearable sensors, making them ubiquitous in our daily lives. Mobile sensors have found widespread use due to their ever-decreasing cost, ease of deployment, and ability to provide continuous monitoring as opposed to sensors installed at fixed locations. Various techniques have been proposed for fall detection, gait analysis, activity monitoring, and heart rate and sleep sensing by wearable sensors and mobile phones. Compared to works that use inertial measurement unit (IMU) data or static cameras installed in the environment, there has been relatively less work using egocentric videos, meaning providing the first-person view from wearable cameras. Moreover, most of the existing studies on egocentric videos are based on only one sensor modality, namely the camera. There have been even fewer approaches that combine egocentric video data with IMU data. In this chapter, we will describe three different applications using wearable cameras together with IMU data. First, we will present an overview of a fall detection system using wearable devices, e.g., smartphones and tablets, equipped with cameras and accelerometers. Since the portable device is worn by the subject, monitoring is not limited to confined areas, and extends to wherever the subject may travel, as opposed to static sensors installed in certain rooms. Second, we will present an autonomous and robust method for counting footsteps, and tracking and calculating stride length by using both accelerometer and camera data from smartphones or Google™ glass. To provide higher precision, instead of using a preset stride length, the proposed method calculates the distance traveled with each step by using the camera data. This method is compared with the commercially available accelerometer-based step counter apps. The results show that the proposed method provides a significant increase in accuracy, and has the lowest average error rate both in number of steps taken and the distance traveled. Finally, we will provide an overview of a robust and autonomous method to detect activities with more details and context by using accelerometer and egocentric video data obtained from a smartphone.

Clinicians lack objective means for monitoring if their knee osteoarthritis patients are improving outside of the clinic (e.g., at home). Previous human activity recognition (HAR) models using wearable sensor data have only used data from healthy people and such models are typically imprecise for people who have medical conditions affecting movement. HAR models designed for people with knee osteoarthritis have classified rehabilitation exercises but not the clinically relevant activities of transitioning from a chair, negotiating stairs and walking, which are commonly monitored for improvement during therapy for this condition. Therefore, it is unknown if a HAR model trained on data from people who have knee osteoarthritis can be accurate in classifying these three clinically relevant activities. Therefore, we collected inertial measurement unit (IMU) data from 18 participants with knee osteoarthritis and trained convolutional neural network models to identify chair, stairs and walking activities, and phases. The model accuracy was 85% at the first level of classification (activity), 89–97% at the second (direction of movement) and 60–67% at the third level (phase). This study is the first proof-of-concept that an accurate HAR system can be developed using IMU data from people with knee osteoarthritis to classify activities and phases of activities.

Virtual Inertial Measurement Units (IMUs) offer a promising approach to generating synthetic motion data for training and evaluating human activity recognition (HAR) systems. However, existing virtual IMU workflows remain fragmented and technically demanding, requiring users to switch between 3D editors, scripting tools, and offline signal processing pipelines. These limitations hinder usability and iterative refinement for researchers and developers who design HAR systems. We present Vsens , an XR-based system that unifies virtual IMU configuration, visualization, and data synthesis within an immersive workspace. Vsens allows developers to directly manipulate sensor placements on digital avatars and observe synthesized IMU signals in real time. To understand how developers with different expertise levels benefit from such an environment, we conducted a user study with 20 HAR developers covering a broad range of experience levels. Results show that Vsens improves configuration efficiency and fosters deeper spatial understanding of sensor behavior across the participants. Beyond usability, our findings reveal how embodied interaction and real-time feedback inform the design of practical virtual data collection systems that foster more effective and human-centered workflows.

Activity recognition systems are used in rehabilitation centres to monitor activity of daily living in order to assess daily functional status of elderly. A low-cost, non-invasive, and continuous wearable activity monitoring system can be realized by one or multiple wearable sensor nodes to form a self-managing wireless medical body area network. There are several arising challenges essential to be dealt within developing wearable activity recognition systems, namely sensor node lifetime and detection accuracy. This paper investigates existing solutions, which address the key opposing challenges. We propose a feedback controller algorithm to dynamically adapt sampling rate for maintaining the tradeoff between the energy efficiency and accuracy. The Number of samples and transmitted data packets is the main sources of energy consumption that impacts the system accuracy. To validate the accuracy of our proposed algorithm, a public wearable activity recognition data set is constructed. The data set is collected from 20 healthy subjects over 7 activity types excluding the transition states, using up to four accelerometer sensors connected with IEEE 802.15.4 enabled nodes in our setup. Our proposed feedback controller algorithm nearly doubles the activity recognition system lifetime. This, in turn improves the users' quality of experience by reducing the demand for battery replacements while the accuracy of detection is maintained at the same level.

IoT-FAR: A multi-sensor fusion approach for IoT-based firefighting activity recognition

Attitude estimation arises in many applications, and is crucial for the aerospace and aeronautic industry. A classical approach to perform the attitude estimation is via Inertial Measurement Unit (IMU) data and a set of measured observation vectors, both in the object and inertial reference frame. The attitude estimation can be then formulated as a constrained least-squares problem. As a possible alternative, attitude estimation can be performed using IMU data and a set of measured positions, hence removing the need for observation vectors. Attitude estimation based on IMU and position measurements is already undergoing industrial research. However, the underlying estimation problem does not necessarily admit a unique solution, and can therefore degenerate depending on the object trajectories. This paper theoretically investigates this approach to attitude estimation, and provides a simple, explicit relationship between the standard deviation of the attitude estimation and the object trajectories, which is independent of the choice of parametrization of the special orthogonal group SO(3).

Assembly carries paramount importance in manufacturing. Being able to support workers in real time to maximize their positive contributions to assembly is a tremendous interest of manufacturers. Human action recognition has been a way to automatically analyze and understand worker actions to support real-time assistance for workers and facilitate worker–machine collaboration. Assembly actions are distinct from activities that have been well studied in the action recognition literature. Actions taken by assembly workers are intricate, variable, and may involve very fine motions. Therefore, recognizing assembly actions remains a challenging task. This paper proposes to simply use only two wearable devices that respectively capture the inertial measurement unit data of each hand of workers. Then, two convolutional neural network models with an identical architecture are independently trained using the two sources of inertial measurement unit data to respectively recognize the right-hand and the left-hand actions of an assembly worker. Classification results of the two convolutional neural network models are fused to yield a final action recognition result because the two hands often collaborate in assembling operations. Transfer learning is implemented to adapt the action recognition models to subjects whose data have not been included in dataset for training the models. One operation in assembling a Bukito three-dimensional printer, which is composed of seven actions, is used to demonstrate the implementation and assessment of the proposed method. Results from the study have demonstrated that the proposed approach effectively improves the prediction accuracy at both the action level and the subject level. Work of the paper builds a foundation for building advanced action recognition systems such as multimodal sensor-based action recognition.

Inertial Measurement Units (IMUs) are widely utilized in shoulder rehabilitation due to their portability and cost-effectiveness, but their reliance on spatial motion data restricts their use in comprehensive musculoskeletal analyses. To overcome this limitation, we propose SWCTNet (Sliding Window CNN + Channel-Time Attention Transformer Network), an advanced neural network specifically tailored for multichannel temporal tasks. SWCTNet integrates IMU and surface electromyography (sEMG) data through sliding window convolution and channel-time attention mechanisms, enabling the efficient extraction of temporal features. This model enables the prediction of muscle activation patterns and kinematics using exclusively IMU data. The experimental results demonstrate that the SWCTNet model achieves recognition accuracies ranging from 87.93% to 91.03% on public temporal datasets and an impressive 98% on self-collected datasets. Additionally, SWCTNet exhibits remarkable precision and stability in generative tasks: the normalized DTW distance was 0.12 for the normal group and 0.25 for the patient group when using the self-collected dataset. This study positions SWCTNet as an advanced tool for extracting musculoskeletal features from IMU data, paving the way for innovative applications in real-time monitoring and personalized rehabilitation at home. This approach demonstrates significant potential for long-term musculoskeletal function monitoring in non-clinical or home settings, advancing the capabilities of IMU-based wearable devices.

Autonomous driving relies on reliable localization, but the Global Navigation Satellite System (GNSS) often loses accuracy or becomes unavailable in harsh environments, such as tunnels and dense urban canyons. Visual-Inertial Simultaneous Localization and Mapping (VI-SLAM), which fuses camera and Inertial Measurement Unit (IMU) data, is a key alternative in GNSS-denied scenarios. However most VI-SLAM methods overlook vehicles' inherent periodic motion features, such as zero velocity during stops or gear shifts, and linear motion during cruising. To address this, this paper proposes a visual-inertial localization method integrating these features into a tightly coupled VI-SLAM framework. First, based on IMU data, the Generalized Likelihood Ratio Test is used to detect zero velocity and linear motion. Then, Zero Velocity Update (ZUPT) residuals and Nonholonomic Constraint (NHC) residuals are constructed. These two residuals are jointly optimized with visual residuals, IMU pre-integration residuals, and marginalization residuals within a sliding-window nonlinear backend for Visual-Inertial Odometry. Experiments on self-collected datasets show that compared to the baseline algorithm, both the ZUPT-integrated algorithm and the NHC-integrated algorithm yield higher accuracy, lower error variance, and stronger robustness.

The conventional Human Activity Recognition (HAR) systems use smartwatches, smartphones, and other wearable devices to autonomously derive consumer physical activities. However, implementing real-time HAR in resource-constrained applications is challenging. Computational offloading approaches must address network reliance, latency, and data privacy, which affect system performance. The present study proposes a low-cost, wearable HAR device that collects and transmits the Inertial Measurement Unit (IMU) telemetry signals to smartphones, enabling real-time on-device activity detection. Human actions are subjective and affected by physiological, environmental, and behavioural factors, complicating HAR systems' performance in personalized contexts. Generalized methods for existing systems reduce performance for new consumers or due to alterations in consumer locomotive signatures. Thus, our second goal is to enable the on-device personalization of wearable HAR models with minimum customer calibration. The HAR device, which is in the form of a smart-band, transfers data to and fro smartphones through Bluetooth Low Energy (BLE). A lightweight 1D convolutional neural network (CNN) is built and, using transfer learning, the network model is fine-tuned from real-time sensed data. On-device HAR inferences ensure real-time processing on smartphones without computational offloading. Smartphone-based on-device training for fine-tuning the model allows the HAR system to get customized without compromising privacy or computational costs. The experimental results show that the personalized classification achieves on average 98% accuracy when assessed using four benchmark datasets.

Deep learning techniques for human activity recognition are gaining popularity due to their effectiveness in identifying intricate tasks and their cost-effectiveness compared to conventional machine learning methods. Human Activity Recognition (HAR) is a research domain concerned with detecting the everyday activities performed by individuals using time-series data captured by sensors. HAR encompasses a wide range of applications, including surveillance, baby monitoring, elderly healthcare, and smart driving. This article provides a brief introduction to the application of deep learning in HAR. It covers the fundamental concepts of CNNs and LSTMs, their strengths in capturing spatial and temporal features, and their integration for enhanced activity recognition. Different approaches are employed in HAR to address problems with efficiency and precision. Conventional human activity recognition (HAR) systems rely on wearable devices like IMUs and stretch sensors to identify different activities. These systems have proven to be effective in recognizing simple user actions like sitting, standing, and walking. However, when it comes to more intricate activities like running, jumping, wrestling, and swinging, sensor-based HAR systems encounter greater misclassification rates due to inaccuracies in sensor readings. These errors significantly impact the accuracy of the HAR system, resulting in suboptimal classification outcomes. In contrast, employing vision-based HAR systems enables improved accuracy in identifying complex activities, leading to enhanced overall performance.

Assessment of varus thrust using inertial measurement units.

Objective: Older adults’ falls are a critical public health problem. The majority of free-living fall risk assessment methods have investigated fall predictive power of step-related digital biomarkers extracted from wearable inertial measurement unit (IMU) data. Alternatively, the examination of characteristics and frequency of naturally-occurring compensatory balance reactions (CBRs) may provide valuable information on older adults’ propensity for falls. To address this, models to automatically detect naturally-occurring CBRs are needed. However, compared to steps, CBRs are rare events. Therefore, prolonged collection of criterion standard data (along with IMU data) is required to validate model’s performance in free-living conditions. Methods: By investigating 11 fallers’ and older non-fallers’ free-living criterion standard data, 8 naturally-occurring CBRs, i.e., 7 trips (self-reported using a wrist-mounted voice-recorder) and 1 hit/bump (verified using egocentric vision data) were localized in the corresponding trunk-mounted IMU data. Random forest models were trained on independent/unseen datasets curated from multiple sources, including in-lab data captured using a perturbation treadmill. Subsequently, the models’ translation/generalization to older adults’ out-of-lab data were assessed. Results: A subset of models differentiated between naturally-occurring CBRs and free-living activities with high sensitivity (100%) and specificity (≥99%). Conclusions: The findings suggest that accurate detection of naturally-occurring CBRs is feasible. Clinical/Translational Impact- As a multi-institutional validation study to detect older adults’ naturally-occurring CBRs, suitability for larger-scale free-living studies to investigate falls etiology, and/or assess the effectiveness of perturbation training programs is discussed.

Freezing-of-gait (FOG) and impaired walking are common features of Parkinson’s disease (PD). Provision of external stimuli (cueing) can improve gait, however, many cueing methods are simplistic, increase task loading or have limited utility in a real-world setting. Closed-loop (automated) somatosensory cueing systems have the potential to deliver personalised, discrete cues at the appropriate time, without requiring user input. Further development of cue delivery methods and FOG-detection are required to achieve this. In this feasibility study, we aimed to test if FOG-initiated vibration cues applied to the lower-leg via wearable devices can improve gait in PD, and to develop real-time FOG-detection algorithms. 17 participants with Parkinson’s disease and daily FOG were recruited. During 1 h study sessions, participants undertook 4 complex walking circuits, each with a different intervention: continuous rhythmic vibration cueing (CC), responsive cueing (RC; cues initiated by the research team in response to FOG), device worn with no cueing (NC), or no device (ND). Study sessions were grouped into 3 stages/blocks (A-C), separated by a gap of several weeks, enabling improvements to circuit design and the cueing device to be implemented. Video and onboard inertial measurement unit (IMU) data were analyzed for FOG events and gait metrics. RC significantly improved circuit completion times demonstrating improved overall performance across a range of walking activities. Step frequency was significantly enhanced by RC during stages B and C. During stage C, > 10 FOG events were recorded in 45% of participants without cueing (NC), which was significantly reduced by RC. A machine learning framework achieved 83% sensitivity and 80% specificity for FOG detection using IMU data. Together, these data support the feasibility of closed-loop cueing approaches coupling real-time FOG detection with responsive somatosensory lower-leg cueing to improve gait in PD.