Attitude estimation based on inertial and position measurements

The vast majority of modern consumer cameras employ a rolling shutter (RS) mechanism which has a price and electronic advantage to global shutter (GS). However, in geometric computer vision applications such as visual simultaneous localization and mapping (VSLAM), performances of accuracy and robustness are usually deteriorated due to the rolling shutter effect when using the RS cameras. This paper introduced the Wuhan University Rolling Shutter Visual-Inertial (WHU-RSVI) synthetic dataset for evaluating VSLAM and VI-SLAM (visual-inertial SLAM) methods in which RS cameras or IMU data are typically used. The proposed synthetic dataset contains RS images, time-synchronized GS images, inertial measurement unit (IMU) measurements, and accurate ground truth. It provides camera images with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480 $ </tex-math></inline-formula> resolution at 30 Hz and IMU measurements from 90 Hz to 14400 Hz. The cubic B-spline curves are used to model the motion of trajectories. Based on the known trajectories, an image of each pose can be rendered, and the corresponding IMU measurement model is then established. The dataset provides realistic images and IMU measurements by modeling the sensor noise in RGB and IMU data. Two trajectories with three sequences of different motion speeds (i. e., slow, medium and fast corresponding to different rolling shutter effects) are contained in the proposed dataset. Herein, the proposed dataset can be applied to compare the impact of different rolling shutter effects on a specific method.

There has been a proliferation of smartphones, smart watches, and wearable sensors, making them ubiquitous in our daily lives. Mobile sensors have found widespread use due to their ever-decreasing cost, ease of deployment, and ability to provide continuous monitoring as opposed to sensors installed at fixed locations. Various techniques have been proposed for fall detection, gait analysis, activity monitoring, and heart rate and sleep sensing by wearable sensors and mobile phones. Compared to works that use inertial measurement unit (IMU) data or static cameras installed in the environment, there has been relatively less work using egocentric videos, meaning providing the first-person view from wearable cameras. Moreover, most of the existing studies on egocentric videos are based on only one sensor modality, namely the camera. There have been even fewer approaches that combine egocentric video data with IMU data. In this chapter, we will describe three different applications using wearable cameras together with IMU data. First, we will present an overview of a fall detection system using wearable devices, e.g., smartphones and tablets, equipped with cameras and accelerometers. Since the portable device is worn by the subject, monitoring is not limited to confined areas, and extends to wherever the subject may travel, as opposed to static sensors installed in certain rooms. Second, we will present an autonomous and robust method for counting footsteps, and tracking and calculating stride length by using both accelerometer and camera data from smartphones or Google™ glass. To provide higher precision, instead of using a preset stride length, the proposed method calculates the distance traveled with each step by using the camera data. This method is compared with the commercially available accelerometer-based step counter apps. The results show that the proposed method provides a significant increase in accuracy, and has the lowest average error rate both in number of steps taken and the distance traveled. Finally, we will provide an overview of a robust and autonomous method to detect activities with more details and context by using accelerometer and egocentric video data obtained from a smartphone.

There are several large datasets available that are captured from motion tracking systems which could be useful to train wearable human activity recognition (HAR) systems, if only their spatial data could be mapped into the equivalent inertial measurement unit (IMU) data that would be sensed on the body. In this paper, we describe a mapping from 3D Vicon motion tracking data to data collected from a BlueSense on-body IMU. We characterise the error incurred in order to discern the extent to which it is possible to generate useful training data for a wearable activity recognition system from data collected with a motion capture system. We analyse this by mapping Vicon motion tracking data to rotational velocity and linear acceleration at the head, and compare this to actual gyroscope and accelerometer data collected by an IMU mounted on the head. In a 15 min dataset comprising three static activities—sitting, standing and lying down—we find that 95% of the reconstructed gyroscope data is within an error of [−7.25;+7.46] \(deg \cdot s^{-1}\), while 95% of the reconstructed accelerometer data was contained within [−96.1;+72.9] \(m \cdot G\). However, when we introduce more movement by including data collected while walking this increases to [−19.0;+18.2] \(deg \cdot s^{-1}\) for the gyroscope and [−208;+186] \(m \cdot G\) for the accelerometer. We conclude that generating accurate IMU data from motion capture datasets is possible and could be useful in providing larger volumes of data for activity recognition tasks and in helping enable advanced, data-hungry techniques such as deep learning to be employed on a larger scale within the domain of human activity recognition.

Inertial Measurement Units (IMUs) are widely utilized in shoulder rehabilitation due to their portability and cost-effectiveness, but their reliance on spatial motion data restricts their use in comprehensive musculoskeletal analyses. To overcome this limitation, we propose SWCTNet (Sliding Window CNN + Channel-Time Attention Transformer Network), an advanced neural network specifically tailored for multichannel temporal tasks. SWCTNet integrates IMU and surface electromyography (sEMG) data through sliding window convolution and channel-time attention mechanisms, enabling the efficient extraction of temporal features. This model enables the prediction of muscle activation patterns and kinematics using exclusively IMU data. The experimental results demonstrate that the SWCTNet model achieves recognition accuracies ranging from 87.93% to 91.03% on public temporal datasets and an impressive 98% on self-collected datasets. Additionally, SWCTNet exhibits remarkable precision and stability in generative tasks: the normalized DTW distance was 0.12 for the normal group and 0.25 for the patient group when using the self-collected dataset. This study positions SWCTNet as an advanced tool for extracting musculoskeletal features from IMU data, paving the way for innovative applications in real-time monitoring and personalized rehabilitation at home. This approach demonstrates significant potential for long-term musculoskeletal function monitoring in non-clinical or home settings, advancing the capabilities of IMU-based wearable devices.

Autonomous driving relies on reliable localization, but the Global Navigation Satellite System (GNSS) often loses accuracy or becomes unavailable in harsh environments, such as tunnels and dense urban canyons. Visual-Inertial Simultaneous Localization and Mapping (VI-SLAM), which fuses camera and Inertial Measurement Unit (IMU) data, is a key alternative in GNSS-denied scenarios. However most VI-SLAM methods overlook vehicles' inherent periodic motion features, such as zero velocity during stops or gear shifts, and linear motion during cruising. To address this, this paper proposes a visual-inertial localization method integrating these features into a tightly coupled VI-SLAM framework. First, based on IMU data, the Generalized Likelihood Ratio Test is used to detect zero velocity and linear motion. Then, Zero Velocity Update (ZUPT) residuals and Nonholonomic Constraint (NHC) residuals are constructed. These two residuals are jointly optimized with visual residuals, IMU pre-integration residuals, and marginalization residuals within a sliding-window nonlinear backend for Visual-Inertial Odometry. Experiments on self-collected datasets show that compared to the baseline algorithm, both the ZUPT-integrated algorithm and the NHC-integrated algorithm yield higher accuracy, lower error variance, and stronger robustness.

IntroductionDigital home rehabilitation systems require accurate segmentation methods to provide appropriate feedback on repetition counting and exercise technique. Current segmentation methods are not suitable for clinical use; they are not highly accurate or require multiple sensors, which creates usability problems. We propose a model for accurately segmenting inertial measurement unit data for shoulder rehabilitation exercises. This study aims to use inertial measurement unit data to train and test a machine learning segmentation model for single- and multiple-inertial measurement unit systems and to identify the optimal single-sensor location.MethodsA focus group of specialist physiotherapists selected the exercises, which were performed by participants wearing inertial measurement units on the wrist, arm and scapula. We applied a novel machine learning based segmentation technique involving a convolutional classifier and Finite State Machine to the inertial measurement unit data. An accuracy score was calculated for each possible single- or multiple-sensor system.ResultsThe wrist inertial measurement unit was chosen as the optimal single-sensor location for future system development (mean overall accuracy 0.871). Flexion and abduction based exercises mostly could be segmented with high accuracy, but scapular movement exercises had poor accuracy.ConclusionA wrist-worn single inertial measurement unit system can accurately segment shoulder exercise repetitions; however, accuracy varies depending on characteristics of the exercise.

Assessment of varus thrust using inertial measurement units.

Clinicians lack objective means for monitoring if their knee osteoarthritis patients are improving outside of the clinic (e.g., at home). Previous human activity recognition (HAR) models using wearable sensor data have only used data from healthy people and such models are typically imprecise for people who have medical conditions affecting movement. HAR models designed for people with knee osteoarthritis have classified rehabilitation exercises but not the clinically relevant activities of transitioning from a chair, negotiating stairs and walking, which are commonly monitored for improvement during therapy for this condition. Therefore, it is unknown if a HAR model trained on data from people who have knee osteoarthritis can be accurate in classifying these three clinically relevant activities. Therefore, we collected inertial measurement unit (IMU) data from 18 participants with knee osteoarthritis and trained convolutional neural network models to identify chair, stairs and walking activities, and phases. The model accuracy was 85% at the first level of classification (activity), 89–97% at the second (direction of movement) and 60–67% at the third level (phase). This study is the first proof-of-concept that an accurate HAR system can be developed using IMU data from people with knee osteoarthritis to classify activities and phases of activities.

Objective: Older adults’ falls are a critical public health problem. The majority of free-living fall risk assessment methods have investigated fall predictive power of step-related digital biomarkers extracted from wearable inertial measurement unit (IMU) data. Alternatively, the examination of characteristics and frequency of naturally-occurring compensatory balance reactions (CBRs) may provide valuable information on older adults’ propensity for falls. To address this, models to automatically detect naturally-occurring CBRs are needed. However, compared to steps, CBRs are rare events. Therefore, prolonged collection of criterion standard data (along with IMU data) is required to validate model’s performance in free-living conditions. Methods: By investigating 11 fallers’ and older non-fallers’ free-living criterion standard data, 8 naturally-occurring CBRs, i.e., 7 trips (self-reported using a wrist-mounted voice-recorder) and 1 hit/bump (verified using egocentric vision data) were localized in the corresponding trunk-mounted IMU data. Random forest models were trained on independent/unseen datasets curated from multiple sources, including in-lab data captured using a perturbation treadmill. Subsequently, the models’ translation/generalization to older adults’ out-of-lab data were assessed. Results: A subset of models differentiated between naturally-occurring CBRs and free-living activities with high sensitivity (100%) and specificity (≥99%). Conclusions: The findings suggest that accurate detection of naturally-occurring CBRs is feasible. Clinical/Translational Impact- As a multi-institutional validation study to detect older adults’ naturally-occurring CBRs, suitability for larger-scale free-living studies to investigate falls etiology, and/or assess the effectiveness of perturbation training programs is discussed.

Freezing-of-gait (FOG) and impaired walking are common features of Parkinson’s disease (PD). Provision of external stimuli (cueing) can improve gait, however, many cueing methods are simplistic, increase task loading or have limited utility in a real-world setting. Closed-loop (automated) somatosensory cueing systems have the potential to deliver personalised, discrete cues at the appropriate time, without requiring user input. Further development of cue delivery methods and FOG-detection are required to achieve this. In this feasibility study, we aimed to test if FOG-initiated vibration cues applied to the lower-leg via wearable devices can improve gait in PD, and to develop real-time FOG-detection algorithms. 17 participants with Parkinson’s disease and daily FOG were recruited. During 1 h study sessions, participants undertook 4 complex walking circuits, each with a different intervention: continuous rhythmic vibration cueing (CC), responsive cueing (RC; cues initiated by the research team in response to FOG), device worn with no cueing (NC), or no device (ND). Study sessions were grouped into 3 stages/blocks (A-C), separated by a gap of several weeks, enabling improvements to circuit design and the cueing device to be implemented. Video and onboard inertial measurement unit (IMU) data were analyzed for FOG events and gait metrics. RC significantly improved circuit completion times demonstrating improved overall performance across a range of walking activities. Step frequency was significantly enhanced by RC during stages B and C. During stage C, > 10 FOG events were recorded in 45% of participants without cueing (NC), which was significantly reduced by RC. A machine learning framework achieved 83% sensitivity and 80% specificity for FOG detection using IMU data. Together, these data support the feasibility of closed-loop cueing approaches coupling real-time FOG detection with responsive somatosensory lower-leg cueing to improve gait in PD.

Abstract. Kinematic laser scanning enables efficient and accurate acquisition of 3D data, specifically point clouds. Transforming the laser ranging measurements from the scanner coordinate system to a georeferenced coordinate system requires integration with auxiliary navigation systems, in this case using GNSS (global navigation satellite system) and an IMU (inertial measurement unit). GNSS and IMU data are commonly fused via Kalman filter, and the resulting trajectory is used for georeferencing. Errors in the trajectory propagate through the georeferencing and cause discrepancies in the point cloud. To mitigate this, we present a holistic integration method incorporating GNSS, IMU and LiDAR measurements in a single adjustment, which can be seamlessly adapted to a setup with two GNSS receivers and antennas. This tight coupling of LiDAR and IMU together with the dual-GNSS setup allows the use of an ultra low-cost IMU while still achieving high-quality point clouds. We demonstrate this methodology on two datsets, where we discuss in particular the boresight calibration of such a system, and the impact of the LiDAR measurements and the dual-GNSS set-up on the trajectory and the point cloud.

Attitude and gyro bias estimation for a VTOL UAV

Gait rehabilitation commonly relies on bodyweight unloading mechanisms, such as overhead mechanical support and underwater buoyancy. Lightweight and wireless inertial measurement unit (IMU) sensors provide a cost-effective tool for quantifying body segment motions without the need for video recordings or ground reaction force measures. Identifying the instant when the foot contacts and leaves the ground from IMU data can be challenging, often requiring scrupulous parameter selection and researcher supervision. We aimed to assess the use of machine learning methods for gait event detection based on features from foot segment rotational velocity using foot-worn IMU sensors during bodyweight-supported treadmill walking on land and underwater. Twelve healthy subjects completed on-land treadmill walking with overhead mechanical bodyweight support, and three subjects completed underwater treadmill walking. We placed IMU sensors on the foot and recorded motion capture and ground reaction force data on land and recorded IMU sensor data from wireless foot pressure insoles underwater. To detect gait events based on IMU data features, we used random forest machine learning classification. We achieved high gait event detection accuracy (95–96%) during on-land bodyweight-supported treadmill walking across a range of gait speeds and bodyweight support levels. Due to biomechanical changes during underwater treadmill walking compared to on land, accurate underwater gait event detection required specific underwater training data. Using single-axis IMU data and machine learning classification, we were able to effectively identify gait events during bodyweight-supported treadmill walking on land and underwater. Robust and automated gait event detection methods can enable advances in gait rehabilitation.

Artificial intelligence (AI) has already revolutionized the analysis of image, text, and tabular data, bringing significant advances across many medical sectors. Now, by combining with wearable inertial measurement units (IMUs), AI could transform health care again by opening new opportunities in patient care and medical research. This systematic review aims to evaluate the integration of AI models with wearable IMUs in health care, identifying current applications, challenges, and future opportunities. The focus will be on the types of models used, the characteristics of the datasets, and the potential for expanding and enhancing the use of this technology to improve patient care and advance medical research. This study examines this synergy of AI models and IMU data by using a systematic methodology, following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, to explore 3 core questions: (1) Which medical fields are most actively researching AI and IMU data? (2) Which models are being used in the analysis of IMU data within these medical fields? (3) What are the characteristics of the datasets used for in this fields? The median dataset size is of 50 participants, which poses significant limitations for AI models given their dependency on large datasets for effective training and generalization. Furthermore, our analysis reveals the current dominance of machine learning models in 76% on the surveyed studies, suggesting a preference for traditional models like linear regression, support vector machine, and random forest, but also indicating significant growth potential for deep learning models in this area. Impressively, 93% of the studies used supervised learning, revealing an underuse of unsupervised learning, and indicating an important area for future exploration on discovering hidden patterns and insights without predefined labels or outcomes. In addition, there was a preference for conducting studies in clinical settings (77%), rather than in real-life scenarios, a choice that, along with the underapplication of the full potential of wearable IMUs, is recognized as a limitation in terms of practical applicability. Furthermore, the focus of 65% of the studies on neurological issues suggests an opportunity to broaden research scope to other clinical areas such as musculoskeletal applications, where AI could have significant impacts. In conclusion, the review calls for a collaborative effort to address the highlighted challenges, including improvements in data collection, increasing dataset sizes, a move that inherently pushes the field toward the adoption of more complex deep learning models, and the expansion of the application of AI models on IMU data methodologies across various medical fields. This approach aims to enhance the reliability, generalizability, and clinical applicability of research findings, ultimately improving patient outcomes and advancing medical research.

Human activity recognition is a very active research on pervasive computing and mobile health application. Many human activity systems based on inertial measurement unit (IMU) sensor data were proposed in the past few years. These systems mainly use IMU sensor placed on he torso and limbs to collect data and utilize supervised machine learning algorithms on sensor data. One main issue of these systems is that wearing multiple on-body IMU sensors may bring inconvenience to users' daily life. The other issue of these exiting methods is that an activity recognition model that is trained on a specific subject does not work well when being applied to predict another subject's activities since IMU activity data always carry information that is specific to the human subject who conducts the activities. In our work, inspired by the principle of domain adaption, we proposed a new deep-learning activity recognition model based on an adversarial network which can remove the subject-specific information within the IMU activity data and extract subject-independent features shared by the data collected on different subjects. We also for the first time use data collected from insole based IMU sensors on 8 participants for 5 common activities to build a new real world human activity dataset which can minimize the inconvenience for users to wear. We conducted experiments with our new real-world dataset. Results show that our subject independent activity recognition model outperforms state-of-art supervised learning techniques and eliminates the effects of individual differences between subjects successfully. The average recognition accuracy under the leave-one-out (L1O) condition achieves 99.0% which is higher than the performance of traditional human activity recognition system based on CNNs.