A feasibility study of a video-based heart rate estimation method with convolutional neural networks

Remote photoplethysmography (rPPG) is a kind of noncontact technique to measure heart rate (HR) from facial videos. As the demand for long-term health monitoring grows, rPPG attracts much attention from researchers. However, the performance of conventional rPPG methods is easily degenerated due to noise interference. Recently, some deep learning-based rPPG methods have been introduced and they revealed good performance against noise. In this article, we propose a new rPPG method with convolutional neural networks (CNNs) to build a mapping between a spatiotemporal HR feature image to its corresponding HR value. The feature map is constructed in a time-delayed way with noise-contaminated pulse signals extracted from existing rPPG methods. The CNN model is trained using transfer learning where images built from synthetic rPPG signals are taken to train the model first in order to generate initials for the practical one. The synthetic rPPG signals are interpolated from blood volume pulses or electrocardiograms through a modified Akima cubic Hermite interpolation. The proposed method is tested in both within-database and cross-database configurations on public databases. The results demonstrate that our method achieves overall the best performance compared to some other typical rPPG methods. The mean absolute error reaches 5.98 beats per minute and the mean error rate percentage is 7.97% in the cross-database testing on MAHNOB-HCI data set. Besides, some key factors that affect the performance of our method are also discussed which indicates potential ways for further improvements.

Imaging photoplethysmography (iPPG) technique has a bright prospect in noncontact heart rate (HR) and other types of vital signs detection; however, the iPPG method is highly susceptible to the motion artifacts in blood volume pulse (BVP) detected from facial video, and this is crucial for HR estimation. Although a number of signal processing algorithms regarding the resistance to motion interference have been proposed in recent years, their stability and computational complexity are hard to meet the requirements of practical applications. To address this problem, we suggest using the head-mounted camera (HMC) instead of a fixed position camera (FPC) in iPPG method because the HMC can move synchronously with the head, which can effectively alleviate the impact of motion artifacts. To this end, a prototype of HMC was designed and implemented. By referring to the results of CMS50E pulse oximeter synchronized with the subject's videos, the comparison study of HMC and FPC in iPPG-based HR estimation was conducted in three scenarios of resting, talking and head rotation. Furthermore, a HMC-based iPPG database named IPPG-HMC database was built. The experimental results show that HMC-based iPPG methods can deal with motion artifacts efficiently, and significantly improve the reliability and accuracy of HR estimation.

Blood volume pulse (BVP) measurements obtained through the noninvasive photoplethysmography technique are often analyzed to derive physiological variables such as heart rate (HR) and HR variability, blood pressure, and respiratory rate. Physical activity is one of the predominant factors inducing changes in the BVP signal. The goal of the present study is to develop a mathematical model to characterize the exercise-induced changes in the BVP signal. A two-step procedure is developed that first models the HR using specified exercise intensity information, and then characterizes the change in the BVP waveform based on the estimated HR. The HR model parameters are identified using experiments involving 18 subjects participating in bouts of running and cycling exercise. Speed and incline for the treadmill and cycling power for the stationary bike are inputs to the HR model. The mean (standard deviation) for the root-mean-square error of the HR predictions are 16.1 (4.0) and 12.47 (4.6) BPM during running and cycling, respectively. The HR is transformed to the BVP signal through a model developed based on log-normal functions where model parameters change due to HR variability.

The current definition of sinus tachycardia is a heart rate >100 beats per minute (bpm).1 This limit was set arbitrarily when heart rate was not yet regarded as a risk factor for cardiovascular disease, probably with the main purpose of distinguishing between a disease state (fever, thyrotoxicosis, anemia, congestive heart failure, etc) and a normal condition. In recent years, interest has been aroused by the awareness that fast heart rate is a potent precursor of hypertension, atherosclerosis, and their sequelae.2 3 4 5 6 7 8 9 10 In addition, many leading epidemiological studies have shown that tachycardia is associated with an increased risk of death from cardiovascular and noncardiovascular causes. This relationship has been found in general populations,3 4 5 6 7 in elderly individuals,9 and in hypertensive cohorts.10 In all of these studies, the heart rate value above that in which a significant increase in risk was observed was below the 100 bpm threshold (Table 1⇓). Only in the study by Levy et al3 was tachycardia defined as a heart rate >99 bpm, but in that study the cutoff between normal and high heart rate was chosen arbitrarily, and the highest heart rate value measured during the examination was taken to define the subject’s heart rate. In all the other studies, the threshold level between normal and fast heart rate was between 79 and 90 bpm. View this table: Table 1. Heart Rate Values Above Which a Significant Increase in Risk Was Found: Data From 9 Epidemiological Studies The normalcy limits of a clinical variable can be established according to different criteria. For many parameters, such as most biochemical indexes, the 95% confidence interval is calculated to identify the upper normal limit of the variable. This statistical approach does not appear suitable for those clinical variables …

Objective. Monitoring changes in human heart rate variability (HRV) holds significant importance for protecting life and health. Studies have shown that Imaging Photoplethysmography (IPPG) based on ordinary color cameras can detect the color change of the skin pixel caused by cardiopulmonary system. Most researchers employed deep learning IPPG algorithms to extract the blood volume pulse (BVP) signal, analyzing it predominantly through the heart rate (HR). However, this approach often overlooks the inherent intricate time-frequency domain characteristics in the BVP signal, which cannot be comprehensively deduced solely from HR. The analysis of HRV metrics through the BVP signal is imperative. Approach. In this paper, the transformation invariant loss function with distance equilibrium (TIDLE) loss function is applied to IPPG for the first time, and the details of BVP signal can be recovered better. In detail, TIDLE is tested in four commonly used IPPG deep learning models, which are DeepPhys, EfficientPhys, Physnet and TS_CAN, and compared with other three loss functions, which are mean absolute error (MAE), mean square error (MSE), Neg Pearson Coefficient correlation (NPCC). Main results. The experiments demonstrate that MAE and MSE exhibit suboptimal performance in predicting LF/HF across the four models, achieving the Statistic of Mean Absolute Error (MAES) of 25.94% and 34.05%, respectively. In contrast, NPCC and TIDLE yielded more favorable results at 13.51% and 11.35%, respectively. Taking into consideration the morphological characteristics of the BVP signal, on the two optimal models for predicting HRV metrics, namely DeepPhys and TS_CAN, the Pearson coefficients for the BVP signals predicted by TIDLE in comparison to the gold-standard BVP signals achieved values of 0.627 and 0.605, respectively. In contrast, the results based on NPCC were notably lower, at only 0.545 and 0.533, respectively. Significance. This paper contributes significantly to the effective restoration of the morphology and frequency domain characteristics of the BVP signal.

Photoplethysmography (PPG) is a noninvasively technique used to detect vital signs such as heart rate (HR), saturation of peripheral oxygen (SpO2) and blood pressure (BP). Recently, wearable finger-type PPG devices are increasingly developed toward convenience of person under monitoring (PUM). Two most critical features of wearable PPG devices are high accuracy and long operation time. To enhance these functions, this paper proposes a new architecture to select, process and transfer only high quality PPG signals. Hence, data quality is significantly improved and power consumption on wireless module is minimized. In the proposed architecture, parameters of PPG sensor are reconfigured in real-time being suitable with skin characteristics of PUM. Moreover, the adaptive LED current control algorithm is proposed to dynamically change the LED current to remove various motion artifacts (MA) and get high amplitude PPG signal. We also develop a heart rate (HR) estimation framework utilizing deep learning (DL) model based on convolutional neural network (CNN) and long short-term memory (LSTM) network. The combination of CNN-LSTM can extract features from both spatial and temporal correlation. A light-weight model with two CNN-LSTM layers is built to estimate HR with in only 5s. For validation, we conduct experiments with volunteers doing various physical exercises. The results show PPG signals with high amplitude and signal-to-noise ratio (SNR). HR estimation is more accurate even during irregular and muscle strength exercises. The proposed adaptive architecture and DL-based HR estimation can overcome MA and minimize the power consumption on wireless transfer module of wearable PPG devices.

Respiratory rate (RR) is a vital sign widely used to assess patient health. However, estimating RR using photoplethysmography (PPG) signals remains challenging due to noise and signal variability. This paper proposes RRWaveNet, an end-to-end model based on multi-scale convolutional networks with residual connections for robust RR estimation from PPG signals. The model is evaluated on four public datasets BIDMC, CapnoBase, WESAD, and SensAI with optimal performance observed at a 32-second window. RRWaveNet achieves mean absolute errors of 1.66±1.01, 1.59±1.08, 1.92±0.96, and 1.23±0.61 breaths per minute, respectively, outperforming state-of-the-art methods across all datasets and the transfer learning using ICU datasets, which reduces MAE by up to 21%, demonstrating that leveraging prior knowledge can improve model performance. Extending this analysis, we examine the effect of training strategies on model performance by contrasting task-specific training with a traditional pretraining approach on both RR and heart rate (HR) estimation tasks. For HR estimation, pretrained models achieve a 15\% improvement in MAE when trained on combined datasets, although performance degrades by up to 30% on individual datasets. For RR estimation, pretraining and task-specific training yield comparable performance, suggesting limited benefit from pretraining for this task. These findings demonstrate the practicality and robustness of deep learning-based RR and HR estimation on wearable devices, supporting deployment in real-world health monitoring applications.

This paper explores a novel method for anxiety detection in older adults using simple wristband sensors such as electrodermal activity (EDA) and photoplethysmogram (PPG) and a context-based feature. The proposed method for anxiety detection combines features from a single physiological signal with an experimental context-based feature to improve the performance of the anxiety detection model. The experimental data for this work are obtained from a year-long experiment on 41 healthy older adults (26 females and 15 males) in the age range 60–80 with mean age \(73.36 \pm 5.25\) during a Trier Social Stress Test (TSST) protocol. The anxiety level ground truth was obtained from State–Trait Anxiety Inventory (STAI), which is regarded as the gold standard to measure perceived anxiety. EDA and blood volume pulse (BVP) signals were recorded using a wrist-worn EDA and PPG sensor, respectively. 47 features were computed from EDA and BVP signal, out of which a final set of 24 significantly correlated features were selected for analysis. The phases of the experimental study are encoded as unique integers to generate the context feature vector. A combination of features from a single sensor with the context feature vector is used for training a machine learning model to distinguish between anxious and not-anxious states. Results and analysis showed that the EDA and BVP machine learning models that combined the context feature along with the physiological features achieved 3.37% and 6.41% higher accuracy, respectively, than the models that used only physiological features. Further, end-to-end processing of EDA and BVP signals was simulated for real-time anxiety level detection. This work demonstrates the practicality of the proposed anxiety detection method in facilitating long-term monitoring of anxiety in older adults using low-cost consumer devices.

BackgroundAlthough electrocardiography is the gold standard for heart rate (HR) recording in clinical trials, the increasing availability of smartwatch-based HR monitors opens up possibilities for drug development studies. Smartwatches allow for inexpensive, unobtrusive, and continuous HR estimation for potential detection of treatment effects outside the clinic, during daily life.ObjectiveThe aim of this study is to evaluate the repeatability and sensitivity of smartwatch-based HR estimates collected during a randomized clinical trial.MethodsThe data were collected as part of a multiple-dose, investigator-blinded, randomized, placebo-controlled, parallel-group study of 12 patients with Parkinson disease. After a 6-day baseline period, 4 and 8 patients were treated for 7 days with an ascending dose of placebo and clenbuterol, respectively. Throughout the study, the smartwatch provided HR and sleep state estimates. The HR estimates were quantified as the 2.5th, 50th, and 97.5th percentiles within awake and asleep segments. Linear mixed models were used to calculate the following: (1) the intraclass correlation coefficient (ICC) of estimated sleep durations, (2) the ICC and minimum detectable effect (MDE) of the HR estimates, and (3) the effect sizes of the HR estimates.ResultsSleep duration was moderately repeatable (ICC=0.64) and was not significantly affected by study day (P=.83), clenbuterol (P=.43), and study day by clenbuterol (P=.73). Clenbuterol-induced changes were detected in the asleep HR as of the first night (+3.79 beats per minute [bpm], P=.04) and in the awake HR as of the third day (+8.79 bpm, P=.001). The median HR while asleep had the highest repeatability (ICC=0.70). The MDE (N=12) was found to be smaller when patients were asleep (6.8 bpm to 11.7 bpm) than while awake (10.7 bpm to 22.1 bpm). Overall, the effect sizes for clenbuterol-induced changes were higher while asleep (0.49 to 2.75) than while awake (0.08 to 1.94).ConclusionsWe demonstrated the feasibility of using smartwatch-based HR estimates to detect clenbuterol-induced changes during clinical trials. The asleep HR estimates were most repeatable and sensitive to treatment effects. We conclude that smartwatch-based HR estimates obtained during daily living in a clinical trial can be used to detect and track treatment effects.Trial RegistrationNetherlands Trials Register NL8002; https://www.trialregister.nl/trial/8002

Non-contact video based heart rate estimation has great various potential in areas such as medical, health care, and affective computing. Recently, the methods using a digital camera for remote heart rate (HR) measurement are widely spread. However, the accuracy of these methods are not sufficient enough to calculate heart rate variability (HRV) accurately. If HRV are measured with high accuracy, we can visualize the sympathetic nerve system by calculating heart rate variability spectrogram (HRVS). In this paper, we propose a method of HR and HRV spectrogram estimation by analyzing extracted hemoglobin concentration from facial color images. Our method does not require any special camera. Furthermore, it is possible to have a stable presumption even under the ambient light sources. As a result, we could obtain 99% accuracy of HR and HRV spectrogram with the same accuracy electrocardiogram. Therefore we succeeded to identify whether participants were relaxed or stressed using conventional DSLR camera.

Video based heart rate estimation based on the PPG technique is a remote optical technique which allows us to determine the heart and breath rates through the intensity of motion variations. This paper proposes a new monitoring method for estimating the heart rate using a vision system with the VIS and SWIR camera.

Pulse rate variability (PRV) refers to the change in the interval between pulses in the blood volume pulse (BVP) signal acquired using photoplethysmography (PPG). PRV is an indicator of the health status of an individual’s autonomic nervous system. A representative method for measuring BVP is contact PPG (CPPG). CPPG may cause discomfort to a user, because the sensor is attached to the finger for measurements. In contrast, noncontact remote PPG (RPPG) extracts BVP signals from face data using a camera without the need for a sensor. However, because the existing RPPG is a technology that extracts a single pulse rate rather than a continuous BVP signal, it is difficult to extract additional health status indicators. Therefore, in this study, PRV analysis is performed using lab-based RPPG technology that can yield continuous BVP signals. In addition, we intended to confirm that the analysis of PRV via RPPG can be performed with the same quality as analysis via CPPG. The experimental results confirmed that the temporal and frequency parameters of PRV extracted from RPPG and CPPG were similar. In terms of correlation, the PRVs of RPPG and CPPG yielded correlation coefficients between 0.98 and 1.0.

Round the clock monitoring of key markers of health is a reason why healthcare, fitness, and remote patient care are relevant. However, traditional methods like Electrocardiography (ECG) and Photoplethysmography (PPG) are not only invasive but are also difficult to be used for long term monitoring because of their requirement of physical contact with the skin. In order to overcome this limitation, this paper presents – VitaSi, a new system that is non-contact and non-intrusive which estimates the heart rate (HR) and breathing rate (BR) in real time via facial video analysis. The system uses Pixel-wise Video Motion Processing (PVMP), a technology used to detect small facial movements associated with physiological functions, and a Convolutional Neural Network (CNN), which is able to precisely calculate HR and BR from the extracted signals. VitaSi is based on a standard webcam, and no wearable sensors are required; the interface of VitaSi is user friendly for the monitoring in real time, historical data visualization, annotations and health reminders. Experimental outcomes have shown that VitaSi provides a high level of accuracy in estimating HR, BR similar to the traditional methods while providing convenience and usability. The system is especially convenient for applications in telemedicine, fitness tracking, and home healthcare, that are a major step forward in contactless health monitoring technologies.

With the rapid development of science and technology, the living habits of people have also changed from those in the past; the diet, living environment, various life pressures, etc., all overwhelm the body and mind, meaning that, nowadays, more people are suffering from mental illness and cardiovascular disease year on year. Therefore, a non-contact measurement of heart rate and heart rate variability (HRV) is proposed to assist physicians in diagnosing symptoms related to mental illness and cardiovascular disease. In this paper, continuous images are obtained by general network cameras with non-contact, facial feature points and regions of interest (ROI) employed to track faces and regional images; HRV parameters were analyzed with the green wavelength of RGB color space. The artifact signal is eliminated by a hybrid algorithm of independent component analysis (ICA) and particle swarming optimization (PSO). Finally, the values of heart rate and HRV are obtained with signal processes of using band-pass filter, fast Fourier transform (FFT), and power spectrum analysis in the time and frequency domains, respectively. The non-contact measurement performance of the proposed method can effectively not only avoid infection doubts and obtain heart rate and HRV quickly, but also provide better physiological parameters, root mean square error (RMSE), and mean absolute percentage error (MAPE), than those of recent published papers.

Photoplethysmograph (PPG) is a simple and cost effective method to assess cardiovascular related parameters such as heart rate, arterial blood oxygen saturation and blood pressure. PPG signal consists of not only synchronized heart beats, but also the rhythms of respiration. The PPG sensor, which consists of infrared light-emitting diodes (LEDs) and a photodetector, allows a simple, reliable and low-cost means of monitoring the pulse rate. In this project, PPG signals are acquired through a customized data acquisition process using Arduino board to control the pulse circuit and to obtain the PPG signals from human subjects. Using signal processing techniques, including filters, peak detections, wavelet transform analysis and power spectral density, the heart rate (HR) and breathing rate (BR) are be obtained simultaneously. Estimations of HR and BR are conducted using MATLAB algorithm developed based on the wavelet decomposition techniques to extract the heart and respiration activities from the PPG signals. The values of HR and BR obtained using the algorithm are similar to the values obtained by manual estimation for seven sample subjects where the range of percentage errors are small about 0–9.5% for the breathing rate and 2.1–5.7% for the heart rate.