A Reinforcement Learning Method Using Multifunctional Principal Component Analysis for Human-Like Grasping

Cardiovascular diseases (CVDs) are one of the primary reasons for death worldwide. These diseases often occur due to the occlusion of coronary arteries, thereby leading to insufficient blood and oxygen supply that damages cardiac muscle cells. Electrocardiogram (ECG) signals which reflect heart electrical activity are being used for diagnosing various cardiac diseases. Typically, a standard ECG consists of 12 channels referred to as leads which enable practitioners to monitor heartbeats through different channels where each heartbeat lasts approximately 600 ms. The majority of studies focus on the classification and early diagnosis of arrhythmias. Although the current studies on change-point methods have acquired massive accuracy in detecting potential shifts during a multi-channel process, they lack flexibility in manually assigning more weights to the channels, which are of more importance for experts. This could be addressed by implementing the weighted multivariate functional principal component analysis (WMFPCA). The objective of this study is to develop a novel change-point detection method to monitor long-term cardiovascular treatment. A third-order tensor structure was employed to represent the 12-lead ECG data in three dimensions (beats × samples × leads). Exploiting intra-beat, inter-beat, and inter-lead correlations along with channel significance in the third-order tensor, the WMFPCA is incorporated into Hotelling's statistic to construct monitoring schemes. Simulation results show that the proposed approach outperforms the existing methods in monitoring multi-channel processes. Finally, applying the suggested model on a real-world dataset containing Myocardial Infarction (MI) subjects verifies the model.

We address the problem of performing dimension reduction on multivariate functional data observed on different domains in an endogenously stratified sampling context. The aim is to propose a new multivariate functional principal component analysis \(MFPCA) approach for data sampled by a stratification of a population according to a binary variable of interest. This estimation strategy is derived from a direct relationship between univariate and multivariate FPCA for finite Karhunen-Loève decompositions. The proposed methodology yields encouraging results and can be applied to data with measurement errors. Computational results on simulated data highlight the good performance of the proposed methodology compared to the classical MFPCA, which ignores the type of data sampling. A real-life application considering breast cancer cells data is also presented.

The paper entitled \(\textit{Multivariate Functional Principal Component Analysis for Endogenously Stratified Data}\) has been published in the journal Afrika Statistika, Vol. 17 (4), 2023, pp. 3321–3366 (DOI: 10.16929/as/2022.3321.308). The authors have identified the need for substantial modifications, including additions, deletions, and revisions. This erratum is published to address these necessary corrections and updates.

Modern metro systems in big cities have accumulated large amounts of passenger transit transaction data via the deployment of automatic fare collection (AFC) devices, which could facilitate a thorough analysis of passenger flow dynamics. To discover the underlying passenger flow patterns over all stations in an entire metro system, this paper proposes a multivariate functional principal component analysis (MFPCA) method. The functional integrity of a single daily passenger flow profile at each station is explicitly utilized, and the complex correlation among a multitude of daily passenger flow profiles from all stations is comprehensively modeled. Moreover, to simultaneously improve the interpretability of eigenvectors and mitigate the sensitivity to outliers, the MFPCA is formulated as a minimization problem with both a sparsity and a robustness penalty term. A computationally efficient algorithm is developed accordingly to obtain the eigenvectors. The superiority of our proposed sparse and robust MFPCA (SRMFPCA) is validated using a Hong Kong Mass Transit Railway (MTR) dataset. The derived sparse and smooth eigenvectors can be well interpreted as empirically meaningful passenger flow patterns. The results of our method can be further used as solid foundations for station clustering, correlation analysis and outlier identification.

The standard paradigm in Affective Computing involves acquiring one/several markers (e.g., physiological signals) of emotions and training models on these to predict emotions. However, due to the internal nature of emotions, labelling/annotation of emotional experience is done manually by humans using specially developed annotation tools. To effectively exploit the resulting subjective annotations for developing affective systems, their quality needs to be assessed. This entails, (i) evaluating the variations in annotations, across different subjects and emotional stimuli, to detect spurious/unexpected patterns; and (ii) developing strategies to effectively combine these subjective annotations into a ground truth annotation. This article builds on our previous work by presenting a novel Functional Data Analysis based approach to assess the quality of annotations. Specifically, the bivariate annotation time-series are transformed into functions, such that each resulting functional annotation then becomes a sample element for analysis like Multivariate Functional Principal Component Analysis (MFPCA) that evaluate variation across all annotations. The resulting scores from MFPCA provide interesting insights into annotation patterns and facilitate the use of multivariate statistical techniques to address both (i) and (ii). Given the presented efficacy of these methods, we believe they offer an exciting new approach to assessing the quality of annotations.

Practical Implementation Method of Reinforcement Learning for Power Converter

P2P power trading based on reinforcement learning for nanogrid clusters

SummaryThe transitional phase of menopause induces significant hormonal fluctuations, exerting a profound influence on the long-term well-being of women. In an extensive longitudinal investigation of women’s health during mid-life and beyond, known as the Study of Women’s Health Across the Nation (SWAN), hormonal biomarkers are repeatedly assessed, following an asynchronous schedule compared to other error-prone covariates, such as physical and cardiovascular measurements. We conduct a subgroup analysis of the SWAN data employing a semiparametric mixture regression model, which allows us to explore how the relationship between hormonal responses and other time-varying or time-invariant covariates varies across subgroups. To address the challenges posed by asynchronous scheduling and measurement errors, we model the time-varying covariate trajectories as functional data with reduced-rank Karhunen–Loéve expansions, where splines are employed to capture the mean and eigenfunctions. Treating the latent subgroup membership and the functional principal component (FPC) scores as missing data, we propose an Expectation-Maximization algorithm to effectively fit the joint model, combining the mixture regression for the hormonal response and the FPC model for the asynchronous, time-varying covariates. In addition, we explore data-driven methods to determine the optimal number of subgroups within the population. Through our comprehensive analysis of the SWAN data, we unveil a crucial subgroup structure within the aging female population, shedding light on important distinctions and patterns among women undergoing menopause.

In this paper, we propose a generalized functional linear regression model for a binary outcome indicating the presence/absence of a cardiac disease with multivariate functional data among the relevant predictors. In particular, the motivating aim is the analysis of electrocardiographic traces of patients whose pre-hospital electrocardiogram (ECG) has been sent to 118 Dispatch Center of Milan (the Italian free-toll number for emergencies) by life support personnel of the basic rescue units. The statistical analysis starts with a preprocessing of ECGs treated as multivariate functional data. The signals are reconstructed from noisy observations. The biological variability is then removed by a nonlinear registration procedure based on landmarks. Thus, in order to perform a data-driven dimensional reduction, a multivariate functional principal component analysis is carried out on the variance-covariance matrix of the reconstructed and registered ECGs and their first derivatives. We use the scores of the Principal Components decomposition as covariates in a generalized linear model to predict the presence of the disease in a new patient. Hence, a new semi-automatic diagnostic procedure is proposed to estimate the risk of infarction (in the case of interest, the probability of being affected by Left Bundle Brunch Block). The performance of this classification method is evaluated and compared with other methods proposed in literature. Finally, the robustness of the procedure is checked via leave-j-out techniques.

With the emergence of numerical sensors in many aspects of every- day life, there is an increasing need in analyzing multivariate functional data. This work focuses on the clustering of such functional data, in order to ease their modeling and understanding. To this end, a novel clustering technique for multivariate functional data is presented. This method is based on a func- tional latent mixture model which fits the data in group-specific functional subspaces through a multivariate functional principal component analysis. A family of parsimonious models is obtained by constraining model parameters within and between groups. An EM algorithm is proposed for model inference and the choice of hyper-parameters is addressed through model selection. Nu- merical experiments on simulated datasets highlight the good performance of the proposed methodology compared to existing works. This algorithm is then applied to the analysis of the pollution in French cities for one year.

Methods and tools for implementing regularized multivariate functional principal component analysis ('ReMFPCA') for multivariate functional data whose variables might be observed over different dimensional domains. 'ReMFPCA' is an object-oriented interface leveraging the extensibility and scalability of R6. It employs a parameter vector to control the smoothness of each functional variable. By incorporating smoothness constraints as penalty terms within a regularized optimization framework, 'ReMFPCA' generates smooth multivariate functional principal components, offering a concise and interpretable representation of the data. For detailed information on the methods and techniques used in 'ReMFPCA', please refer to Haghbin et al. (2023) &lt;<a href="https://doi.org/10.48550%2FarXiv.2306.13980" target="_top">doi:10.48550/arXiv.2306.13980</a>&gt;.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

The widespread deployment of smart meters and communication technologies brings opportunities to improve the adaptability and flexibility of future manufacturing systems under growing complexity and uncertainty. As a result, data-driven approaches, especially Reinforcement Learning (RL) methods, are gaining wide attention in recent years. Although in the current literature, RL approaches show superiority over traditional methods in many applications, a comprehensive review and comparison of different RL methods and their use in discrete manufacturing system control under changing environments have not yet been established. This paper first provides a literature review of RL algorithms for decision-making in discrete manufacturing systems, and then systematically discusses the underlying mechanisms of four most commonly used RL algorithms. In addition, the performance of these RL algorithms is compared by solving a production control problem, which aims to maximize the profit under time-varying production costs. The comparison results show that the single-agent RL methods with discrete action space can provide satisfactory results in small-scale systems, while multi-agent RL algorithms are more suitable to solve problems in complex, large-scale systems to remit the curse of dimensionality. This research sheds light on the impacts of RL method selection on the search for high-quality solutions in manufacturing systems, which is envisioned to drive future research on RL-supported manufacturing systems.

Recent advancement on reinforcement learning (RL) algorithms shows that effective learning of parametric action- selection policies can often be achieved through direct opti- mization of a performance lower bound subject to pre-defined policy behavioral constraints. Driven by this understanding, this paper seeks to develop new policy search techniques where RL is achieved through maximizing a performance lower bound ob- tained originally based on an Expectation-Maximization method. For reliable RL, our new learning techniques must also simul- taneously guarantee constrained policy behavioral changes mea- sured through KL divergence. Two separate approaches will be pursued to tackle our constrained policy optimization problems, resulting in two new RL algorithms. The first algorithm utilizes a conjugate gradient technique and a Bayesian learning method for approximate optimization. The second algorithm focuses on minimizing a loss function derived from solving the Lagrangian for constrained policy search. Both algorithms have been experi- mentally examined on several benchmark problems provided by OpenAI GYM. The experiment results clearly demonstrate that our algorithms can be highly effective in comparison to several well-known RL algorithms.

This work analyses the performance of Reinforcement Learning (RL) versus Swarm Intelligence (SI) for coordinating multiple unmanned High Altitude Platform Stations (HAPS) for communications area coverage. It builds upon previous work which looked at various elements of both algorithms. The main aim of this paper is to address the continuous state-space challenge within this work by using partitioning to manage the high dimensionality problem. This enabled comparing the performance of the classical cases of both RL and SI establishing a baseline for future comparisons of improved versions. From previous work, SI was observed to perform better across various key performance indicators. However, after tuning parameters and empirically choosing suitable partitioning ratio for the RL state space, it was observed that the SI algorithm still maintained superior coordination capability by achieving higher mean overall user coverage (about 20% better than the RL algorithm), in addition to faster convergence rates. Though the RL technique showed better average peak user coverage, the unpredictable coverage dip was a key weakness, making SI a more suitable algorithm within the context of this work.