On the problems of sequential statistical inference for Wiener processes with delayed observations

In the past paper, we have formulated the structural Change Point Detection (CPD) problem in time series data as an optimal stopping one using the concept of DP (Dynamic Programming). And, we have shown the optimal solution theorem and its proof using reduction to absurdity. In this paper, we present the direct and constructive proof of the optimal solution theorem for the problem.

Early detection of fuel leakage at service stations with underground petroleum storage systems is a crucial task to prevent catastrophic hazards. Current data-driven fuel leakage detection methods employ offline statistical inventory reconciliation, leading to significant detection delays. Consequently, this can result in substantial financial loss and environmental impact on the surrounding community. In this paper, we propose a novel framework called memory-based online change point detection (MOCPD) which operates in near real-time, enabling early detection of fuel leakage. MOCPD maintains a collection of representative historical data within a size-constrained memory, along with an adaptively computed threshold. Leaks are detected when the dissimilarity between the latest data and historical memory exceeds the current threshold. An update phase is incorporated in MOCPD to ensure diversity among historical samples in the memory. With this design, MOCPD is more robust and achieves a better recall rate while maintaining a reasonable precision score. We have conducted a variety of experiments comparing MOCPD to commonly used online change point detection (CPD) baselines on real-world fuel variance data with induced leakages, actual fuel leakage data and benchmark CPD datasets. Overall, MOCPD consistently outperforms the baseline methods in terms of detection accuracy, demonstrating its applicability to fuel leakage detection and CPD problems.

The problem of sequential detection of spontaneous changes in equations for unobserved state variables and observations of linear and nonlinear multivariate state-space models is considered. We consider the case of new additive terms and the case of changing coefficients of these equations. The proposed nonparametric method uses the idea of the “moving window” of observations. In Theorem 4.1 the exponential rate of convergence to zero for type 1 error probability (a wrong decision about a change) is proved. In Theorem 4.2 we consider type 2 error probability and the normalized (by the volume N of the moving window) delay time in change-point detection. Asymptotic optimality of the proposed method is proved in Theorem 5.1. Experimental study includes Monte Carlo examples of sequentially detected changes in coefficients of linear and nonlinear state-space models.

Wikipedia has been turned into an immensely popular crowd-sourced encyclopedia for information dissemination on numerous versatile topics in the form of subscription free content. It allows anyone to contribute so that the articles remain comprehensive and updated. For enrichment of content without compromising standards, the Wikipedia community enumerates a detailed set of guidelines, which should be followed. Based on these, articles are categorized into several quality classes by the Wikipedia editors with increasing adherence to guidelines. This quality assessment task by editors is laborious as well as demands platform expertise. As a first objective, in this paper, we study evolution of a Wikipedia article with respect to such quality scales. Our results show novel non-intuitive patterns emerging from this exploration. As a second objective we attempt to develop an automated data driven approach for the detection of the early signals influencing the quality change of articles. We posit this as a change point detection problem whereby we represent an article as a time series of consecutive revisions and encode every revision by a set of intuitive features. Finally, various change point detection algorithms are used to efficiently and accurately detect the future change points. We also perform various ablation studies to understand which group of features are most effective in identifying the change points. To the best of our knowledge, this is the first work that rigorously explores English Wikipedia article quality life cycle from the perspective of quality indicators and provides a novel unsupervised page level approach to detect quality switch, which can help in automatic content monitoring in Wikipedia thus contributing significantly to the CSCW community.

This paper addresses the change point detection problem in time series. A methodology based on the Metropolis-Hastings algorithm applied to time series modeled as a process with Beta distribution is discussed. In order to make this methodology useful in practice, a fuzzy cluster technique is applied to the initial time series at first, generating a new data set with Beta distribution. Bayesian procedures are considered for inference and the Metropolis-Hastings algorithm is used to sample from the posteriors. In the clustering process, a Kohonen neural network is used having as objective to find the best centers of the time series to be used in the fuzzyfication process. Finally, it will be presented a simulation results in the series of the electric energy consumption in Brazil, between January of 1976 and December of 2000, five months before the blackout occurred in 2001. Such result illustrates the efficiency of the proposed methodology for change point detection in time series.

Change-point detection has a long history of research in statistical signal processing and remains a fundamental problem in many real-world applications involving information extraction from streaming data. Exploiting low-dimensional structures (subspace in particular) of high-dimensional signals is another research topic of significance, as it improves not only efficiency in computation, storage, and communication, but also interpretability and understanding of data. In this paper, we formulate the problem of subspace change-point detection, where a stream of high-dimensional data points lie on a low-dimensional subspace that abruptly changes at a certain time instant. Then, we propose a cumulative-sum-based algorithm for the task of detection in this problem setup. From the algorithmic perspective, the proposed method is computationally efficient, as it proceeds in an online and recursive manner, and each iteration involves simple matrix computation; it is also practical and flexible, as it works under mild conditions on the distribution of data points after change point. From the theoretical perspective, we derive rigorous performance bounds of the proposed algorithm in terms of false alarm rate and detection delay. We also conduct extensive numerical experiments on both synthetic and real-world data to validate effectiveness of our algorithm and correctness of theoretical analysis.

As video content is responsible for more than 70% of the global IP traffic, related resource allocation approaches, e.g., using content caching, become increasingly important. In this context, to avoid under-provisioning, it is important to rapidly detect and respond to changes in content popularity dynamics, including volatility, i.e., changes in the second order moment of the underlying process. In this paper, we focus on the early identification of changes in the variance of video content popularity, which we address as a statistical change point (CP) detection problem. Unlike changes in the mean that can be well captured by non-parametric statistical approaches, to address this more demanding problem, we construct a hypothesis test that uses in the test statistic both parametric and non-parametric approaches. In the context of parametric models, we consider linear, in the form of autoregressive moving average (ARMA), and, nonlinear, in the form of generalized autoregressive conditional heteroskedasticity (GARCH) processes. We propose an integrated algorithm that combines off-line and on-line CP schemes, with the off-line scheme used as a training (learning) phase. The algorithm is first assessed over synthetic data; our analysis demonstrates that non parametric and GARCH model based approaches can better generalize and are better suited for content views time series with unknown statistics. Finally, the non-parametric and the GARCH based variations of our proposed integrated algorithm are applied on real YouTube video content views time series, to illustrate the performance of the proposed approach of volatility change detection.

What mathematicians, scientists, engineers, and statisticians mean by “inverse problem” differs. For a statistician, an inverse problem is an inference or estimation problem. The data are finite in number and contain errors, as they do in classical estimation or inference problems, and the unknown typically is infinite-dimensional, as it is in nonparametric regression. The additional complication in an inverse problem is that the data are only indirectly related to the unknown. Standard statistical concepts, questions, and considerations such as bias, variance, mean-squared error, identifiability, consistency, efficiency, and various forms of optimality apply to inverse problems. This article discusses inverse problems as statistical estimation and inference problems, and points to the literature for a variety of techniques and results.

The problem of change point detection in multiparametric technological processes having a normal distribution and consisting in a shift from a given value of the sample mean and sample variance is investigated. Various types of control charts are considered, which make it possible to effectively detect simultaneous changes in the mean value and variance in multiparametric technological processes. By the method of statistical modeling, an analysis of the comparative effectiveness of control charts is carried out, practical recommendations are given.

In statistics and engineering the problem of change point detection in dynamical systems can be addressed using the class of sequential detection rules called limited generalized likelihood ratio (GLR) schemes. In this paper, some open problems concerning the statistical theory and practical implementation of these procedures are discussed. A method to select the window size and the threshold value, based on the joint use of Monte Carlo simulation and theoretical results, is proposed and illustrated through an example.

ABSTRACTMost studies in real-time change-point detection either focus on the linear model or use the cumulative sum (CUSUM) method under classical assumptions on model errors. This article considers the sequential change-point detection in a nonlinear quantile model. A test statistic based on the CUSUM of the quantile process subgradient is proposed and studied. Under the null hypothesis that the model does not change, the asymptotic distribution of the test statistic is determined. Under the alternative hypothesis that at some unknown observation there is a change in the model, the proposed test statistic converges in probability to ∞. These results allow building the critical regions on open-end and on closed-end procedures. Simulation results, using a Monte Carlo technique, investigate the performance of the test statistic, especially for heavy-tailed error distributions. We also compare it with the classical CUSUM test statistic. An example on real data is also given.

This paper presents an automatic extraction approach of worm signatures based on behavioral footprint analysis. Firstly, the suspicious worm traffic is detected based on worm behavior and is an instance of the Sequential Change Point Detection. To make the detection mechanism insensitive to site and access pattern, a non-parametric Cumulative Sum (CUSUM) method is applied, thus making the detection mechanism much more generally applicable and its deployment much easier. Secondly, worm behavioral footprint is defined and classified. The chronicle formalism is applied to correlate the footprint in suspicious worm traffic. Finally, worm signatures are extracted and ascertained by the evaluation function. The experiment shows that the approach can extract worm signatures effectively and accurately. There is a conclusion that worm behavioral footprint can't identify the worm accurately, but it helps to locate worm signatures, so signatures can be extracted effectively.

It is very often the case that at some moment a time series process abruptly changes its underlying structure and, therefore, it is very important to accurately detect such change-points. In this problem, which is called a change-point (or break-point) detection problem, we need to find a method that divides the original nonstationary time series into a piecewise stationary segments. In this paper, we develop a flexible method to estimate the unknown number and the locations of change-points in autoregressive time series. In order to find the optimal value of a performance function, which is based on the Minimum Description Length principle, we develop a Cross-Entropy algorithm for the combinatorial optimization problem. Our numerical experiments show that the proposed approach is very efficient in detecting multiple change-points when the underlying process has moderate to substantial variations in the mean and the autocorrelation coefficient. We also apply the proposed method to real data of daily AUD/CNY exchange rate series from 2 January 2018 to 24 March 2020.

Many statistical inference problems correspond to recovering the values of a set of hidden variables from sparse observations on them. For instance, in a planted constraint satisfaction problem such as planted 3-SAT, the clauses are sparse observations from which the hidden assignment is to be recovered. In the problem of community detection in a stochastic block model, the community labels are hidden variables that are to be recovered from the edges of the graph. Inspired by ideas from statistical physics, the presence of a stable fixed point for belief propogation has been widely conjectured to characterize the computational tractability of these problems. For community detection in stochastic block models, many of these predictions have been rigorously confirmed. In this work, we consider a general model of statistical inference problems that includes both community detection in stochastic block models, and all planted constraint satisfaction problems as special cases. We carry out the cavity method calculations from statistical physics to compute the regime of parameters where detection and recovery should be algorithmically tractable. At precisely the predicted tractable regime, we give: (i) a general polynomial-time algorithm for the problem of detection: distinguishing an input with a planted signal from one without; (ii) a general polynomial-time algorithm for the problem of recovery: outputting a vector that correlates with the hidden assignment significantly better than a random guess would. Analogous to the spectral algorithm for community detection [1], [2], the detection and recovery algorithms are based on the spectra of a matrix that arises as the derivatives of the belief propagation update rule. To devise a spectral algorithm in our general model, we obtain bounds on the spectral norms of certain families of random matrices with correlated and matrix valued entries. We then demonstrate how eigenvectors of various powers of the matrix can be used to partially recover the hidden variables.

Machine Learning, Probabilistic Inference, System Identification and Control