Efficient Tensor Completion for Internet Traffic Data Recovery

Traffic sensor networks allow convenient collection of travel data that are of great significance for intelligent transportation systems (ITSs). However, the universality of missing data impedes the application of ITS and thus accurate missing data recovery is indispensable in practice. Typically, the global low-rankness and local spatiotemporal smoothness exist in underlying traffic tensor data. In light of this, this article proposes an improved low-rank tensor completion (LRTC) model by exploiting abundant structural information from incomplete tensors. Specifically, a logarithm power composite (LPC)-norm is first proposed as a nonconvex substitute of the rank function, leading to a flexible characterization of tensor multidimensional correlation. Then, a joint structural regression (JSR) model is presented to simultaneously leverage the intrinsic temporal continuity and profile similarity of traffic data. By doing so, we construct a novel nonconvex LRTC model by integrating the global low-rankness and fine-grained spatiotemporal structure that are complementary to each other. To solve the proposed model, following the optimization framework of the alternating direction method of multipliers (ADMMs), we develop an efficient iterative algorithm where each step can be solved in a closed form. Extensive experiments on four real-world traffic data are conducted to evaluate the effectiveness of the proposed approach. The results demonstrate that compared with other tensor completion methods, our model significantly improves the recovery performance.

The Internet of Things (IoT) has enormous potential to transform the transport industry by improving passenger experiences, safety, and efficiency. However, the collected spatiotemporal data by traffic sensor network often suffer from missing values (MVs), which affect the overall performance of the system. As a result, accurate recovery of MVs is essential for the successful application of IoT in transportation. In this article, we propose a novel MVs imputation model by integrating low-rank tensor completion (LRTC) and sparse self-representation into a unified framework. In doing so, the global multidimensional correlation, as well as sample self-similarity, can be well leveraged for imputation. In order to solve the proposed model, an elaborate solution algorithm is developed, following the principle of alternating direction method of multipliers (ADMMs). Importantly, each step in ADMM can be implemented efficiently by analyzing the problem structure. Moreover, in order to select proper parameters for the model, an improved harmony search heuristic algorithm based on dual harmonies generation strategy is developed, thus sufficiently considering the information contained in current harmony memory. The experiments on two real-world traffic data are carried out to evaluate the proposed approach. The results verify that in comparison with the classic matrix/tensor completion and other competing algorithms, our method significantly improves the imputation performance.

Tensor completion via multi-shared-modes canonical correlation analysis

A parallel low rank matrix optimization method for recovering internet traffic network data via link flow measurement

Complete traffic sensor data is considered to be one of the critical ingredients for intelligent transportation systems (ITS). However, the traffic measurements prevalently suffer from the inevitable and ubiquitous missing values. Current missing data completion algorithms are difficult to leverage the global low-rank property and the fine-grained spatial-temporal structure simultaneously. To circumvent this problem, this work presents a novel collaborative nonconvex low-rank spatial-temporal data tensor completion model that can take full advantage of the inherent spatial-temporal characteristics of traffic measurement data. First, the tensor Schatten p-norm, as an effective nonconvex surrogate of rank function, is used to exploit the global multi-dimensional correlation of traffic data. Then, we present an elastic net self-representation method and utilize an autoregression model in order to simultaneously capture the self-similarity and the temporal continuity of traffic data acquired in the same sensor network. By integrating the above elements in a unified nonconvex learning model, our method can explore the inherent structure of traffic data from the viewpoints of both global multi-dimensional correlation and fine-grained spatial and temporal dependency. Then, in the view of the general framework of the alternating directions method of multipliers (ADMM), an efficient iterative algorithm is designed to solve our model. Besides, to optimize the parameter combination of the model, a Bayesian optimization-based parameter selection algorithm is developed, which avoids manual parameter adjustment. Extensive experiments and analyses on two real-world traffic datasets are carried out. The results demonstrate the advantages of our model under diverse missing patterns and missing ratios.

Data missing remains a difficult and important problem in the transportation information system, which seriously restricts the application of the intelligent transportation system (ITS), dominatingly on traffic monitoring, e.g., traffic data collection, traffic state estimation, and traffic control. Numerous traffic data imputation methods had been proposed in the last decade. However, lacking of sufficient temporal variation characteristic analysis as well as spatial correlation measurements leads to limited completion precision, and poses a major challenge for an ITS. Leveraging the low-rank nature and the spatial-temporal correlation of traffic network data, this paper proposes a novel approach to reconstruct the missing traffic data based on low-rank matrix factorization, which elaborates the potential implications of the traffic matrix by decomposed factor matrices. To further exploit the temporal evolvement characteristics and the spatial similarity of road links, we design a time-series constraint and an adaptive Laplacian regularization spatial constraint to explore the local relationship with road links. The experimental results on six real-world traffic data sets show that our approach outperforms the other methods and can successfully reconstruct the road traffic data precisely for various structural loss modes.

Traffic data is a fundamental component for applications and researches in transportation systems. However, real traffic data collected from loop detectors or other channels often include missing data which affects the relative applications and researches. This paper proposes an approach based on deep learning to impute the missing traffic data. The proposed approach treats the traffic data including observed data and missing data as a whole data item and restores the complete data with the deep structural network. The deep learning approach can discover the correlations contained in the data structure by a layer-wise pre-training and improve the imputation accuracy by conducting a fine-tuning afterwards. The authors analyze the imputation patterns that can be realized with the proposed approach and conduct a series of experiments. The results show that the proposed approach can keep a stable error under different traffic data missing rate. Deep learning is promising in the field of traffic data imputation.

Missing data are critical deficiency in the investigation of displacement measurement in urban Internet of Things system. In the insight of recovering missing displacement data, this article presents a data-driven and high-dimensional gap-imputation method, Tucker decomposition with L2 regularization. Results on the global navigation satellite system (GNSS) time series collected from an intelligent structural health monitoring system show that the recovery accuracy is improved compared with some popular benchmark methods. When the missing rate is 50%, compared with singular spectrum analysis, singular value decomposition, CP optimization algorithm, k -nearest neighbors, and Tucker decomposition via alternating least squares, Tucker decomposition with L2 regularization can improve the average mean absolute error by about 4.74, 4.95, 5.82, 2.29, and 5.67 mm for all locations. It can be concluded that the consideration of multiple temporal correlations is necessary for missing data imputation. Compared with matrix decomposition, tensor decomposition can improve the ability for high-dimensional correlations in the GNSS time series.

Detecting anomalous traffic is a crucial task of managing networks. Many anomaly detection algorithms have been proposed recently. However, constrained by their matrix-based traffic data model, existing algorithms often suffer from low detection accuracy. To fully utilize the multi-dimensional information hidden in the traffic data, this paper takes an initiative to investigate the potential and methodologies of performing tensor factorization for more accurate Internet anomaly detection. Only considering the low-rank linearity features hidden in the data, current tensor factorization techniques would result in low anomaly detection accuracy. We propose a novel Graph-based Tensor Recovery model (Graph-TR) to well explore both low rank linearity features as well as the non-linear proximity information hidden in the traffic data for better anomaly detection. We encode the non-linear proximity information of the traffic data by constructing nearest neighbor graphs and incorporate this information into the tensor factorization using the graph Laplacian. Moreover, to facilitate the quick building of neighbor graph, we propose a nearest neighbor searching algorithm with the simple locality-sensitive hashing (LSH). We have conducted extensive experiments using Internet traffic trace data Abilene and GEANT. Compared with the state of art algorithms on matrix-based anomaly detection and tensor recovery approach, our Graph-Trcan achieve significantly lower False Positive Rate and higher True Positive Rate.

With the development and popularity of information technology and ITS, Searching for traffic data and solving traffic problems usually requires collaborative work among a group of people in different geographic locations. A Distributed Collaborative Urban Traffic Big Data system (DCUTBD) is an Internet-based virtual big data environment which provides users with shared resource space and a collaborative platform for publishing multidimensional traffic data, and for visualizing and analyzing complex traffic problems. Cloud Computing technology provides effective solutions for constructing DCUTBD systems because of its ability to provide shared computer processing and storage resources to computers and other devices on demand, and also because of its ability to share programs, data, and software. This paper sums up the urban traffic big data, and analyzes the characteristics of cloud computing technology. The architecture and working mechanism of the DCUTBD system based on cloud services will be elaborated. To demonstrate DCUTBD systems based on cloud computing, we examine a case study of traffic data visualization in Xicheng district, Beijing, China, using a prototype DCUTBD system. The experimental results show that the scheme developed in this paper is efficient and feasible.

On the relationship between the activity at point of interests and road traffic

As machine learning (ML)-based traffic classification develops, Internet traffic data is published in public to serve as test data. Although the IP addresses therein are anonymized, it is given explicitly which data belongs to an identical user. Then using the information, an adversary can identify a user from the anonymized users. The paper first gives a k-anonymity method to reduce the probability of information leak to P/k, where P is the probability of information leak without k-anonymity. Assume the number of the flows belonging to an IP address follows Normal distribution, the information loss is shown (μ2+σ2)/(kμ2+σ2), where μ and σ are respectively the mean and the variance of the Normal distribution. Later, random noise is added to further reduce the probability of information leak to P/k2, with an expected distortion rate of approximately 2d+log k-log|X|, where d is the number of dimensions and |X| is the number of the vectors. At last, real-world Internet traffic data is used to evaluate the utility of the anonymized traffic data. According to the experimental results, the k-anonymized noised data can be clustered with an overall accuracy rate close to the state-of-the-art results for non-anonymized traffic data.

Analyzing of traffic data is an important task for urban planners and managers. Traffic data have the spatio-temporal characteristics, which can reflect the variation of the presence of vehicle in different places over time as well as traffic flow dynamics among different places. The analysis of large-scale GPS trajectory data is a very challenging research due to the complexity of data and the need to extract useful information under cover in data. In this study, we combine temporal and geospatial aggregation of traffic data for obtaining key areas and creating legible traffic flow maps; meanwhile, we make full use of the topic model to capture latent semantic information. Nevertheless, most of the topic models always encounter the plague of choosing the optimal number of topics and cannot easily incorporate numerous types of user feedback. To tackle these problems, we propose an interactive topic modeling equipped with various interactive capabilities which empowers users to explore data from different levels of detail. Finally, we design and implement an interactive visual analytics prototype system based on the spatio-temporal graphs and the interactive topic modeling. The feasibility and validity of our system is demonstrated by conducting two case studies with a real-world traffic data in Hangzhou.

A data fusion framework is proposed to predict vehicle trajectories in urban signalized arterials. Recent advancements in data collection techniques and availability of traffic data from fixed and probe sensors suggest data fusion-based models as an alternative approach for traffic prediction. Yet, majority of the existing data fusion approaches are based on statistical models without considerations for traffic engineering principles. In addition, existing approaches do not use probe trajectory data efficiently. The proposed framework in this research is based on the kinematic wave theory and is capable of fully utilizing the probe trajectory data to reconstruct the trajectories of the other vehicles within a ‘prediction window’. The data fusion framework combines real-time and historical traffic data to predict future traffic patterns at upstream and downstream boundaries. The modelling approach is based on the kinematic wave theory and applies the variational theory as the solution method. Predicted traffic patterns are used to set the boundary conditions in the solution domain and probe trajectories are used to set additional boundary conditions. Given the boundary conditions, a dynamic programming approach is applied to reconstruct vehicle trajectories within the prediction window. The performance of the proposed framework is evaluated by using real-world traffic data, and possible directions for improving the accuracy of the model are discussed.

Recent advances in traffic monitoring systems have made real-time traffic velocity data ubiquitously accessible for drivers. This paper develops a traffic data-enabled predictive energy management framework for a power-split plug-in hybrid electric vehicle (PHEV). Compared with conventional model predictive control (MPC), an additional supervisory state of charge (SoC) planning level is constructed based on real-time traffic data. A power balance-based PHEV model is developed for this upper level to rapidly generate battery SoC trajectories that are utilized as final-state constraints in the MPC level. This PHEV energy management framework is evaluated under three different scenarios: 1) without traffic flow information; 2) with static traffic flow information; and 3) with dynamic traffic flow information. Numerical results using real-world traffic data illustrate that the proposed strategy successfully incorporates dynamic traffic flow data into the PHEV energy management algorithm to achieve enhanced fuel economy.