Real-Time In-Class Data Mining of Moodle Click Logs for Learning Pattern Classification and Outlier Detection

Stream computing has emerged as a new processing paradigm that processes incoming data streams from large numbers of sensors in real time. At the same time, many recent efforts have shown the suitability of GPGPU s for batch-typed long-running applications. However, few studies conduct the applicability of GPGPU to stream computing and also our first experiment shows that the performance does not scale as expected if one introduces the GPGPU to stream computing applications. This paper presents the workload characterization of GPGPU-based stream computing. We especially focus on computing SVD (Singular Value Decomposition) with GPGPUs for small-sized matrix since it can be widely applicable to real-time stream-based data mining applications such as real-time anomaly detection. In this paper, we not only show the workload characterization of GPGPU-based stream computing but also propose the optimization approach of SVD for stream computing applications called “GPU task Parallelism” to leverage the sufficient capability of GPGPUs. This optimization offers new levels of scalability for real-time operations with large numbers of sensors. Our experimental results show that GPU task parallelism provides roughly four-fold performance gains against a quad-core CPU for matrixes of 300 × 300 to 500 × 500 data values. We also implemented a stream-based change-point and anomaly detection system based on the optimized SVD and stream computing system called System S. By porting the optimized version of SVD to the distributed stream computing system, it was easy to exploit multiple GPUs on multiple nodes. The performance results showed performance around 7.6 times faster than CPUs. The scalability of our proposed system was tested up to 1,525 sensors, which were simultaneously handled for change-point detection every 5 seconds.

Outlier detection is one of the most important data mining techniques. It has broad applications like fraud detection, credit approval, computer network intrusion detection, anti-money laundering, etc. The basis of outlier detection is to identify data points which are “different” or “far away” from the rest of the data points in the given dataset. Traditional outlier detection method is based on statistical analysis. However, this traditional method has an inherent drawback—it requires the availability of the entire dataset. In practice, especially in the real time data feed application, it is not so realistic to wait for all the data because fresh data are streaming in very quickly. Outlier detection is hence done in batches. However two drawbacks may arise: relatively long processing time because of the massive size, and the result may be outdated soon between successive updates. In this paper, we propose several novel incremental methods to process the real time data effectively for outlier detection. For the experiment, we test three types of mechanisms for analyzing the dataset, namely Global Analysis, Cumulative Analysis and Lightweight Analysis with Sliding Window. The experiment dataset is “household power consumption” which is a popular benchmarking data for Massive Online Analysis.

With recent advancements in field instrumentation technologies and the expansion of Saudi Aramco's state-of-the-art intelligent field infrastructure, a wealth of valuable knowledge is being derived from integrating intelligent field capabilities with best-in-class reservoir management practices. Intelligent field applications have already leveraged real-time reservoir surveillance data to enable timely responses and fast interventions for events observed from real-time data. As richer and more granular datasets are collected from the field, deeper insights and understanding of the reservoirs are gained at fractions of the needed time where intelligent field capabilities are not present. The abundance and timely processing of intelligent field data enables more tactical adjustments in reservoir management strategy. This type of real-time reservoir management has already shown great value in reservoir performance optimization. This paper discusses several cases in which an intelligent field enabled early detection of anomalies and shortened the time-cycle for an analysis, decision and action. The first case discusses the early detection of localized pressure support anomalies due to poor rock quality. This timely detection, facilitated by intelligent field infrastructure, enabled quick decision making and actions to overcome the anomaly. The second case shows how real-time monitoring of reservoir pressure propagation has helped to detect over-injection in two injectors with malfunctioned meters. This has allowed timely action to ensure uniform pressure propagation and repair of faulty equipment. The third case shows that the utilization of real-time data has assisted in detecting a casing leak in a short time, which helped in making a corrective action to restore the well's capacity. In all three cases, had an intelligent field not been there, the time-cycle from detection to analysis, and to decision and action, would have been substantially longer, resulting in higher operational costs and an impact on reservoir performance.

In today's fast changing world, cybercrimes are growing at perturbing pace. At the very definition of it, cybercrimes get engendered by capitalizing on threats and exploitation of vulnerabilities. However, recent history reveals that such crimes often come with surprises and seldom follow the trends. This puts the defense systems behind in the race, because of their inability to identify new patters of cybercrime and to ameliorate to the required levels of security. This paper visualizes the empowerment of security systems through real-time data mining by the virtue of which these systems will be able to dynamically identify patterns of cybercrimes. This will help those security systems stepping up their defense capabilities, while adapting to the required levels posed by newly germinating patterns. In order to confine within scope of this paper, the application of this approach is being discussed in the context of selected scenarios ofcybercrime.

Anomaly detection plays a critical role in large-scale industrial systems, where real-time streaming data is generated at high volumes. This research journal presents an in-depth study on developing an efficient anomaly detection algorithm specifically designed for such industrial systems. The goal is to identify abnormal patterns and deviations from normal behavior, enabling proactive maintenance, improved operational efficiency, and reduced downtime. The proposed algorithm leverages machine learning and stream processing techniques to handle the challenges associated with real- time streaming data analysis. The research covers algorithm design, implementation, evaluation, and performance analysis using large-scale datasets from industrial domains. Keywords: anomaly detection, real-time streaming data, large-scale industrial systems, machine learning, stream processing

Recently, artificial intelligence (AI) techniques have been used to describe the characteristics of information, as they help in the process of data mining (DM) to analyze data and reveal rules and patterns. In DM, anomaly detection is an important area that helps discover hidden behavior within the data that is most vulnerable to attack. It also helps detect network intrusion. Algorithms such as hybrid K-mean array and sequential minimal optimization (SMO) rating can be used to improve the accuracy of the anomaly detection rate. This paper presents an anomaly detection model based on the machine learning (ML) technique. ML improves the detection rate, reduces the false-positive alarm rate, and is capable of enhancing the accuracy of intrusion classification. This study used a dataset known as network security-knowledge and data discovery (NSL-KDD) lab to evaluate a proposed hybrid ML technology. K-mean cluster and SMO were used for classification. In the study, the performance of the proposed anomaly detection was tested, and results showed that the use of K-mean and SMO enhances the rate of positive detection besides reducing the rate of false alarms and achieving a high accuracy at the same time. Moreover, the proposed algorithm outperformed recent and close work related to using similar variables and the environment by 14.48% and decreased false alarm probability (FAP) by (12%) in addition to giving a higher accuracy by 97.4%. These outcomes are attributed to the common algorithm providing an appropriate number of detectors to be generated with an acceptable accurate detection and a trivial false alarm probability (FAP). The proposed hybrid algorithm could be considered for anomaly detection in future data mining systems, where processing in real-time is highly likely to be reduced dramatically. The justification is that the hybrid algorithm can provide appropriate detectors numbers that can be generated with an acceptable detection accuracy and trivial FAP. Given to the low FAP, it is highly expected to reduce the time of the preprocessing and processing compared with the other algorithms.

Distillation anomaly and fault detection based on unsupervised dimensionality principal component analysis method

Outlier detection (or anomaly detection) is a fundamental task in data mining. Outliers are data that deviate from the norm and outlier detection is often compared to “finding a needle in a haystack.” However, the outliers may generate high value if they are found, value in terms of cost savings, improved efficiency, compute time savings, fraud reduction and failure prevention. Detection can identify faults before they escalate with potentially catastrophic consequences. Big Data refers to large, dynamic collections of data. These vast and complex data appear problematic for traditional outlier detection methods to process but, Big Data provides considerable opportunity to uncover new outliers and data relationships. This chapter highlights some of the research issues for outlier detection in Big Data and covers the solutions used and research directions taken along with an analysis of some current outlier detection approaches for Big Data applications.

This paper contains review of algorithms, methods and tools nowadays used for anomaly detection.Anomaly detection is used in many domains of science and industry, some authors classify anomaly detection as data mining and data science tool, others state it is decision support tool under artificial intelligence domain and indeed the use cases of anomaly detection are very different.The article describes the main algorithms used for anomaly detection from perspective of theory of computer science and practical use cases of anomaly detection in different domains of industry. Several paragraphs are dedicated to the frameworks used by one of the most popular and powerful anomaly detection tools available in the market - Microsoft Anomaly detector.

Outlier detection or anomaly detection is applied in numerous applications, such as fraud detection, network intrusion detection, manufacturing, and environmental monitoring. Due to the continuous and dynamic characteristics of streaming data, outlier detection over data streams becomes a very challenging task. When analyzing real-time data streams, it is typically impossible to store the entire set of data due to space limitations. Also due to the high data rates, it is necessary to produce the results in a limited amount of time. Parallel processing power of Graphics Processing Units (GPUs) can be used to accelerate the outlier detection process, and thus address the challenges of outlier detection over data streams. This paper proposes a GPU-accelerated outlier detection algorithm for continuous data streams using kernel density estimation approach. Experiments show that the proposed SOD_GPU algorithm is efficient for detecting outliers in high-dimensional, high-speed data streams, and produces results in a timely manner without compromising the outlier detection accuracy. The proposed method achieved up to 20X speedup compared to a respective multi-core CPU implementation, and the speedup increases with the number of data attributes and the input data rate.

Cloud computing and big data: Technologies and applications

Nowadays, more and more unmanned systems are adopting time series anomaly detection algorithms as the primary tool for KPI monitoring. However, many anomalous data in real environments often have complex features such as large data fluctuations, irregular data fluctuations and large data drift. Traditional univariate or multivariate anomaly detection algorithms can hardly meet the demand for anomaly detection of time series with many features and large fluctuations. Therefore, we propose a multivariate anomaly detection framework adapted to complex environments, which can effectively reduce the complexity of time series data and improve the overall performance of anomaly detection algorithms in complex real systems. First, data preprocessing is performed on the time series before anomaly detection to reduce the impact of temporal bias of different data items in multivariate time series data. Second, to intuitively analyze the anomalous fluctuation characteristics of the time series, we use Spectral Residual to process the significance of the time series. Then, we combine the Spectral Residual processed data and Logistic Regression algorithm to construct anomaly classifiers for anomaly detection of real-time data in complex real systems. Finally, by experimenting with a prototype system on a private cloud platform, our approach obtains superior results under multiple anomaly patterns compared with traditional univariate and multivariate anomaly detection algorithms.KeywordsUnmanned systemsMultivariate anomaly detectionSpectral residualConcept drift

In this chapter, I focus on data-mining and data analytics. It is obvious that without integrated databases set up for collecting, collating, managing and disseminating data derived from various identification, monitoring and surveillance systems, all efforts in this direction would be in vain – which is why we need to discuss such computerized databases first. But databases are just the beginning: nowadays, computers can tap into the World Wide Web to proactively search all kinds of social media including listen in on phone calls to detect suspicious behaviour, to profile individual terrorists or suspects. This process is known as data mining and acquisition, and similar to what many private companies such as Google or Facebook routinely do ways as well. I argue that currently, the ‘holy grail’ of data-mining and acquisition is to be able to do so in real time, while ‘it’ happens. With a discussion of this current cutting edge of research and the implications for our civil liberties, I conclude this chapter.

The Outlier detection is very active area of research in data mining where outlier is a mismatched data in dataset with respect to the other available data.In existing approaches the outlier detection done only on numeric dataset.For outlier detection if we use clustering method , then they mainly focus on those elements as outliers which are lying outside the clusters but it may possible that some of the unknown elements with any possible reasons became the part of the cluster so we have to concentrate on that also.The Proposed method uses hybrid approach to reduce the number of outliers.The number of outlier can only reduce by improving the cluster formulation method.The proposed method uses two data mining techniques for cluster formulation i.e. weighted k-means and neural network where weighted kmeans is the clustering technique that can apply on text and date data set as well as numeric data set.Weighted kmeans assign the weights to each element in dataset.The output of weighted k-means becomes the input for neural network where the neural network is the classification and clustering technique of data mining.Training is provided to the neural network and according to that neurons performed the testing.The neural network test the cluster formulated by weighted k-means to ensure that the clusters formulated by weighted k-means are group accordingly.There is lots of outlier detection methods present in data mining.The proposed method use Integrating Semantic Knowledge (SOF) for outlier detection.This method detects the semantic outlier where the semantic outlier is a data point that behaves differently with other data points in the same class or cluster.The main motive of this research work is to reduce the number of outliers by improving the cluster formulation methods so that outlier rate reduces and also to decrease the mean square error and improve the accuracy.The simulation result clearly shows that proposed method works pretty well as it significantly reduces the outlier.

With the advent of the era of network, video surveillance technology is more and more integrated into the IT emerging technologies. Big data technology in the field of video surveillance has shown a broad development path. The research of surveillance video data mining based on big data is committed to achieve a perfect fusion of the big data technology and video surveillance business, in order to create a solution of intelligent video surveillance in big data era. In this paper, based on the application requirements of video surveillance on big data, we designed the framework of big data analysis for surveillance video, and study the key technology of storage and processing for surveillance video data based on big data and the value of data mining. The study on real-time intelligence analysis and data mining, progresses the video surveillance from manual sampling to efficiently advance warning, post analysis, realization of intelligent information analysis, forecasting. It brings profound changes in the field of video surveillance business development.