Prediction and Postdiction: Real-Time Data Mining and Data Analytics

The world generates an unfathomable amount of data every day. The speed with which the data gets generated, transmitted, ingested, and crunched is nothing but spectacular in the recent past. Expert thinkers and pundits across the globe are of the opinion that data is the transformation agent. Data is positioned as a strategic asset for any institution, innovator, and individual to grow and glow and is being reasoned as the new fuel for bringing in real and sustainable business transformation. It is a universally accepted fact that data can be methodically processed, mined, and analyzed to produce actionable insights. There are batch and real-time data processing methods to make sense of data heaps. Further on, there are integrated data analytics platforms in plenty, to extract hidden patterns, associations, and other useful insights out of data volumes. The data analytics ecosystem grows continuously considering the importance of data-driven insights and insights-driven decisions for businesses as well as people to be agile, adaptive and adroit in their dealings and deeds.Off late real-time data capture, storage, analytics, decision-making, and action are being insisted upon vehemently considering the evolving business dynamics. Any data or message has to be carefully captured, cleansed, and crunched immediately in order to be really beneficial for businesses and commoners. It is indisputable that the data value goes down sharply with time. Another facet is that agile and autonomous business systems are extremely event-driven. That is, any business event may trigger a suite of events across. Thus, all kinds of event data/messages have to be received and processed in real time in order to activate and automate one or more business operations. In short, for envisaging and realizing real-time services, applications and enterprises, real-time data analytics capability is very much indispensable. Enterprises are therefore keenly strategizing and setting up analytics infrastructure modules with all the clarity and alacrity to make sense out of both internal and external data in time. Such a futuristic and flexible capability helps business houses to be sagaciously steered in the right direction.

The rapid development of smart technologies and digital infrastructure has transformed the way governments and organizations monitor and manage urban environments. Smart societies rely on advanced technologies such as artificial intelligence, Internet of Things devices, biometric identification systems, and large-scale data analytics to improve public services, enhance security, and optimize urban management. Digital surveillance systems have become a central component of smart city governance by enabling real time monitoring of public spaces, transportation systems, communication networks, and citizen activities. While these technologies provide numerous benefits in terms of public safety, crime prevention, and efficient resource management, they also raise significant concerns regarding privacy protection and civil liberties. Digital surveillance technologies collect and analyze vast amounts of personal and behavioral data, which may create risks related to unauthorized monitoring, data misuse, and excessive governmental control. Civil liberty advocates argue that widespread surveillance may undermine fundamental democratic principles such as freedom of expression, freedom of association, and the right to privacy. As smart societies continue to expand the use of surveillance technologies, it becomes increasingly important to understand how these systems influence public perceptions of privacy, governance transparency, and civil rights protection. This study analyzes the effect of digital surveillance systems on civil liberties within smart societies. The research develops a conceptual model that examines the relationships between digital surveillance intensity, perceived privacy risk, governance transparency, and protection of civil liberties. Data were collected from citizens, technology professionals, and policy analysts involved in smart city initiatives. Structural Equation Modeling using Smart Partial Least Squares was employed to test the relationships between constructs. The results indicate that increased digital surveillance intensity significantly raises perceived privacy risks among citizens. However, governance transparency and regulatory oversight mechanisms play a critical role in mitigating negative perceptions and protecting civil liberties. The study contributes to digital governance and technology policy research by providing empirical insights into the complex relationship between surveillance technologies and civil liberty protection in smart societies. The findings highlight the need for balanced governance frameworks that ensure security benefits while safeguarding individual rights and democratic values.

In the rapidly evolving landscape of data mining, the effective extraction of valuable insights from large datasets is paramount. This survey paper investigates the pivotal roles of data visualization and analytics in the intricate process of data mining abstraction. We delve into the symbiotic relationship between these two components, examining how they synergistically contribute to the extraction, representation, and interpretation of meaningful patterns and trends within complex datasets. The survey begins by elucidating the fundamental concepts of data mining abstraction and the significance of distilling actionable knowledge from raw data. It subsequently explores the multifaceted benefits of data visualization, elucidating its role in pattern identification, insight generation, and seamless communication of findings to diverse stakeholders.In parallel, the paper navigates through the landscape of data analytics, unraveling its diverse methods such as descriptive analytics, predictive analytics, and prescriptive analytics. Emphasis is placed on how these analytical techniques enhance the abstraction process, providing statistical rigor and predictive power to unveil hidden insights.The integration of data visualization and analytics is a focal point, showcasing their collective impact on various stages of data mining. From exploratory data analysis (EDA) for initial dataset understanding to the evaluation of mining models, the survey illuminates the collaborative nature of these components. Interactive dashboards emerge as a powerful tool, allowing users to dynamically explore datasets, visualize trends, and perform real-time analytics. Key Words:Data visualization , Data analytics ,Data mining , Tools and Software , Metrics and Key Performance Indicators (KPIs).

The unprecedented proliferation of miniaturized sensors and intelligent communication, computing, and control technologies have paved the way for the development of the Industrial Internet of Things. The IIoT incorporates machine learning and massively parallel distributed systems such as clouds, clusters, and grids for big data storage, processing, and analytics. In IIoT, end devices continuously generate and transmit data streams, resulting in increased network traffic between device-cloud communication. Moreover, it increases in-network data transmissions. requiring additional efforts for big data processing, management, and analytics. To cope with these engendered issues, this article first introduces a novel concentric computing model (CCM) paradigm composed of sensing systems, outer and inner gateway processors, and central processors (outer and inner) for the deployment of big data analytics applications in IIoT. Second, we investigate, highlight, and report recent research efforts directed at the IIoT paradigm with respect to big data analytics. Third, we identify and discuss indispensable challenges that remain to be addressed for employing CCM in the IIoT paradigm. Lastly, we provide several future research directions (e.g., real-time data analytics, data integration, transmission of meaningful data, edge analytics, real-time fusion of streaming data, and security and privacy).

In the digital era, data is one of the most important assets since it conceals valuable information. Developers of data-intensive systems have new challenges at each level of streaming, storing, and processing large quantities of data in a variety of forms and speeds. Obtaining useful information at the proper time and place is also crucial. Since the value of information is inversely proportional to time, real-time data processing and analytics are receiving more attention. Due to the importance of real-time data processing and analytics, this study focuses on real-time data processing concepts and terminology, popular technologies used in real-time data processing and analytics, popular NoSQL storage technologies used in real-time data processing, and real-time data processing application areas. The purpose of this paper is to provide researchers of real-time analysis and developers of data-intensive systems with a comparative perspective on real-time data processing by highlighting the key characteristics of real-time data processing technologies, NoSQL storage technologies, their application domains, and selected examples from previous studies.

Analysis and assessment of a knowledge based smart city architecture providing service APIs

With the advent of the era of network, video surveillance technology is more and more integrated into the IT emerging technologies. Big data technology in the field of video surveillance has shown a broad development path. The research of surveillance video data mining based on big data is committed to achieve a perfect fusion of the big data technology and video surveillance business, in order to create a solution of intelligent video surveillance in big data era. In this paper, based on the application requirements of video surveillance on big data, we designed the framework of big data analysis for surveillance video, and study the key technology of storage and processing for surveillance video data based on big data and the value of data mining. The study on real-time intelligence analysis and data mining, progresses the video surveillance from manual sampling to efficiently advance warning, post analysis, realization of intelligent information analysis, forecasting. It brings profound changes in the field of video surveillance business development.

In order to ensure the economic viability of micro-grids with diverse types of distributed energy resources, there is need to reduce maintenance costs and improve long-term reliability. A real-time condition monitoring system with data analytics would be an important step to achieving this objective. This paper introduces an experimental condition monitoring and health prognosis system (CMHPS) for an isolated micro-grid testbed located on a remote small island off the main Singapore island. Within this micro-grid monitoring system, module level monitoring for solar photovoltaic system and cell level monitoring for battery energy storage system has been achieved in real time. All the transmitted data are stored and processed in a PI database server located in our campus located far away from the microgrid. This experimental setup is serving as a blueprint to develop smart multi-energy microgrid systems at district and estate levels.

Water management has become a challenge and a key factor to achieve proper field management in a brownfield in the Amazon region. Apart from historically increasing the basic sediments and water (BSW) and flow assurance events, the longevity, ullage, and lack of maintenance of the processing facilities have come key conditions to sustain production and reduce lifting costs. An integrated digital solution was implemented to enable predictive analysis and behavior of the water system at the facilities level, such as horizontal pump system (HPS) failures, line plugging, injection rate deviations, as well as forecasting of injection rates in real time to improve the efficiency of operations and avoid production deferment. Several failures had occurred in the water-handling system caused by the lack of real-time monitoring or fast event detection and corrective actions. This has led to many shut-ins and failures of electric submersible pumps (ESPs), and injectivity losses in the waterflooding and disposal wells, triggering production losses. Hardware for data collection in selected points and customized digital advanced workflows using data analytics and machine learning (ML) were implemented and developed to optimize processes and production. Therefore, using the connectivity provided by a satellite system, supervisory control, and data acquisition (SCADA) optical fiber and operations monitoring platform, the variables are now monitored in real time to enable early identification of events, give a rapid response, and optimize production of the field. The digital solution was implemented using the following steps: Data capturing: Data gathering was improved by adding sensors at key points, aiming to minimize data gaps in the field, and ensuring comprehensive availability in the database. Surveillance workflows: Multiple workflows were created using edge computing and ML to have an early alert and recommendations to make a rapid response. Opportunity identification: Workflows were implemented in real time, enabling early identification of events in the field to give a rapid response, debottlenecking, and optimize production by proper water-handling management through data shown in engineering dashboards. This innovative and integrated digital solution has shown outstanding results. Monitoring the data from the water-handling system in real time and applying engineering workflows has led to a 76% reduction in processing time, 75% reduction in commuting for data gathering, and 5 t of carbon dioxide (CO2) reduction per year. Due to the early event identification, the prediction of potential failures, and a rapid response time, the operational team has reduced deferred production by 283,000 bbl of oil, saved USD 9 million related to HPS failure reduction and saved production, which contributed to extending the ESP's run life (RF), optimizing maintenance costs, and reducing lifting costs. In this paper we outline how the implementation of a novel digital solution can optimize the process, production, and cost of the critical points for a brownfield by using data analytics, edge computing, and ML, potentiating the digital transformation path in all stages of the energy industry.

In a brown field in Ecuador, water handling has become a challenge to achieve proper field management. Apart from increasing water cuts, longevity and ullage of the processing facilities have turned into a key condition to sustain production and reduce lifting costs. A digital solution was implemented to enable predictive analysis for horizontal pump failures and line plugging, as well as forecasting of injection rates in real-time to improve the efficiency of operations to maximize productivity. Numerous failures occurred in the water handling system because of the lack of real-time monitoring or fast detection. This caused approximately 60 Electrical Submersible Pumps (ESPs) to be shut-in every year in this brown field, triggering production losses. Hardware for data collection in selected points and customized digital workflows using data analytics and machine learning processes were developed and implemented so that, with the help of edge computing, the system was able to predict failures and estimate injection rates in real time. Using the connectivity provided by a satellite system, supervisory control and data acquisition (SCADA) optical fiber and operations monitoring platform, the variables are now monitored in real time to enable early identification of events, give a rapid response, and optimize production of the field. The northern flow station of this brown field, located in the most prolific area and where the water flooding scheme has the highest relevance, was selected to implement the digital pilot. The implementation of this digital initiative has shown outstanding results. For the first time, monitoring the data from the water handling system in real time and applying the engineering workflows (data analytics and machine learning) led to a 76% reduction in the time spent in manual processing, 75% reduction in the time spent in of the time for commuting, and 1-ton carbon dioxide equivalent (CO2eq) reduction in emissions per year. The time saved is now used to improve other engineering workflows that are equally important in increasing the productivity of the field. Because of the early identification of events, the prediction of potential failures, and a timely response to previous functional failures, the operational team can reduce the deferred production associated with ESPs shut-ins, which for the previous years represented 100,000 barrels of oil (approximately USD 2.4 million revenue for the asset). In addition, such actions have contributed to extend the ESPs’ run-life, optimized maintenance costs, and reduce lifting costs by 0.2%. In this paper we show the selection criteria of surface facilities and measurement of critical points for data gathering, the application of data analytics with edge computing, and the development of an innovative digital solution in conjunction with the client and different disciplines. This case shows the benefits of a digital mindset in any oilfield operation to optimize production and cost, potentiating the digital transformation path in the energy industry.

Conducting (big) data analytics in an organization is not just about using a processing framework (e.g. Hadoop/Spark) to learn a model from data currently in a single file system (e.g. HDFS). We frequently need to pipeline real time data from other systems into the processing framework, and continually update the learned model. The processing frameworks need to be easily invokable for different purposes to produce different models. The model and the subsequent model updates need to be integrated with a product that may require a real time prediction using the latest trained model. All these need to be shared among different teams in the organization for different data analytics purposes. In this paper, we propose a real time data-analytics-as-service architecture that uses RESTful web services to wrap and integrate data services, dynamic model training services (supported by big data processing framework), prediction services and the product that uses the models. We discuss the challenges in wrapping big data processing frameworks as services and other architecturally significant factors that affect system reliability, real time performance and prediction accuracy. We evaluate our architecture using a log-driven system operation anomaly detection system where staleness of data used in model training, speed of model update and prediction are critical requirements.

The military establishment, mimicking the business world in many respects, is abuzz with the rapid growth in big data, data analytics, machine learning, and statistical learning using artificial intelligence as well as supervised and unsupervised learning. But the challenge is really understanding what all these concepts mean for the military and how do these concepts relate the fundamental statistical principles historically employed by the defense operations research analyst or statistician. This chapter defines the key terms in the area and discusses the key aspects of the modern data challenge: big data, data engineering, data science, data analytics, and data mining. The chapter then focuses on data analytics and the range of methods and concerns associated with data analytics, and a discussion of the some of the techniques within the data analytics realm. The chapter closes with recommendations and concerns for the future.

In today's fast changing world, cybercrimes are growing at perturbing pace. At the very definition of it, cybercrimes get engendered by capitalizing on threats and exploitation of vulnerabilities. However, recent history reveals that such crimes often come with surprises and seldom follow the trends. This puts the defense systems behind in the race, because of their inability to identify new patters of cybercrime and to ameliorate to the required levels of security. This paper visualizes the empowerment of security systems through real-time data mining by the virtue of which these systems will be able to dynamically identify patterns of cybercrimes. This will help those security systems stepping up their defense capabilities, while adapting to the required levels posed by newly germinating patterns. In order to confine within scope of this paper, the application of this approach is being discussed in the context of selected scenarios ofcybercrime.

article Free Access Share on Electronic surveillance and civil liberties: Testimony of Fred W. Weingarten before the House Judiciary Subcommittee on Courts, Civil Liberties and Administration of Justice Author: Fred Weingarten View Profile Authors Info & Claims ACM SIGCAS Computers and SocietyVolume 15Issue 4Winter 1986 pp 13–17https://doi.org/10.1145/15488.15492Published:01 February 1986Publication History 1citation93DownloadsMetricsTotal Citations1Total Downloads93Last 12 Months9Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Ever-increasing hardware capabilities and computation powers have enabled acquisition and analysis of big scientific data at the nanoscale routine, though much of the data acquired often turn out to be redundant, noisy and/or irrelevant to the problems of interest, and it remains nontrivial to draw clear mechanistic insights from pure data analytics. In this work, we use scanning probe microscopy (SPM) as an example to demonstrate deep data methodology for nanosciences, transitioning from brute-force analytics such as data mining, correlation analysis and unsupervised classification to informed and/or targeted causative data analytics built on sound physical understanding. Three key ingredients of such deep data analytics are presented. A sequential excitation scanning probe microscopy (SE-SPM) technique is first developed to acquire high-quality, efficient and physically relevant data, which can be easily implemented on any standard atomic force microscope (AFM). Brute-force physical analysis is then carried out using a simple harmonic oscillator (SHO) model, enabling us to derive intrinsic electromechanical coupling of interest. Finally, principal component analysis (PCA) is carried out, which not only speeds up the analysis by four orders of magnitude, but also allows a clear physical interpretation of its modes in combination with SHO analysis. A rough piezoelectric material has been probed using such a strategy, enabling us to map its intrinsic electromechanical properties at the nanoscale with high fidelity, where conventional methods fail. The SE in combination with deep data methodology can be easily adapted for other SPM techniques to probe a wide range of functional phenomena at the nanoscale.