Affordable non-traditional source data mining for context assessment to improve distributed fusion system robustness

We have seen in the past decade a sharp increase in the extent that companies use data to optimize their businesses. Variously called the `Big Data' or `Data Science' revolution, this has been characterized by massive amounts of data, including unstructured and nontraditional data like text and images, and the use of fast and flexible Machine Learning (ML) algorithms in analysis. With recent improvements in Deep Neural Networks (DNNs) and related methods, application of high-performance ML algorithms has become more automatic and robust to different data scenarios. That has led to the rapid rise of an Artificial Intelligence (AI) that works by combining many ML algorithms together – each targeting a straightforward prediction task – to solve complex problems. We will define a framework for thinking about the ingredients of this new ML-driven AI. Having an understanding of the pieces that make up these systems and how they fit together is important for those who will be building businesses around this technology. Those studying the economics of AI can use these definitions to remove ambiguity from the conversation on AI's projected productivity impacts and data requirements. Finally, this framework should help clarify the role for AI in the practice of modern business analytics and economic measurement.

Big Data is more than just large amounts of data. Big Data allows companies to use enormous amounts of data from non-traditional sources. A non-traditional source is time-sensitive data, not just past recorded data, used to optimize the industry and ports. Big Data includes texts, audios, videos, and real-time information. Big data is a big category which includes the data in both structured and unstructured forms stored in the cloud. For e-commerce, the supply chain data source might be order management systems, warehouses, payroll, inventory systems and carrier data. For the Maritime analytics market traditional data may come from dockyards, ships, vessel operations, bill of lading, traditional (fixed) data is used to analyze profits and losses. Nontraditional is time sensitive and not always quantifiable. Examples of nontraditional data are weather data, traffic and location data and movement of freight via transportation

Analysis of stock market data is essential for enabling stock investors to make wise choices and create winning investment plans. The significance and advantages of applying data analysis techniques for stock market investors are discussed in this abstract.The abstract opens by emphasizing the stock market's rising complexity and volatility, which calls for the use of sophisticated analytical techniques. In order for investors to recognize trends, evaluate risk, and take advantage of opportunities, it highlights the significance of accurate and timely information.The abstract then goes on to describe different data analysis techniques used in stock market analysis. These techniques include sentiment analysis, machine learning, time series forecasting, statistical analysis, and data visualization. It highlights their importance in discovering correlations, deriving meaningful patterns, and forecasting future market moves.The abstract also highlights the importance of historical data research for investors. Data from the historical stock market can be used to gain understanding of current trends, market behavior, and the performance of certain stocks or industries. Investors may make wise judgments, create investing strategies, and efficiently manage risk with the help of these insights.The abstract also emphasizes how crucial alternate data sources are becoming to stock market analysis. Technology has given investors access to a wide range of non-traditional data, including sentiment on social media, news stories, and satellite imagery. Integrating these non- traditional data sources with conventional financial data might give investors new perspectives and a competitive edge.The abstract also recognizes how technology helps stock investors analyze data more easily. Big data processing, artificial intelligence algorithms, and advanced computational approaches are transforming how investors read and analyze enormous volumes of data. Investors can rapidly and effectively get useful insights thanks to these technologies.Finally, the abstract stresses the revolutionary potential of stock market data analysis in its conclusion. Investors can reduce risks, find untapped possibilities, and improve their investing strategies by utilizing data-driven methodologies. Utilizing the power of data analysis is essential for stock investors to manage the complexity and meet their investing goals in a constantly changing and dynamic market. Key words: stock market, data analysis, traders, traders' strategies, alternative data sources, computational methods, artificial intelligence, risk management.

As we are living in the era of big data, huge volumes of a wide variety of complex data-which can be of different levels of veracity-are generated or collected at a high velocity from rich sources of data in various real-life applications. A rich source of these big data sources is the Internet of Things (IoT), which include a collection of sensors, smartphones and other mobile devices, wearable devices, as well as other "things" that are capable to operate within the existing Internet infrastructure. Embedded in these big data are valuable knowledge and useful information. Hence, the research problem of data mining from big IoT data have drawn attention of many researchers as it aims to discover implicit, previously unknown and potentially useful information and knowledge from the data. For instance, frequent pattern mining finds sets of frequently co-occurring items in the IoT domains. Associative classification discovers rules revealing relationships among items within the frequent patterns and their associations with the corresponding class labels. Induction based classification uses decision tree or random forest to learn from old big IoT for classifying or making predictions on new data. Over the past quarter of a century, many serial, distributed, parallel, and MapReduce-based (Hadook-based and Spark-based) big data mining algorithms have been proposed. These algorithms are run in local computers, distributed and parallel environments, clusters, grids, clouds and/or data centers. In this paper, we review some of these algorithms, discuss issues and research prospective in mining classification patterns from these big IoT data in fog. Our case study on a real-life application shows the feasibility on classifying real-life big IoT data over fog for urban analytics.

The Dual Node Decision Wheels (DNDW) architecture is a new approach to information fusion and decision support systems. By combining cognitive systems engineering organizational analysis tools, such as decision trees, with the Dual Node Network (DNN) technical architecture for information fusion, the DNDW can align relevant data and information products with an organization’s decision-making processes. In this paper, we present the Compositional Inference and Machine Learning Environment (CIMLE), a prototype framework based on the principles of the DNDW architecture. CIMLE provides a flexible environment so heterogeneous data sources, messaging frameworks, and analytic processes can interoperate to provide the specific information required for situation understanding and decision making. It was designed to support the creation of modular, distributed solutions over large monolithic systems. With CIMLE, users can repurpose individual analytics to address evolving decision-making requirements or to adapt to new mission contexts; CIMLE’s modular design simplifies integration with new host operating environments. CIMLE’s configurable system design enables model developers to build analytical systems that closely align with organizational structures and processes and support the organization’s information needs.

Taking a byte out of big data

The purpose of the work is to study the current problems and prospects of the solution for processing big data received or stored in the Internet (web data), as well as the possibility of practical realization of Data Mining technology for big web data on practical example. Materials and methods. The study included a review of bibliographic sources on big data analysis problems.Data Mining technology was used to analyze large web data, as well as computer modeling of a practical problem using the C # programming language and creating a DDL database structure for accumulating web data.Results. In the course of the work, the specifics of big data were described, the main characteristics of big data were highlighted, and modern approaches to processing big data were analyzed. A brief description of the horizontal-scalable architecture and the BI-solution architecture for big data processing is given. The problems of processing large web data are formulated: limiting the speed of access to data, providing access via network protocols through general-purpose networks.An example showing the approach to processing large web data was also implemented. Based on the idea of big data, the described complexities of web data processing and the methods of Data Mining, techniques were proposed for effectively solving the practical problem of processing and searching patterns in a large data array.The following classes have been developed in the C # programming language:Class of receiving web data via the Internet; Data conversion class;Intelligent data processing class;Created DDL script that creates a structure for the accumulation of web data.A single UML class diagram has been developed.The constructed system of data and classes allows to solve the main part of the problems of processing large web data and perform intelligent processing using Data Mining technology in order to solve the problem posed of identifying certain records in a large array. The combination of object-oriented approach, neural networks and BI-analysis to filter data will speed up the process of data processing and obtaining the result of the studyConclusion. According to the results of the study, it can be argued that the current state of technology for analyzing large web data allows you to efficiently process data objects, identify patterns, get hidden data and get full-fledged statistical data.The obtained results can be used both for the purpose of the initial study of big data processing technologies, and as a basis for developing an already real application for analyzing web data. The use of neural networks and the created universal classes-handlers makes the created architecture flexible and self-learning, and the class declarations and the base DDL structure will greatly simplify the development of program code.

There are two fossil records: the physical fossil record, which consists of specimens, and the abstracted fossil record, which is made up of data derived from those specimens. Mseum collections are the conduit between these two fossil records. Over the past several decades, the abstracted fossil record has provided many important insights about the major features of life’s history, but it has relied mostly on limited types of data (primarily taxonomic occurrence data) derived from ultimately finite literature sources. In contrast, specimen collections and modern tools for digitizing information about them present an opportunity to transform paleobiology into a “big data” science. Digitally capturing non-traditional (e.g., paleoecological, taphonomic, geochemical, and morphological) data from millions of specimens in museum collections and then integrating them with other unique big data resources has the potential to lead to the most important paleontological discoveries of the twenty-first century. What we know about the past record relied heavily on museum collections—the cumulation of centuries of investigation of the fossil record. The sample of past biodiversity will accumulate only with continued exploration of the fossil record … and restudy of existing collections…. —J. Sepkoski (1992, p. 80)

Background: Google Trends (GT) is an emerging source of data that can be used to predict, detect, and track infectious disease outbreaks. GT cumulative search volume data has been shown to correlate with cumulative case counts and to produce basic and observed reproduction number estimates analogous to those derived from more traditional epidemiological data sources. An outbreak of Hepatitis-E (Hep-E) occurred in Namibia in the fall and winter of 2017–2018. We used GT data to estimate transmission dynamics of the outbreak and compared these results with those estimated via data from HealthMap, a relatively new digital data source, and with surveillance reports from the government of Namibia published in the World Health Organization Bulletin, which is a traditional data source. Objective: Aim 1: To determine the correlation between GT relative search volume data (RSV) and cumulative case counts from the HealthMap (HM) and World Health Organization (WHO) data sources. Aim 2: To estimate and compare transmission dynamics including basic reproduction numbers (R0), observed reproduction numbers (Robs), and final outbreak size (Imax) for each of the three sources of data. Methods: GT relative search volume data regarding the term “hepatitis” in Namibia was acquired from October 13, 2017–March 2, 2018. Cumulative reported case counts were obtained from the HealthMap and WHO data sources. The Incidence Decay and Exponential Adjustment (IDEA) model was used to calculate R0, Robs, and final outbreak size for the three data sources. Results: The correlation coefficient between GT cumulative relative search volume and both HM and WHO cumulative case counts measured R = 0.93. The mean R0 and Robs estimates for the hepatitis-E outbreak in Namibia were similar between the GT, HM, and WHO data sources and are similar to previously published Hep-E R0 estimates from Uganda. Final outbreak size was similar between HM and WHO data sources; however, estimates using GT-derived data sources were smaller. Conclusions: GT cumulative search volume correlated with cumulative case counts from the HM and WHO data sources. Mean R0 and Robs values were similar among the data sources considered. GT-derived final outbreak size was smaller than both HM and WHO estimates due to diminishing search volume later in the epidemic possibly due to search fatigue; nevertheless, this data source was useful in describing the transmission dynamics of the outbreak including correlation with case counts and reproduction numbers.

The term “big data” is suddenly pervasive. The New York Times deemed this the “Age of Big Data” in a 2012 article, and a Google search for the term yields over 50 million hits. “Big data” is difficult to define precisely, but it is characterized by three attributes known as “the three V's”: its large volume, its variety, and its velocity, that is, the frequency with which it is generated. Medical big data is a particularly rich but sensitive type of big data, and it holds great promise as a resource for researchers and other analysts. Numerous medical big data initiatives are being launched by public and private enterprises. The transition from paper medical files to EHR systems has facilitated the creation of large health information databases. Computer processing of digitized records permits fast and relatively inexpensive data analysis and synthesis. These databases, therefore, can serve as invaluable research resources. Medical big data can be derived from other, nontraditional sources as well. Google retains all users’ web searches, including those involving medical queries. It can use the data for its own purposes, and at times, the data are requested by government and law enforcement authorities as well. Customer purchasing records, tweets, and Facebook entries can also reveal a great deal of health information. Companies such as Acxiom offer to sell “demographic, behavioral, lifestyle, financial and home data” that they obtain from such sources. This book, however, focuses on medical big data that is derived from electronic health records (EHRs) or from reports submitted by healthcare providers. It does not, therefore, extensively address nontraditional sources of medical big data. Analysts can access large-scale collections of EHR data in two primary ways. First, health information can be collected into large databases and deidentified to protect patient privacy. Such databases could be limited to particular hospital systems, be expanded to cover entire regions, or even be national in scope. In the alternative, researchers may use a federated system. A “federated network” can be defined as one that “links geographically and organizationally separate databases to allow a single query to pull information from multiple databases while maintaining the privacy and confidentiality of each database.”

Token-Based Adaptive Time-Series Prediction by Ensembling Linear and Non-linear Estimators: A Machine Learning Approach for Predictive Analytics on big Stock Data

Assessing the methodological quality of big data: an introduction to the Total Error framework The availability and use of big data sources is increasing exponentially. The variety of new and emerging data sources offers opportunities to complement, replace, improve or add to conventional data sources. Survey data are one kind of conventional data sources. In survey research, a framework to assess the accuracy of survey data already existed for quite some time. This framework is known as the Total Survey Error (TSE) framework. The philosophy behind this framework has only recently been universalized to (big) data in general in the form of the Total Error (TE) framework. This generic framework, which allows for assessing the accuracy of (big) data, is outlined in this article. Additionally, the TE framework is applied to big data sources that could be relevant for policing: police-registered crime data, Twitter data and mobile phone data.

Data is becoming the world's new natural resource and big data use grows quickly. The trend of computing technology is that everything is merged into the Internet and 'big data' are integrated to comprise complete information for collective intelligence. With the increasing size of big data, refining big data themselves to reduce data size while keeping critical data (or useful information) is a new approach direction. In this paper, we provide a novel data consumption model, which separates the consumption of data from the raw data, and thus enable cloud computing for big data applications. We define a new Data-as-a-Product (DaaP) concept, a data product is a small sized summary of the original data and can directly answer users' queries. Thus, we separate the mining of big data into two classes of processing modules: the refine modules to change raw big data into small sized data products, and application-oriented mining modules to discover desired knowledge further for applications from well-defined data products. Our practices of mining big stream data, including medical sensor stream data, streams of text data and trajectory data, demonstrated the efficiency and precision of our DaaP model for answering users' queries.

Big data deals with data of very large data size, heterogeneous data types and from different sources. The data is very complex in nature and having growing data. Dealing with big data is one of the emerging areas of research which is expanding at a rapid rate in all domains of engineering and medical sciences. A major challenge imposes on the analysis of big data is originated from big data generation source, which generate data with very fast speed with varying data distribution due to which the classical methods are unable to process big data. This paper discusses the characteristics, challenges, and issues with big data mining. It also illustrates the examples taken from various fields like medical, finance, social networking sites, stock exchange, etc. to realize the application and importance of big data mining. This paper explains about the use of parallel computing in data mining security issues and how to deal with them. Furthermore, this paper also discusses challenges associated with big streaming data with concept drifts.

The thesis concerns the use of big data in Official Statistics; the aim is to bring some new experimental studies into the big data literature. In particular, the purpose is to evaluate if and how big data could be used in Official Statistics. Only a few experiments exists on the use of big data for statistical purposes; it is a challenge task and a lot of experimentation is needed to find out evidence and solutions to use big data for statistical purposes. The analysis performed in the thesis goes into two different directions: 1. combining a traditional data source with a big data source to verify the potential of the latter to replicate official results; 2. analyzing a big data source per se and then trying to combine with an Official Statistics source to identify common patterns. The thesis initially proposes a literature review of definitions of big data and experiments, in particular concerning the use of the new sources combined with traditional data sources. Then, three original studies have been performed: the first two concern mobility in Lombardy region using mobile phone data. They both refer to the same issue (mobility patterns), but they differ in the traditional data source used: Origin/Destination matrix in the first case, an integrated version of the O/D matrix in the second. The objective of these two studies is trying to put in a unique interpretative framework one traditional statistical source and one typical kind of big data in order to evaluate some informative potentialities of this approach. In particular, we wanted to check if the two sources show common patterns, to evaluate future uses of the big data source in Official Statistics. The third study shows the pilot that was carried out during the traineeship I had the opportunity to attend at Eurostat, in collaboration with the Task Force Big Data. It concerns the use of Wikipedia, free online encyclopedia, for Tourism Statistics. The aim is to evaluate the use of Wikipedia page views as a source of information for the identification of factors that drive tourism to an area and whether it is possible to predict tourism flows using these data. A final chapter proposes conclusions and future remarks on the use of big data in Official Statistics. Two of the studies (the first on mobility patterns and the one on Wikipedia) have been or are being published, in a shorter and revised version. The three experiments show some potential in the use of big data in Official Statistics. The study needs more in-depth analysis, many more experiments and considerations will be necessary before we can achieve some definitive and convincing approaches.