Related Work and Concepts

Almost all consumer and firm transactions are achieved using computers and as a result gives rise to increasingly large amounts of data available for analysts. The gold standard in Economic data manipulation techniques matured during a period of limited data access, and the new Large Data Analysis (LDA) paradigm we all face may quickly obfuscate most tools used by Economists. When coupled with an increased availability of numerous unstructured, multi-modal data sets, the impending 'data tsunami' could have serious detrimental effects for Economic forecasting, analysis, and research in general. Given this reality we propose a decision-aid framework for<i> <strong>Augmented-LDA (A-LDA) </strong></i>– a synergistic approach to LDA which combines traditional supervised, rule-based Machine Learning (ML) strategies to iteratively uncover hidden sources in large data, the artificial neural network (ANN) Unsupervised Learning (USL) at the minimum Helmholtz free energy for isothermal dynamic equilibrium strategies, and the Economic intuitions required to handle problems encountered when interpreting large amounts of Financial or Economic data. To make the ANN USL framework applicable to economics we define the temperature, entropy, and energy concepts in Economics from non-equilibrium molecular thermodynamics of Boltzmann viewpoint, as well as defining an information geometry, on which the ANN can operate using USL to reduce information saturation. An exemplar of such a system representation is given for firm industry equilibrium. We demonstrate the traditional ML methodology in the economics context and leverage firm financial data to explore a frontier concept known as behavioral heterogeneity. Behavioral heterogeneity on the firm level can be imagined as a firm's interactions with different types of Economic entities over time. These interactions could impose varying degrees of institutional constraints on a firm's business behavior. We specifically look at behavioral heterogeneity for firms that are operating with the label of 'Going-Concern' and firms labeled according to institutional influence they may be experiencing, such as constraints on firm hiring/spending while in a Bankruptcy or a Merger procedure. Uncovering invariant features, or <i><strong>behavioral data metrics </strong></i>from observable firm data in an economy can greatly benefit the FED, World Bank, etc. We find that the ML/LDA communities can benefit from Economic intuitions just as much as Economists can benefit from generic data exploration tools. The future of successful Economic data understanding, modeling, simulation, and visualization can be amplified by new A-LDA models and approaches for new and analogous models of Economic system dynamics. The potential benefits of improved economic data analysis and real time decision aid tools are numerous for researchers, analysts, and federal agencies who all deal with increasingly large amounts of complex data to support their decision making.

Big data analysis technology can be used to transform massive data of oil production engineering into knowledge that can be used to guide oilfield production. Due to the lack of suitable large data analysis tools, a large data analysis software system was developed to simplify the workload of data analysis. This paper illustrates the research and application of large energy consumption data analysis through the development of big data analysis methods, process improvement and software platform, then achieves the goal of reducing costs and improving efficiency.

A programming system has been implemented in which annotated Petri nets are used as machine-processable high-evel design representations. The nets can be used to express the parallelism and the dynamic sequential dependencies found in complex software. They can then be interactively fired to facilitate debugging of the design. The nets are processed into a procedure language, called XL/1, to which a variety of transformations are applied in order to produce more efficient programs. These programs are generated for either a serial or a parallel processing environment. Finally, the XL/1 programs may be translated into PL/I or PL/S. The serial processing versions have been compiled and run successfully, but the parallel processing versions have not yet been run in a parallel processing environment.

Big Data is driving radical changes in traditional data analysis platforms. To perform any kind of analysis on such voluminous and complex data, scaling up the hardware platforms becomes impending. With the entire buzz surrounding Big Data; it is being collected at an unprecedented scale. Big Data has potential to revolutionize much more than just research. Loading large data-sets is often a challenge. Another shift of this Big Data processing is the move towards cloud computing. As many communities begin to rely on cloud based data management, large shared data goes up extensively. Analysis of such large data on distributed processing system or cloud is a bit difficult task to handle. The aim of this chapter is to provide a better understanding of the design challenges of cloud computing and analytics of big data on it. The challenge is related to how a large extent of data is being harnessed, and the opportunity is related to how effectively it is used for analyzing the information from it.

Family-based association analysis unconditional on parental genotypes models the effects of observed genotypes. This approach has been shown to have greater power than conditional methods. In this chapter, we review popular association analysis methods accounting for familial correlations: the marginal model using generalized estimating equations (GEE), the mixed model with a polygenic random component, and genome-wide association analyses. The marginal approach does not explicitly model familial correlations but uses the information to improve the efficiency of parameter estimates. This model, using GEE, is useful when the correlation structure is not of interest; the correlations are treated as nuisance parameters. In the mixed model, familial correlations are modeled as random effects, e.g., the polygenic inheritance model accounts for correlations originating from shared genomic components within a family. These unconditional methods provide a flexible modeling framework for general pedigree data to accommodate traits with various distributions and many types of covariate effects. Genome-wide association studies usually test more than 10,000 SNPs and thus traditional statistical methods accounting for the familial correlations often suffer from a computational burden. Multiple approaches that have been recently proposed to avoid this computational issue are reviewed. The single-marker analysis procedures are demonstrated using the R package gee and the ASSOC program in the S.A.G.E. package, including how to prepare input data, conduct the analysis, and interpret the output. ASSOC allows models to include random components of additional familial correlations that may be not sufficiently explained by a polygenic effect and addresses nonnormality of response variables by transformation methods. With its ease of use, ASSOC provides a useful tool for association analysis of large pedigree data.

Nowadays, MapReduce has become an effective tool for large scale data analysis. It is naturally designed for group-by aggregation tasks rather than join operator which is common in real analysis works. The existing join methods in MapReduce may earn different performances in different cases, which makes how to choose a good join plan from a join list difficult. The current static optimization can't generate an efficient evaluation plan for a given join list. In this paper, we will introduce some custom join technologies and then propose an adaptive join plan generator for multiple join depending on both rule-based model and cost-based model considering the intermediate data.

Recent progress in the algorithm as well as the processor power have made the dynamics simulation of complex kinematic chains more realistic in various fields such as human motion simulation and molecular dynamics. The computation can be further accelerated by employing parallel processing on multiple processors. In fact, parallel processing environment is becoming more affordable thanks to recent release of multiplecore processors. Although several parallel algorithms for the forward dynamics computation have been proposed in literature, there still remains the problem of automatic scheduling, or load distribution, for handling arbitrary kinematic chains on a given parallel processing environment. In this paper, we propose a method for finding the schedule that minimizes the computation time. We test the method using three human character models with different complexities and show that parallel processing on two processors reduces the computation time by 35–36%.

The purpose of the workis to explore the current problems and prospects of mining solution, big web data in real time, as well as the possibility of practical implementation of Web Mining technology for big web data on a practical example.Materials and methods.The study included a review of bibliographic sources on big data mining. We used Web Mining technology for associative analysis of large web data, as well as computer modeling of the practical task of transaction analysis using a general-purpose scripting language (PHP).Results.During the work, the specifics of the Data Mining technology were described, and a modern approach to the analysis of large web data –Web Mining was analyzed. A brief classification of tasks solved using Web Mining technology is given. The problem of data mining of large web data in a general-purpose scripting language (PHP) has been solved: the lack of libraries for data mining, the difficult normalization of data to the form necessary for associative analysis, interaction with the database management system. Also, an example showing an approach to the mining of large web data was implemented. Based on the understanding of Web Mining technology and the described difficulties of analyzing web data in the PHP language, methods for effectively solving the practical problem of analyzing web data based on transactions committed in a dynamic web application have been proposed. A module for associative analysis of customer transactions in the programming language PHP was developed. The module includes an intelligent data processing class. The structural scheme of the module and system architecture were developed. The constructed module allows us to solve the main part of the problem of associative analysis of large web data using Web Mining technology in order to solve the problem of identifying patterns in a large array of web data. Associative analysis of web data is much faster because of the combination of a general-purpose scripting language and an object-oriented approach.Conclusion.According to the results of the study, it can be argued that the current state of the technology for the analysis of large web data allows efficiently process data objects, identify patterns, obtain hidden data and receive complete statistical data in real time. The results can be used both for the purpose of the initial research of technologies for analyzing large web data, and as an addition to the content management system for the intelligent analysis of web data. The usage of the technology of associative analysis and the created universal handler class makes the created module flexible, while the possibility of manual integration makes this module universal. With manual integration, the database management system is not important. Algorithm methods work with selected data. This factor greatly simplifies the further development of program code.

InterMine is a data integration warehouse and analysis software system developed for large and complex biological data sets. Designed for integrative analysis, it can be accessed through a user-friendly web interface. For bioinformaticians, extensive web services as well as programming interfaces for most common scripting languages support access to all features. The web interface includes a useful identifier look-up system, and both simple and sophisticated search options. Interactive results tables enable exploration, and data can be filtered, summarized, and browsed. A set of graphical analysis tools provide a rich environment for data exploration including statistical enrichment of sets of genes or other entities. InterMine databases have been developed for the major model organisms, budding yeast, nematode worm, fruit fly, zebrafish, mouse, and rat together with a newly developed human database. Here, we describe how this has facilitated interoperation and development of cross-organism analysis tools and reports. InterMine as a data exploration and analysis tool is also described. All the InterMine-based systems described in this article are resources freely available to the scientific community.

An important task in knowledge discovery is feature selection. This paper describes a practical approach to feature subset selection proposed as part of a hybrid rough sets/neural network framework for knowledge discovery for decision support. In this framework neural networks and rough sets are combined and used cooperatively during the system life cycle. The reason for combining rough sets with neural networks in the proposed framework is twofold. Firstly, rough sets based systems provide domain knowledge expressed in the form of If-then rules as well as tools for data analysis. Secondly, rough sets are used in this framework in the task of feature selection for neural network models. This paper examines the feature selection aspect of the framework. An empirical study that tested the approach on artificial datasets and real-world datasets was carried out. Experimental results indicate that the proposed approach can improve the performance of neural network models. The framework was also applied in the development of a real-world decision support system. The experience with this application has shown that the approach can support the users in the task of feature selection.

Rough sets serves as a tool for data analysis and knowledge discovery from data databases. Attribute reduction is a basic issue in knowledge representation and data mining. This paper deals with distribution reduction in an inconsistent interval ordered information systems. The distribution reduction and maximum distribution reduction are proposed in inconsistent interval ordered information systems. Moreover, properties and relationship between them are discussed. Furthermore, judgement theorem and discernibility matrix are obtained, from which approaches to distribution reductions can be provided in inconsistent interval ordered information systems.

Exposure to environmental risk factors as well as weather conditions are known to have negative effects on health. Until recently, there was little a society could do for an individual at risk, other than provide general warnings when the concentration of pollutants or weather conditions deviate from the norm. Similarly, the assessment of individuals’ exposure over time has been confined to population and geographic averages, rather than individualized estimates. Recent advances in sensors and mobile technology have enabled real-time measurements of environmental variables and, at the same time, provided information about the spatio-temporal behavior of individuals. This can dramatically change the way health and wellness are assessed as well as how care and treatment are delivered. This paper presents a system framework called “Smart and Connected Health Alert System (SCHAS)” for individual-level environmental exposure in an attempt to better understand the relationships among exposures, symptoms and human health conditions. We demonstrate user interface, data acquisition and visual evaluation tools for large mobile sensor data analysis.

Deconvolving distribution of relaxation times, resistances and inductance from electrochemical impedance spectroscopy via statistical model selection: Exploiting structural-sparsity regularization and data-driven parameter tuning

Enhancing our knowledge of the complexities of cities in order to empower ourselves to make more informed decisions has always been a challenge for urban research. Recent developments in large-scale computing, together with the new techniques and automated tools for data collection and analysis are opening up promising opportunities for addressing this problem. The main motivation that served as the driving force behind this research is how these developments may contribute to urban data analysis. On this basis, the thesis focuses on urban data analysis in order to search for findings that can enhance our knowledge of urban environments, using the generic process of knowledge discovery using data mining. A knowledge discovery process based on data mining is a fully automated or semi-automated process which involves the application of computational tools and techniques to explore the “previously unknown, and potentially useful information” (Witten a Frank, 2005) hidden in large and often complex and multi-dimensional databases. This information can be obtained in the form of correlations amongst variables, data groupings (classes and clusters) or more complex hypotheses (probabilistic rules of co-occurrence, performance vectors of prediction models etc.). This research targets researchers and practitioners working in the field of urban studies who are interested in quantitative/computational approaches to urban data analysis and specifically aims to engage the interest of architects, urban designers and planners who do not have a background in statistics or in using data mining methods in their work.

Guest Editorial: Big data-driven theory building: Philosophies, guiding principles, and common traps