Robust regression for non-randomly distributed big data with application to nonconvex penalized learning

Finding a new drug which is both safe and efficient is an expensive and time-consuming endeavour. In particular, establishing the ‘dose-effect relationship’ – how beneficial a drug is at different dosages – can be challenging. Predicting this curve requires gathering experimental data by exposing and recording how cells respond to various levels of the drug. However, extreme values are often observed at low and high dosages, potentially introducing errors that are hard to correct in the prediction process. Yet, these extreme observations are sometimes genuine so researchers cannot just ignore them. To improve dose-effect estimation, Zhou, Liu, Fang et al. developed a new general-purpose approach. It uses advanced statistical modelling to account for extremes in lab data. This strategy outperformed other methods when dealing with these observations while also providing higher efficiency in data analysis with more uniform data in experiments. To facilitate implementation, Zhou, Liu, Fang et al. set up a user-friendly tool baptized ‘REAP’; this free online resource allows scientists without advanced statistical experience to harness the new approach and to perform dose-effect analysis more easily and accurately. This could boost research across many different disciplines that examine the effects of chemicals on cells.

Unified distributed robust regression and variable selection framework for massive data

Distributed optimal subsampling for quantile regression with massive data

Massive amounts of building operational data are collected and stored in modern buildings, which provide rich information for in-depth investigation and assessment of actual building operational performance. However, the current utilization of big building operational data is far from being effective due to the gaps between building engineering and advanced big data analytics. Data mining is a promising technology for extracting previously unknown yet potentially useful insights from big data. This paper aims to explore the potential application of advanced data mining techniques for effective utilization of big building operational data. A case study of mining the operational data of an educational building for performance improvement is presented. Decision tree, clustering analysis, and association rule mining are adopted to analyze the operational data. The results show that useful knowledge can be extracted for identifying typical building operation patterns, detecting operation deficiencies, and spotting energy conservation opportunities. Practical application:The current utilization of big building operational data in the building industry is rather limited due to the lack in experience of using advanced big data analytics. This study presents a data mining-based method for analyzing massive building operational data. The case study results validate the efficiency and effectiveness of the method proposed. It can help building professionals to discover valuable insights into building operation patterns and thereby developing strategies for improving building energy efficiency. The method can be fully realized using the open-source software R, which provides great flexibilities in its integration with building automation systems.

As we are living in the era of big data, huge volumes of a wide variety of complex data-which can be of different levels of veracity-are generated or collected at a high velocity from rich sources of data in various real-life applications. A rich source of these big data sources is the Internet of Things (IoT), which include a collection of sensors, smartphones and other mobile devices, wearable devices, as well as other "things" that are capable to operate within the existing Internet infrastructure. Embedded in these big data are valuable knowledge and useful information. Hence, the research problem of data mining from big IoT data have drawn attention of many researchers as it aims to discover implicit, previously unknown and potentially useful information and knowledge from the data. For instance, frequent pattern mining finds sets of frequently co-occurring items in the IoT domains. Associative classification discovers rules revealing relationships among items within the frequent patterns and their associations with the corresponding class labels. Induction based classification uses decision tree or random forest to learn from old big IoT for classifying or making predictions on new data. Over the past quarter of a century, many serial, distributed, parallel, and MapReduce-based (Hadook-based and Spark-based) big data mining algorithms have been proposed. These algorithms are run in local computers, distributed and parallel environments, clusters, grids, clouds and/or data centers. In this paper, we review some of these algorithms, discuss issues and research prospective in mining classification patterns from these big IoT data in fog. Our case study on a real-life application shows the feasibility on classifying real-life big IoT data over fog for urban analytics.

Based on information asymmetry theory, principal‐agent theory, and risk management theory, this paper analyzes the mechanism of the impact of big data on financial decision‐making, analyzing four dimensions: how big data enhances the information base for forecasting, how big data improves the relevance of decision‐making, how big data builds new competitive advantages, and how big data promotes dynamic decision‐making. Secondly, through the analysis of specific implementation cases of enterprise big data in financial decision‐making, we focus on the real problems faced in management and the effect of solving problems through big data platform. The enterprise realizing the integration of business and finance not only can better lead business expansion, but also can improve the management level within the enterprise, which is conducive to the improvement of core competitiveness. The integration of industry and finance is essentially achieved through the application of various financial management modules to the business operations of enterprises, including budget management, capital management, fixed asset management, and financial accounting. If we want to implement the whole process of financial integration, it is impossible to achieve this manually, and we must first build a powerful information system as a guarantee. Under the guidance of theories of information asymmetry, stakeholders, and management information systems, Internet finance enterprises should build their own integrated business finance system based on the demand for business finance integration in the Internet finance industry, to enhance the matching of business finance data of Internet finance enterprises, improve the efficiency of enterprise management, and realize business finance integration. Finally, through the research of this paper, we hope to provide reference for other similar enterprises to apply big data for financial decision‐making. Through the application of big data, higher economic benefits are achieved in procurement management, production control, capital budget, and investment decision compared with the previous ones. It is concluded that in the era of big data, massive data can be used to serve enterprise decision‐making in depth, which can break the business and financial barriers, improve the efficiency and quality of decision‐making, optimize the organizational structure and personnel, and enhance the prediction and early warning capability. The application of big data tools has become the key to assisting financial decision‐making and enhancing enterprise value.

In the era of the construction engineering industry informatization, the collection, storage, and reuse of big construction data is the only way to apply and inform big data in the construction industry. Relying on the reverse logistics management system, modern construction technology can simulate the timeliness of materials, funds, and time limits of the actual construction site according to the logistics reverse engineering principle. Then, reverse the active management status of the internal logistics model of the actual project. It is one of the critical problems for construction engineering informatization to study applying big data and cloud computing technology to solve problems from massive and complicated data or mass data. Since the research on big data and cloud computing technology started late in construction engineering, it is worth studying which aspects of these technologies can be used in the whole life cycle of this field. This paper proposes a structural construction optimization identification method based on IPEM rapid reanalysis. USES the co-solubility of reverse logistics structure construction technology and applied the data mining method to solve many problems such as refined construction management leftover material surplus in the later construction stage. Finally, a construction optimization model of reverse logistics building system based on improved data mining is proposed according to the actual engineering requirements.

Ambient intelligence is an emerging platform that provides advances in sensors and sensor networks, pervasive computing, and artificial intelligence to capture the real time climate data. This result continuously generates several exabytes of unstructured sensor data and so it is often called big climate data. Nowadays, researchers are trying to use big climate data to monitor and predict the climate change and possible diseases. Traditional data processing techniques and tools are not capable of handling such huge amount of climate data. Hence, there is a need to develop advanced big data architecture for processing the real time climate data. The purpose of this paper is to propose a big data based surveillance system that analyzes spatial climate big data and performs continuous monitoring of correlation between climate change and Dengue. Proposed disease surveillance system has been implemented with the help of Apache Hadoop MapReduce and its supporting tools.

The application of big data in the field of ecological environment has achieved remarkable results, but there are still problems such as insufficient innovation and lack of shared service ability. This research takes the big data platform as the technical support and makes full use of advanced technologies such as internet of things, cloud computing, and big data to realize the integration of ecological environment big data resources and real-time data sharing. Through network technology, the ecological environment metadata can be automatically obtained from the internet and stored. When sharing ecological environment metadata, realize the data response and declassification of geospatial data for different user access levels, and finally safely feedback the ecological environment spatial big data integrated with the target ecological environment metadata and declassified geospatial data to the access initiator, so as to realize the centralized sharing of decentralized environmental information resources and give full play to the supporting role of big data, so that big data can be effectively applied in environmental monitoring. Find out the changes of pollution sources and pollutants through massive data, so as to provide necessary data support for the whole environmental monitoring.

Power grid operation and maintenance decision-making reform is an important part of power system reform. With the construction of massive historical quasi real-time data management platform, the reform of power system is also advancing. However, in the face of massive data explosion, the business level and business logic become disorganized and redundant. Based on the actual situation of Shenzhen Power Supply Bureau, the sg-erp data center is composed of structured data center, massive data center, unstructured data center and power grid GIS data center. With the unprecedented growth of business application data, the data center can improve business logic and promote power system reform. The experimental results show that big data technology has a broad application prospect in the reform of power industry.

With the construction and development of industrial informatization, industrial big data has become a trend within the smart industry. To obtain valuable information on massive data, achieving the acquisition, storage, analysis, and mining is becoming an important area of research. Focusing on the application requirements for industrial fields, we propose a data acquisition and analysis system based on the NB‐IoT for industrial applications. The system is an integrated system that includes sensor data acquisition, data transmission, data storage, and analysis mining. In this study, we mainly focused on the use of the NB‐IoT network to collect and transmit real‐time data for sensors. First, for the long time series (e.g., if we collect the data streams for one year for the sensor with a frequency of 1 Hz, the length of the series will reach 107). Then, we propose DSCS‐LTS, a distributed storage and calculation model, and CCCA‐LTS, an algorithm for the correlation coefficient of long time series in a distributed environment. Third, we propose a granularity selection algorithm and query process logic for visualization. We tested the platform in our laboratory and an automated production line for one year, and the experimental results using real data sets show that our approach is effective and scalable, can achieve efficient data management, and provide the basis for intelligent enterprise decision‐making.

The rapid development of ocean observation technology has resulted in the accumulation of a large amount of data and this is pushing ocean science towards being data-driven. Based on the types and distribution of oceanographic data, this paper analyzes the present and makes predictions for the future regarding the use of big and small data in ocean science. The ocean science has not fully entered the era of big data. There are two ways to expand the amount of oceanographic data to better understanding and management of the ocean. On the data level, fully exploit the potential value of big and small ocean data, and transform the limited, small data into rich, big data, will help to achieve this. On the application level, oceanographic data are of great value if realize the federation of the core data owners and the consumers. The oceanographic data will provide not only a reliable scientific basis for climate, ecological, disaster and other scientific research, but also provide an unprecedented rich source of information that can be used to make predictions of the future.

The management of big data is mainly affected by the size of the big graph data that represents the huge volumes of data. The size of this structure may increase with the size of data to be handled over the time. Facing this issue, the querying time may be affected and the introduced delay may not be tolerated by running applications. Moreover, the investigation of attacks through the collected massive data could not be ensured using traditional approaches, which do not support big data constraints. In this context, we propose in this paper, a novel temporal conceptual graph to represent the big data and to optimize the size of the derived graph. The proposed scheme built on this novel graph structure enables tracing back of attacks using big data. The efficiency of the proposed scheme for the reconstruction of attack scenarios is illustrated using a case study in addition to a conducted comparative analysis showing how smart big graph data is obtained through the optimization of the graph size.

Blasting vibration is one of the main negative effects of blasting construction. It is influenced by many external factors and has a complex nonlinear relationship with them. How to predict and reduce the blasting vibration effect while completing the blasting construction is a long-term concern in the blasting industry. With the development of the Internet of Things and mobile devices, various types of data collection in blasting projects have become quick and varied. Many massive, multi-source, heterogeneous blasting vibrations, and geological data have been gathered. Such a large-scale data set implies potentially useful information and presents significant challenge relevant to traditional computation theories and methods as well. Based on the massive data related to blasting vibration, this paper uses Hadoop to establish a distributed computing framework, establishes a big data analysis platform to carry out massive data mining and analysis, and builds a parallel SVM blasting vibration intelligent prediction model based on the Spark parallel computing framework. The blasting vibration velocity and frequency of blasting construction of a project in southwest are selected and compared with the calculated and measured values of the traditional Sadovski's formula to verify the intelligent blasting prediction model in the big data environment. The practice shows that blasting vibration intelligent prediction has high efficiency and higher accuracy than traditional models based on parallel SVM used in large data environment. In this paper, the parallel computing and data mining technologies of big data clusters are applied to intelligent prediction of blasting vibrations, changed the traditional random sampling and single mining analysis models, and extracted the useful information for management, decision-making and production from massive data, improved the efficiency and precision of the intelligent blasting prediction models.

The real value of Big Data lies in its hidden insights, but the current focus of the Big Data community is on the technologies for mining insights from massive data, rather than the data itself. The biggest challenge facing industries is not how to identify the right data, but instead, it is how to use insights obtained from Big Data to improve the business. To address this challenge, we propose GOMA, a goal-oriented modeling approach to Big Data analytics. Powered by Big Data insights, GOMA uses a goal-oriented approach to capture business goals, reason about business situations, and guide decision-making processes. GOMA provides a systematic approach for integrating two types of the resulting insight from data analytics to goal-oriented reasoning and decision-making processes: descriptive insights are the ones that describe the current state (e.g., the current customer retention rate) and predictive insights are the ones that predict likely future phenomena by inference from the data (e.g., customers who are likely to defect). To aid in the description and illustration of the GOMA approach, a retail banking churning scenario is used as a running example throughout this paper.