Multi-thread performance on a single thread in-memory database

Resting state functional magnetic resonance imaging (rfMRI) can be used to measure functional connectivity and then identify brain networks and related brain disorders and diseases. To explore these complex networks, however, huge amounts of data are necessary. Recent advances in neuroimaging technologies, and the unique methodological approach of rfMRI, have enabled us to an era of Biomedical Big Data. The recent progress of big data sharing projects with their challenges are discussed. This increasing amount of neuroimaging data has greatly increased the importance of developing preprocessing pipelines and advanced analytic techniques, which are better at handling large-scale datasets. Before applying any analysis method on rfMRI data, several preprocessing steps need to be applied to reduce all unwanted effects. Three alternative ways to get access to big preprocessed rfMRI data are presented involving the minimal preprocessing pipelines. There are several commonly used methods to examine functional connectivity. However, they become limited in the analysis of big data, and a new tool to explore such data is necessary. We propose a number of novel methods rooted in algebraic topology and collectively referred to as Topological Data Analysis to rfMRI functional connectivity. Their properties for big data analysis are also discussed.

Challenges and Techniques for Testing of Big Data

Ambient intelligence is an emerging platform that provides advances in sensors and sensor networks, pervasive computing, and artificial intelligence to capture the real time climate data. This result continuously generates several exabytes of unstructured sensor data and so it is often called big climate data. Nowadays, researchers are trying to use big climate data to monitor and predict the climate change and possible diseases. Traditional data processing techniques and tools are not capable of handling such huge amount of climate data. Hence, there is a need to develop advanced big data architecture for processing the real time climate data. The purpose of this paper is to propose a big data based surveillance system that analyzes spatial climate big data and performs continuous monitoring of correlation between climate change and Dengue. Proposed disease surveillance system has been implemented with the help of Apache Hadoop MapReduce and its supporting tools.

PhilDB is an open-source time series database that supports storage of time series datasets that are dynamic, that is it records updates to existing values in a log as they occur. PhilDB eases loading of data for the user by utilising an intelligent data write method. It preserves existing values during updates and abstracts the update complexity required to achieve logging of data value changes. It implements fast reads to make it practical to select data for analysis. Recent open-source systems have been developed to indefinitely store long-period high-resolution time series data without change logging. Unfortunately such systems generally require a large initial installation investment before use because they are designed to operate over a cluster of servers to achieve high-performance writing of static data in real time. In essence, they have a 'big data' approach to storage and access. Other open-source projects for handling time series data that avoid the 'big data' approach are also relatively new and are complex or incomplete. None of these systems gracefully handle revision of existing data while tracking values that changed. Unlike 'big data' solutions, PhilDB has been designed for single machine deployment on commodity hardware, reducing the barrier to deployment. PhilDB takes a unique approach to meta-data tracking; optional attribute attachment. This facilitates scaling the complexities of storing a wide variety of data. That is, it allows time series data to be loaded as time series instances with minimal initial meta-data, yet additional attributes can be created and attached to differentiate the time series instances when a wider variety of data is needed. PhilDB was written in Python, leveraging existing libraries. While some existing systems come close to meeting the needs PhilDB addresses, none cover all the needs at once. PhilDB was written to fill this gap in existing solutions. This paper explores existing time series database solutions, discusses the motivation for PhilDB, describes the architecture and philosophy of the PhilDB software, and performs a simple evaluation between InfluxDB, PhilDB, and SciDB.

This report examines the potential for Big Data and analytics in the upstream and midstream sectors of the oil and gas industry. IDC Energy Insights provides insight on areas where Big Data and analytics has the potential to provide business value. This report covers newer technologies, such as Hadoop/MapReduce and streaming analytics, rather than the more traditional high-performance computing. The survey data is provided from oil and gas respondents based in the United States. Guidance is provided to CIOs, enterprise architects, IT strategists, and business executives on the potential of this new technology for the oil and gas industry. Just like other industries, the oil and gas industry is experiencing an increase in the number of sensing devices in the oil fields or pipelines. Of course, the industry has always been plagued with too much data and the inability to make productive use of that data. What has changed in the past few years is the availability of new technology based on commodity hardware – Big Data and analytics – pioneered by firms like Facebook and Google. This new technology can process high volumes of various types of data at relatively quicker speeds than conventional technology. Now, oil and gas CIOs are considering whether to invest in a Big Data and analytics infrastructure. This report reviews the current attitudes toward Big Data and analytics in the oil and gas industry and explores potential applications of the technology with an eye toward speeding time to first oil, reducing costs and increasing the safety of drilling, and enhancing production.

The production process of modern enterprises gives out a huge amount of data with increasing potential security risks. As new technology and way of thinking, big data has the most basic application in the process of safety, which is to analyze the potential laws of safety from the data generated by the massive production activities, and then eliminate the potential hazards for effectively curbing the occurrence of accidents. This paper analyzes the relationship between operational risk and big data and solves the basic theoretical problems of how to apply big data in production safety based on the relevant theories and characteristics of big data. On this basis, the paper studies the operational risk situation model based on big data, analyzes the operational risk factors of different dimensions and granularity by using the multidimensional model analysis method, and studies the behavior in the production operation.

Risk monitoring model of intelligent agriculture Internet of Things based on big data

PHP136 - Big Data In Healthcare - Opportunities And Challenges

Java has proven to be an indispensable part of the big data infrastructure with its strength, scalability and large ecosystem. The huge amount of information associated with big data needs advanced technologies and architecture to be captured, stored, and analyzed. Traditional computer models have a hard time handling such huge amounts of data, especially when it comes to speed, scalability, and management. Because of its maturity and independence from specific platforms, Java is well-suited for building distributed data processing systems with exceptional performance. Java is a popular language for big data analytics, and this article delves into its history, current frameworks, and optimizations at the JVM level that promote efficient use of resources and horizontal scalability. In addition to addressing data accuracy, scalability, and security as major issues with big data frameworks, the study also discusses potential solutions, such as incorporating machine learning, cloud-native frameworks, and containerization systems like Kubernetes and Docker. The results indicate that Java cannot be replaced in handling complicated processing applications and big data, which strengthens its use as a core technology in fueling innovations in data-oriented sectors.

An Analysis on Future Prospects of Big Data in Healthcare

With the rapid development of cloud computing and mobile Internet technology, huge amounts of data are generated on the network at every moment. In the era of big data, most of the data are designed in distributed storage, stored on many independent sites. The typical data mining algorithm can not meet the requirement of privacy data security in the big data background. In the big data background, distributed data mining is faced with the need to solve data security, privacy disclosure and other technical problems. Therefore, this paper applies homomorphic encryption technology to typical K-means clustering mining algorithm, and designs a distributed privacy protection PP-kmeans algorithm based on big data. Experiments show that this algorithm can realize data privacy security protection. And more accurate big data clustering mining effect is achieved.

Big Data Analytics (BDA) is becoming a research focus in transportation systems, which can be seen from many projects within the world. By using sensor and Internet of Things (IoT) technology in transportation system, huge amount of data is been generated from different sources. This data can be integrated, analyzed and visualized for efficient and effective decision-making for maintaining transportation systems. The key challenges that exist in managing Big Data are the designing of the systems, which would be able to handle huge amount of data efficiently and effectively and to filter the most significant information from all the collected data. This chapter will draw attention towards the present scenario and future projections of big data in transportation systems. It also presents big data tools and techniques and then presents one brief case study of BDA in each type of transportation system. In this chapter, a broad overview of Big Data definitions, its history, present, and future prospects are briefed. Several tools and technologies especially for transportation are pointed out for maintaining transportation systems. At the end of the chapter, a definitive case studies on each transportation area is demonstrated.

Nowadays, Next Generation Sequeencing (NGS) is a catch-all term used to describe different modern DNA sequencing applications that produce big genomics data that can be analysed in a faster fashion than in the past. For this reason, NGS requires more and more sophisticated algorithms and high-performance parallel processing systems able to analyse and extract knowledge from a huge amount of genomics and molecular data. In this context, researchers are beginning to look at emerging deep learning algorithms able to perform efficient big data analytics. In this paper, we analyse and classify the major current deep learning solutions that allow biotechnology researchers to perform big genomics data analytics. Moreover, by means of a taxonomic analysis, we provide a clear picture of the current state of the art also discussing future challenges.

Nowadays, a huge quantity of data are produced by means of multiple data sources. The existing tools and techniques are not capable of handling such voluminous data produced from a variety of sources. This continuous and varied generation of data requires advanced technologies for processing and storage, which seems to be a big challenge for data scientists. Some research studies are well defined in the area of streaming in big data. Streaming data are the real-time data or data in motion such as stock market data, sensor data, GPS data and twitter data. In stream processing, the data are not stored in databases instead it is processed and analyzed on the fly to get the value as soon as they are generated. There are a number of streaming frameworks proposed till date for big data applications that are used to pile up, evaluate and process the data that are generated and captured continuously. In this chapter, we provide an in-depth summary of various big data streaming approaches like Apache Storm, Apache Hive and Apache Samza. We also presented a comparative study regarding these streaming platforms.

The increasing availability of Earth Observation and geographic data implies great opportunities for those capable to efficiently address the problems arising from the management of this huge amount of data. An efficient management of these data means to respond to the paradigm of the 4 V that usually applies to the problem of the Big Data: Volume - the sheer size of the “data at rest”, Velocity - the speed of new data arriving, Variety - the different manifold, and Veracity - trustworthiness and issues of provenance. The problem of Big Data is not only in the geospatial realm of Earth observation data, but in general in all the location-based data which basically are the main contributors to the deluge of big data. In addition to the need of managing (store, search & find when needed) these data efficiently, the problem arises from the analysis of these data. They need to be quickly processed in order to quickly extract the information content, then they must be analysed in conjunction with other data sources in order to express their real value in the construction of new knowledge. These processes are hastened by the advent of an increasing machine-to-machine communication. The automation of the data analysis requires standardized and linked data so that they can be processed by machines without human intervention. The problem with the standardization of geospatial data is solved by simply observing not only the best practices shared at European level, but mainly the regulatory scenario dictated by the INSPIRE Directive . The publication of spatial data as Linked Open Data may then leverage the reuse of common ontologies and vocabularies that allow the connection of geospatial data with other heterogeneous information. This way new scenarios and business opportunities may arise, as in the case of the real estate market that is mentioned in this article. This contribution aims to identify some business opportunities, related to Linked Open Data and arising from the imminent availability of the Sentinel satellite data, with the European program Copernicus, for companies operating in the so-called downstream services of Earth observation.