Big Data challenges, techniques, technologies, and applications and how deep learning can be used

8 - Deep learning techniques for optimizing medical big data

At this time big data, with large data volumes that continue to grow and become complex, is a challenge and opportunity for governments and organizations. This is said to be a challenge considering that the increase in data volume will produce a lot and variety of information, thus requiring efficient ways of storing and managing data. Big data is said to be an opportunity considering that the magnitude of the study results and proper information management will open up new opportunities to recognize the potential for improving services in various government and organizational sectors. Big data can open up opportunities to increase knowledge about various aspects of people's lives so that it can encourage the creation of new trends and innovations in various fields, improve and optimize operations, and determine future conditions so that governments and organizations can produce appropriate basic products. In the future, big data will change the way governments and organizations run their operations, creating challenges and opportunities for decision-makers. Even so, in the future big data can be utilized more optimally for the benefit of the wider community. In the future, big data has great potential to be used in many sectors to improve people's welfare. By using the right methods, big data can be very valuable in improving the welfare and quality of life of the wider community. Therefore, to be able to implement big data to improve the welfare and quality of life of the wider community, it is necessary to understand the challenges and opportunities of big data in the development of digital technology.

In today’s digital environment, the pattern is everywhere. It is present in many aspects of our daily life. Algorithms can be used to detect or mathematically observe a pattern physically. Vector feature values represent a pattern in the digital world. With the advent of artificial intelligence (AI) techniques in the recent era, there are so many machine learning (ML) and deep learning (DL) models that have been developed. ML is the branch of AI, which can perform tasks like data analysis, analytical model building, and classification without being explicitly programmed. DL is the subset of ML in AI. In DL, mostly artificial neural networks (ANNs) are utilized. It has the capability of work based upon unsupervised learning from the data that is unlabeled and unstructured. Using these DL and ML models, extract the meaningful features from the given image or video is known as pattern recognition (PR). PR is used in many engineering applications such as computer vision, natural language processing, character recognition, robotics, speech recognition, and so on. It is also used in many medical image processing applications and telemedicine. The present work discussed the pattern recognition problem and its various stages in detail. In addition to that, the application of deep learning and machine learning in pattern recognition is also explained briefly.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

The Internet of Things (IoT) is an ecosystem comprised of multiple devices and connections, a large number of users, and a massive amount of data. Deep learning is especially suited for these scenarios due to its appropriateness for "big data" difficulties and future concerns. Nonetheless, guaranteeing security and privacy has emerged as a critical challenge for IoT administration. In many recent cases, deep learning algorithms have proven to be increasingly efficient in performing security assessments for IoT devices without resorting to handcrafted rules. This research work integrates principal component analysis (PCA) for feature extraction with superior performance. Besides, the primary objective of this research work is to gather a comprehensive survey data on the types of IoT deployments, along with security and privacy challenges with good recognition rate. The deep learning method is performed through PCA feature extraction for improving the accuracy of the process. Our other primary goal in this study paper is to achieve a high recognition rate for IoT based image recognition. The CNN approach was trained and evaluated on the IoT image dataset for performance evaluation using multiple methodologies. The initial step would be to investigate the application of deep learning for IoT image acquisition. Additionally, when it comes to IoT image registering, the usefulness of the deep learning method has been evaluated for increasing the appropriateness of image recognition with good testing accuracy. The research discoveries on the application of deep learning in the Internet of Things (IoT) system are summarized in an image-based identification method that introduces a variety of appropriate criteria.

As the speed of information growth exceeds in this new century, excessive data is making great troubles to human beings. However, there are so much potential and highly useful values hidden in the huge volume of data. Big Data has drawn huge attention from researchers in information sciences, policy and decision makers in governments and enterprises. Data analytic is the science of examining raw data with the purpose of drawing conclusions about that information. Data analytics is about discovering knowledge from large volumes data and applying it to the business. Machine learning is ideal for exploiting the opportunities hidden in big data. This chapter able to discover and display the patterns buried in the data using machine learning.

Using big marine data to train deep learning models is not efficient, or sometimes even possible, on local computers. In this paper, we show how distributed learning in the cloud can help more efficiently process big data and train more accurate deep learning models. In addition, marine big data is usually communicated over wired networks, which if possible to deploy in the first place, are costly to maintain. Therefore, wireless communications dominantly conducted by acoustic waves in underwater sensor networks, may be considered. However, wireless communication is not feasible for big marine data due to the narrow frequency bandwidth of acoustic waves and the ambient noise. To address this problem, we propose an optimized deep learning design for low-energy and real-time image processing at the underwater edge. This leads to trading the need to transmit the large image data, for transmitting only the low-volume results that can be sent over wireless sensor networks. To demonstrate the benefits of our approaches in a real-world application, we perform fish segmentation in underwater videos and draw comparisons against conventional techniques. We show that, when underwater captured images are processed at the collection edge, 4 times speedup can be achieved compared to using a landside server. Furthermore, we demonstrate that deploying a compressed DNN at the edge can save 60% of power compared to a full DNN model. These results promise improved applications of affordable deep learning in underwater exploration, monitoring, navigation, tracking, disaster prevention, and scientific data collection projects.

Nowadays healthcare sector grows tremendously in last few decades. Insight of how we can uncover additional value from the data generated by healthcare and government. Large amount of heterogeneous data is generated by the agencies; this includes patient’s previous medical data, laboratory test values, and current treatment given to patient, doctor’s prescription, and diagnostic reports. However, the complex distributed and highly interdisciplinary nature of medical data has underscored the limitations of traditional data analysis capabilities of data accessing, storage, processing, analyzing, distributing, and sharing. In this paper we will discuss future trends of data mining that are used for analysis and prediction of big data. And also we review the challenges of manipulating large-scale next-generation sequencing (NGS) data and diverse clinical data derived from the EHRs for genomic medicine. Here introduce solutions for different challenges in manipulating, managing, and analyzing genomic and clinical data. Then finally data is analyzed using Apache Mahout for faster query access. These issues include Big Data benefits, its applications and opportunities in medical areas and health care. Methods and technology progress about Big Data are presented in this study.

Identification of outliers is a big challenge in big data although it has drawn a great deal of attention in recent years. Among all big data problems, the detection of outliers in gene expression data warrants extra attention because of its inherent complexity. Although a variety of outlier detection methods are available in the literature, Tomlins et al. (Tomlins et al. Science 310:644–648, 2005) argued that traditional analytical methods, for example, a two-sample t-statistic, which search for common activation of genes across a class of cancer samples, will fail to detect cancer genes, which show differential expression in a subset of cancer samples or cancer outliers. They developed the cancer outlier profile analysis (COPA) method to detect cancer genes and outliers. Inspired by the COPA statistic, some authors have proposed other methods for detecting cancer-related genes with cancer outlier profiles in the framework of multiple testing (Tibshirani and Hastie Tibshirani and Hastie Biostatistics 8:2–8, 2007; Wu Wu Biostatistics 8:566–575, 2007; Lian Lian Biostatistics 9:411–418, 2008; Wang and Rekaya Wang and Rekaya Biomarker Insights 5:69–78, 2010). Such cancer outlier analyses are affected by many problems especially if there is an outlier in the dataset then classical measures of location and scale are seriously affected. So the test statistic using these parameters might not be appropriate to detect outliers. In this study, we try to robustify one existing method. We propose a new technique called expressed robust t-statistic (ERT) for the identification of outliers. The usefulness of the proposed methods is then investigated through a Monte Carlo simulation.

With the development of information and technology, especially with the boom in big data, healthcare support systems are becoming much better. Patient data can be collected, retrieved, and stored in real time. These data are valuable and meaningful for monitoring, diagnosing, and further applications in data analysis and decision-making. Essentially, the data can be divided into three types, namely, statistical, image-based, and sequential data. Each type has a different method of retrieval, processing, and deployment. Additionally, the application of machine learning (ML) and deep learning (DL) in healthcare support systems is growing more rapidly than ever. Numerous high-performance architectures are proposed to optimize decision-making. As reliability and stability are the most important factors in the healthcare support system, enhancing the predicted performance and maintaining the stability of the model are always the top priority. The main idea of our study comes from ensemble techniques. Numerous studies and data science competitions show that by combining several weak models into one, ensemble models can attain outstanding performance and reliability. We propose three deep ensemble learning (DEL) approaches, each with stable and reliable performance, that are workable on the above-mentioned data types. These are deep-stacked generalization ensemble learning, gradient deep learning boosting, and deep aggregation learning. The experiment results show that our proposed approaches achieve more vigorous and reliable performance than traditional ML and DL techniques on statistical, image-based, and sequential benchmark datasets. In particular, on the Heart Disease UCI dataset, representing the statistical type, the gradient deep learning boosting approach dominates the others with accuracy, recall, F1-score, Matthews correlation coefficient, and area under the curve values of 0.87, 0.81, 0.83, 0.73, and 0.91, respectively. On the X-ray dataset, representing the image-based type, the deep aggregation learning approach shows the highest performance with values of 0.91, 0.97, 0.93, 0.80, and 0.94, respectively. On the Depresjon dataset, representing the sequence type, the deep-stacked generalization ensemble learning approach outperforms the others with values of 0.91, 0.84, 0.86, 0.8, and 0.94, respectively. Overall, we conclude that applying DL models using our proposed approaches is a promising method for the healthcare support system to enhance prediction and diagnosis performance. Furthermore, our study reveals that these approaches are flexible and easy to apply to achieve optimal performance.

The increasing availability of Earth Observation and geographic data implies great opportunities for those capable to efficiently address the problems arising from the management of this huge amount of data. An efficient management of these data means to respond to the paradigm of the 4 V that usually applies to the problem of the Big Data: Volume - the sheer size of the “data at rest”, Velocity - the speed of new data arriving, Variety - the different manifold, and Veracity - trustworthiness and issues of provenance. The problem of Big Data is not only in the geospatial realm of Earth observation data, but in general in all the location-based data which basically are the main contributors to the deluge of big data. In addition to the need of managing (store, search & find when needed) these data efficiently, the problem arises from the analysis of these data. They need to be quickly processed in order to quickly extract the information content, then they must be analysed in conjunction with other data sources in order to express their real value in the construction of new knowledge. These processes are hastened by the advent of an increasing machine-to-machine communication. The automation of the data analysis requires standardized and linked data so that they can be processed by machines without human intervention. The problem with the standardization of geospatial data is solved by simply observing not only the best practices shared at European level, but mainly the regulatory scenario dictated by the INSPIRE Directive . The publication of spatial data as Linked Open Data may then leverage the reuse of common ontologies and vocabularies that allow the connection of geospatial data with other heterogeneous information. This way new scenarios and business opportunities may arise, as in the case of the real estate market that is mentioned in this article. This contribution aims to identify some business opportunities, related to Linked Open Data and arising from the imminent availability of the Sentinel satellite data, with the European program Copernicus, for companies operating in the so-called downstream services of Earth observation.

With incoming data era from early 2010's, the word “Big Data” has been taken focus in many of research and development areas. The final purpose of Big Data research is the efficient utilization of useful information; however, the process involves a lot of complex subjects like data processing, data mining, and etc. In this paper, we summarize and discuss the subjects in area of Big Data analysis, for a clearer understanding of scientific challenges in Big Data researches.

Nowadays a lot of massive data is stored and typically, the data itself contains a lot of non-trivial but useful information. Data mining techniques can be used to discover this information which can help the companies for decision-making. However, in real life applications, data is massive and is stored over distributed sites. One of my major research topics is to protect privacy over this kind of data. Previously, the important characteristics, issues and challenges related to management of the large amount of data has been explored. Various open source data analytics frameworks that deal with large amount of Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. Digital universe is flooded with huge amount of data generated by number of users worldwide. These data are of diverse in nature, come from various sources and in many forms. To keep with the desire to store and analyze ever larger volumes of complex data, relational databases vendors have delivered specialized analytical platforms that come in many shapes and sizes from software only to analytical services that run in third party hosted environments. In addition new technologies have emerged to address exploding volumes of complex data, including web traffic, social media content and machine generated data including sensor data, global positioning system data.Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data has become a prominent research field, especially when it comes to decision making and data analysis. However, Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges. Since Big data is a recent upcoming technology in the market which can bring huge benefits to the business organizations, it becomes necessary that various challenges and issues associated in bringing and adapting to this technology are brought into light. Another challenge is that data collection may not have enough accuracy which will lead to a non-consistent analysis that can critically affect the decision based on this analysis. Moreover, it is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is; current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Since data is growing at a huge speed making it difficult to handle such large amount of data (Exabyte). The main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties.If data is to be used to make accurate decisions in time it becomes necessary that it should be available in accurate, complete and timely manner. This makes the data management and governance process bit complex adding the necessity to make Data open and make it available to government agencies in standardized manner with standardized APIs, metadata and formats thus leading to better decision making, business intelligence and productivity improvements.This paper presents a discussion and evaluation for the most prominent techniques used in the processes of data collection and analysis in order to identify the privacy defects in them that affects the accuracy of big data. Depending on the results of this analysis, recommendations were provided for improving data collection and analysis techniques that will help to avoid if not all then most of the problems facing the use of big data in decision making. Keywords: Big Data, Big Data Challenges, Big Data Accuracy, Big Data Collection, Big Data Analytics.

The present study utilizes VirtualBox virtual environment technology to develop the personal big data multi-VM platform with four-node Spark and Hadoop cluster that can effectively replicate and provide an environment for developers to easily design and implement the Spark and Hadoop Map/Reduce programming. Before running their Big Data and deep learning applications in physical multi-node Spark and Hadoop Cluster, developers can conduct Map/Reduce programing simply on the proposed multi-VM platform, which is exactly the same as the physical one. To demonstrate its capability and applicability, this study utilizes the deep learning application as an example for function illustration. In this study, the big data multi-VM platform provides the rapid prototyping of distributed deep learning by using a cutting-edge framework TensorFlowOnSpark (TFoS) for AI developers. To look into deep insight, this study performs the deep-learning benchmark in different types of cluster systems including the multi-node big data VM platform, physical standalone system and the physical small-cluster system. The results indicate that InputMode.SPARK can get 3.3 times faster than InputMode.TENSORFLOW on the big data VM platform and even achieve 6.1 times faster on the physical server.

The importance of agricultural big data is becoming more and more important to people, in the industry market space will have a huge application development space, become an important strategic resources and important technical research direction, the deep integration of big data and agriculture is expected to become the gas pedal of the development of agricultural modernization. The collection and mining of big data is the first priority for the development of "big data + agriculture". Agricultural big data involves agricultural water, soil, light, heat, climate resources, as well as crop breeding, planting, fertilization, plant protection, process management, harvesting, processing, storage, mechanization and other aspects of the collection of multiple types of complex data, processing analysis, mining and application issues. Agricultural big data in the integration and sharing of a variety of agricultural data on the basis of in-depth mining and analysis, to find the hidden value of the data, can greatly improve the government's decision-making and management level, but also can really play out the role of big data on the development of modern agriculture to boost.