In-Mapper combiner based MapReduce algorithm for processing of big climate data

Hadoop Distributed File System (HDFS) is the underlying storage engine of many Big Data processing frameworks such as Hadoop MapReduce, HBase, Hive, and Spark. Even though HDFS is well-known for its scalability and reliability, the requirement of large amount of local storage space makes HDFS deployment challenging on HPC clusters. Moreover, HPC clusters usually have large installation of parallel file system like Lustre. In this study, we propose a novel design to integrate HDFS with Lustre through a high performance key-value store. We design a burst buffer system using RDMA-based Mem cached and present three schemes to integrate HDFS with Lustre through this buffer layer, considering different aspects of I/O, data-locality, and fault-tolerance. Our proposed schemes can ensure performance improvement for Big Data applications on HPC clusters. At the same time, they lead to reduced local storage requirement. Performance evaluations show that, our design can improve the write performance of Test DFSIO by up to 2.6x over HDFS and 1.5x over Lustre. The gain in read throughput is up to 8x. Sort execution time is reduced by up to 28% over Lustre and 19% over HDFS. Our design can also significantly benefit I/O-intensive workloads compared to both HDFS and Lustre.

Big data has emerged as the next technological revolution in IT industry after cloud computing and the Internet of Things. With the development of climate observing systems, particularly satellite meteorological observation and high-resolution climate models, and the rapid growth in the volume of climate data, climate prediction is now entering the era of big data. The application of big data will provide new ideas and methods for the continuous development of climate prediction. The rapid integration, cloud storage, cloud computing, and full-sample analysis of massive climate data makes it possible to understand climate states and their evolution more objectively, thus predicting the future climate more accurately. This paper describes the application status of big data in operational climate prediction in China; it analyzes the key big data technologies, discusses the future development of climate prediction operations from the perspective of big data, speculates on the prospects for applying climatic big data in cloud computing and data assimilation, and puts forward the notion of big data-based super-ensemble climate prediction methods and computerbased deep learning climate prediction methods.

Internet of Things (IoT) is nowadays increasingly becoming a worldwide network of interconnected devices uniquely addressable, via a standard communication protocol. Such devices generate a massive volume of heterogeneous data, which lead a system towards a major computational challenges, such as aggregation, storing, and processing. Also, a major problem arises when there is a need to extract useful information from this massive volume of data. Therefore, to address these needs, this paper proposes an architecture to analyze big data in the IoT. The basic concept involves the partitioning of dynamic data, i.e., big data with the complex magnitude is divided into subsets. These subsets are based on the theoretical model of data fusion, which works in the Hadoop processing server to enhance the computational efficiency. The proposed architecture is tested by analyzing healthcare data sets, mainly comprises of activities including walking, running, ECG. The feasibility and efficiency of the proposed architecture are implemented on Hadoop single node setup on UBUNTU 14.04 LTS core™i5 machine with 3.2 GHz processor and 4 GB memory. The results show that the proposed architecture efficiently analyze the massive volume of data with a maximum throughput.

Ambient intelligence is an emerging platform that provides advances in sensors and sensor networks, pervasive computing, and artificial intelligence to capture the real time climate data. This result continuously generates several exabytes of unstructured sensor data and so it is often called big climate data. Nowadays, researchers are trying to use big climate data to monitor and predict the climate change and possible diseases. Traditional data processing techniques and tools are not capable of handling such huge amount of climate data. Hence, there is a need to develop advanced big data architecture for processing the real time climate data. The purpose of this paper is to propose a big data based surveillance system that analyzes spatial climate big data and performs continuous monitoring of correlation between climate change and Dengue. Proposed disease surveillance system has been implemented with the help of Apache Hadoop MapReduce and its supporting tools.

Extensive usage of Internet based applications in day to day life has led to generation of huge amounts of data every minute. Apart from humans, data is generated by machines like sensors, satellite, CCTV etc. This huge collection of heterogeneous data is often referred as Big Data which can be processed to draw useful insights. Apache Hadoop has emerged has widely used open source software framework for Big Data Processing and it is a cluster of cooperative computers enabling distributed parallel processing. Hadoop Distributed File System is used to store data blocks replicated and spanned across different nodes. HDFS uses an AES based cryptographic techniques at block level which is transparent and end to end in nature. However cryptography provides security from unauthorized access to the data blocks, but a legitimate user can still harm the data. One such example was execution of malicious map reduce jar files by legitimate user which can harm the data in the HDFS. We developed a mechanism where every map reduce jar will be tested by our sandbox security to ensure the jar is not malicious and suspicious jar files are not allowed to process the data in the HDFS. This feature is not present in the existing Apache Hadoop framework and our work is made available in github for consideration and inclusion in the future versions of Apache Hadoop.

Many-Task Computing (MTC) has been a new computing paradigm that aims to bridge the gap between traditional High-Throughput Computing (HTC) and High-Performance Computing (HPC). MTC applications from various scientific domains such as pharmaceuticals, astronomy, physics often consist of a very large number (from thousands to even billions) of data-intensive (tens of MB of I/O per second) tasks with relatively short per task execution times (from seconds to minutes). Each task in MTC applications may require relatively small amount of data processing especially compared to existing Big Data applications typically based on larger data block sizes (e.g. the default block size in Hadoop is 64MB). However, they can consist of much larger numbers of tasks where each task communicates through files instead of message passing interfaces such as MPI in HPC applications. Therefore, MTC can be another type of data-intensive workload where a large number of data processing tasks should be efficiently processed within a relatively short period of time. In this paper, we present design and implementation of MOHA (Many-task computing On HAdoop) which can make an effective convergence of MTC technologies and the existing Big Data platform Hadoop. MOHA is developed as a Hadoop YARN application so that it can transparently co-host existing MTC applications with other Big Data processing frameworks such as MapReduce in a single Hadoop cluster. Our evaluation results based on microbenchmark show that MOHA can substantially reduce the overall execution time of many-task processing with minimal amount of resources compared to an existing Hadoop YARN application. In addition, MOHA can efficiently dispatch a large number of tasks which can be crucial to support challenging MTC applications. MOHA can bring many interesting research issues related to data grouping and declustering on Hadoop Distributed File System (HDFS), scalable job/metadata management, dynamic task load balancing which can ultimately contribute to a new data processing framework in the YARN based Hadoop 2.0 ecosystem.

As the amount of data is increasing rapidly, its storage and security issues also increase. Hadoop is currently a popular framework for big data storage and processing. Since, security is not a primary consideration in Hadoop when it was initially developed makes the data stored in Hadoop distributed file system (HDFS) vulnerable to attacks. For protecting these data, numerous encryption techniques has been developed. This paper analyses and presents a comparison on advanced encryption standard (AES), a symmetric cipher and Rivest-Shamir-Adleman (RSA), an asymmetric cipher cryptographic algorithms. Finally, this paper proposes a hybrid method that combines both RSA and AES technique to improve the performance of encryption-decryption on HDFS files and thereby increases the security of big data in Hadoop framework. The proposed technique is found to be feasible and is able to enhance the confidentiality of HDFS files stored in Hadoop framework and is also able to overcome the limitations when using the AES encryption technique alone.KeywordsBig dataData securityHadoopHDFSEncryptionDecryptionAESRSAHybrid

The fast development of technologies shapes the way companies address and understand their customers’ needs, including the more and more pressing call for sustainability. If, by now, many organizations use the advantages of social media in their marketing strategies, newer technologies, such as Internet of things (IoT) wearables, are not fully used to their whole potential. Thus, we conducted two research studies—a qualitative one in the form of a focus group where eight different companies’ representatives took part, followed by a quantitative one in the form of an online questionnaire, where 84 (Small and Medium Sized Enterprises) SMEs and Large Enterprises answered. The main purpose of our research was to investigate companies’ attitudes and practices about using social media and IoT wearable technologies in developing organizational marketing strategies. The results indicate that, though there are some differences in the perception and use of social media and IoT wearables for developing marketing strategies, these differences are not marked between SMEs and Large Enterprises, but rather between micro enterprises and other companies with higher numbers of employees. Additionally, there are some differences noticed between companies operating in regional, national, or international markets.

As cloud computing developing rapidly, and Internet of Things and social network applying widely, data type and quantity are growing at an amazing speed. The era of big data has come. However, these create various challenges concerning: Capture, transport, storage, sharing, management, analysis, security, etc. How to trustfully and securely capture and transport data, is a very crucial issue for big data. In the paper, this work aims to construct a trusted big data capture and transport architecture. Firstly, the concepts of big data, trust and reputation, and wireless sensor network are discussed. Then a trusted big data capture and transport architecture using wireless sensor network is designed in detail. Moreover, some implementations of this architecture are described. Records and analysis show this trusted architecture is efficient, and can resist internal compromise attack and external attack.

As we are living in the era of big data, huge volumes of a wide variety of complex data-which can be of different levels of veracity-are generated or collected at a high velocity from rich sources of data in various real-life applications. A rich source of these big data sources is the Internet of Things (IoT), which include a collection of sensors, smartphones and other mobile devices, wearable devices, as well as other "things" that are capable to operate within the existing Internet infrastructure. Embedded in these big data are valuable knowledge and useful information. Hence, the research problem of data mining from big IoT data have drawn attention of many researchers as it aims to discover implicit, previously unknown and potentially useful information and knowledge from the data. For instance, frequent pattern mining finds sets of frequently co-occurring items in the IoT domains. Associative classification discovers rules revealing relationships among items within the frequent patterns and their associations with the corresponding class labels. Induction based classification uses decision tree or random forest to learn from old big IoT for classifying or making predictions on new data. Over the past quarter of a century, many serial, distributed, parallel, and MapReduce-based (Hadook-based and Spark-based) big data mining algorithms have been proposed. These algorithms are run in local computers, distributed and parallel environments, clusters, grids, clouds and/or data centers. In this paper, we review some of these algorithms, discuss issues and research prospective in mining classification patterns from these big IoT data in fog. Our case study on a real-life application shows the feasibility on classifying real-life big IoT data over fog for urban analytics.

Big data refers to a massive volume of data collected from heterogeneous data sources including data collected from Internet of Things (IoT) devices. Big data analytics is playing a crucial role in extracting patterns that would benefit efficient and effective decision making. Processing this massive volume of data poses several critical issues such as scalability, security and privacy. To preserve data privacy, numerous privacy-preserving data mining and publishing techniques exist. Data anonymization utilizing data mining techniques for preserving an individual’s privacy is a promising approach to prevent the data against identity disclosure. In this paper, a Parallel Clustering based Anonymization Algorithm (PCAA) is proposed, and the results prove that the algorithm is scalable and also achieves a better tradeoff between privacy and utility. The MapReduce framework is used to parallelize the anonymization process for handling a huge volume of data. The algorithm performs well in terms of classification accuracy, F-measure, and Kullback–Leibler divergence metrics. Moreover, the big data generated from heterogeneous data sources are efficiently protected to meet the ever-growing requirements of the application.

<p>By rapid transformation of technology, huge amount of data (structured data and Un Structured data) is generated every day. With the aid of 5G technology and IoT the data generated and processed every day is very large. If we dig deeper the data generated approximately 2.5 quintillion bytes.<br> This data (Big Data) is stored and processed with the help of Hadoop framework. Hadoop framework has two phases for storing and retrieve the data in the network.</p> <ul> <li>Hadoop Distributed file System (HDFS)</li> <li>Map Reduce algorithm</li> </ul> <p>In the native Hadoop framework, there are some limitations for Map Reduce algorithm. If the same job is repeated again then we have to wait for the results to carry out all the steps in the native Hadoop. This led to wastage of time, resources. If we improve the capabilities of Name node i.e., maintain Common Job Block Table (CJBT) at Name node will improve the performance. By employing Common Job Block Table will improve the performance by compromising the cost to maintain Common Job Block Table.<br> Common Job Block Table contains the meta data of files which are repeated again. This will avoid re computations, a smaller number of computations, resource saving and faster processing. The size of Common Job Block Table will keep on increasing, there should be some limit on the size of the table by employing algorithm to keep track of the jobs. The optimal Common Job Block table is derived by employing optimal algorithm at Name node.</p>

Nowadays, Internet-of-Things (IoT) devices generate data at high speed and large volume. Often the data require real-time processing to support high system responsiveness which can be supported by localised Cloud and/or Fog computing paradigms. However, there are considerably large deployments of IoT such as sensor networks in remote areas where Internet connectivity is sparse, challenging the localised Cloud and/or Fog computing paradigms. With the advent of the Raspberry Pi, a credit card-sized single board computer, there is a great opportunity to construct low-cost, low-power portable cloud to support real-time data processing next to IoT deployments. In this paper, we extend our previous work on constructing Raspberry Pi Cloud to study its feasibility for real-time big data analytics under realistic application-level workload in both native and virtualised environments. We have extensively tested the performance of a single node Raspberry Pi 2 Model B with httperf and a cluster of 12 nodes with Apache Spark and HDFS (Hadoop Distributed File System). Our results have demonstrated that our portable cloud is useful for supporting real-time big data analytics. On the other hand, our results have also unveiled that overhead for CPU-bound workload in virtualised environment is surprisingly high, at 67.2%. We have found that, for big data applications, the virtualisation overhead is fractional for small jobs but becomes more significant for large jobs, up to 28.6%.

Big Data in Internet of Healthcare Things (IoHT) environments includes large volumes of structured and unstructured clinical information. The Hadoop Distributed File System (HDFS) is widely used for its scalability and ability to run on commodity hardware. However, it offers limited native encryption, leaving data vulnerable to security risks. Although several encryption techniques exist, traditional algorithms still face performance and security limitations with large-scale medical datasets. Therefore, this study introduces a hybrid encryption framework designed to enhance security in IoHT environments that process large-scale medical Big Data. The framework combines Attribute-Based Encryption (ABE) with the Blowfish cipher to secure data generated by heterogeneous medical devices across the IoHT infrastructure. The proposed approach is benchmarked against established hybrid schemes-CP-ABE + HE, HE + BF, and CP-ABE + AES-to provide a comparative assessment of its security strength and computational performance. The performance assessment employed key computational metrics, including system efficiency, encryption latency, and decryption latency. Experimental results demonstrate that the proposed hybrid scheme delivers superior performance compared to existing approaches, attaining a peak efficiency of 98.5%. The method further achieved encryption and decryption times of 6.8min and 5.7min, respectively, indicating improved computational handling of large-scale IoHT data.

The discovery of the dynamics of influenza outbreaks remains a great challenge. The research aims to better predict seasonal influenza outbreaks using climate and big data in Brisbane and Gold Coast, Australia.Weekly influenza notifications, climate data (ie., temperature and rainfall) and big data (ie., internet search query (ISQ)) between January, 2011 and December, 2016 were collected from Queensland Health, Bureau of meteorology, Australia and Google Trends, respectively. A time-series cross correlation function (CCF) was used to compute the correlations between independent variables and influenza over a range of time lags. Increasing duration index (α) and increasing intensity index (β) were used to discover the characteristics of influenza outbreaks. Time series seasonal autoregressive integrated moving average (SARIMA) and regression tree model with climate and ISQ were developed to predict influenza epidemics.The CCF showed that temperature at lag of 1–10 week in both two study settings was inversely associated with influenza, whereas ISQ at lag of 1–7 week was positively associated with the influenza. The largest β value was found in 2015. The inclusion of ISQ and temperature improved the predictive performance of SARIMA model (Brisbane: βISQ = 2.42, βtemperate = -25.12, increased agreement: 8%; Gold Coast: βISQ = 0.38, βtemperate = -7.73, increased agreement: 6%; p < 0.05). Regression tree model showed that when internet search query ≥65 and weekly mean maximum temperature <21o C, the relative risk (RR) of influenza was 1.6 and 2.9 respectively in Brisbane, whereas internet search query ≥66 and weekly mean maximum temperature <23 o C, the RR of influenza was 2.8 and 1.1 separately in Gold Coast.Our results suggested big data in conjunction with temperature can be used to predict influenza outbreaks, which can be considered as a pre-requisite for constructing early warning systems using these data.