Big Data Privacy Preservation Using Two Phase Top-Down Specialization Algorithm with Multidimensional Map Reduce Framework on Hadoop

In the big data processing systems, the amount of data is increasing. At the same time, the real-time requirement of data processing and analysis is higher and higher. Therefore, it is required that the big data processing and analysis systems have better performance. Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. However, job scheduling is a difficult NP-hard problem. There are many factors that need to be considered for job scheduling. For example, jobs have dependencies among stages, therefore we should not allocate resources to tasks that are not ready. Sometimes, there are constraints between jobs. These are a challenge to the scheduling performance of big data processing and analysis systems. In this paper, we try to solve the problem by translating it into Boolean Satisfiability Problem (SAT) which is an exact method. SAT-based scheduling algorithm is not a new approach, but in the past it mainly used to solve the static scheduling problems. For dynamic scheduling system, it requires all problems to be solved within a limited time, which is a challenge for SAT encoding. In this paper, we refer to the previous SAT solution to the Job Shop Scheduling Problem, and adjust the algorithm to meet the requirements of the big data processing system. At the same time, we optimized the coding approach and reduced the number of clauses. Thus, the efficiency of the problem solved is improved to meet the performance requirements. The experimental results show that the number of clauses is reduced by more than 30%, and the processing time of the SAT solver to get the solution can be reduced by more than 50%. To demonstrate its effectiveness, we have also implemented our new job scheduler in Apache Hadoop YARN, and validated its effectiveness

Privacy is an important issue for big data including sensitive attributes. In the case of directly sharing or publishing these data, privacy breach occurs. In order to overcome this problem, previous studies were focused on developing big data anonymization techniques on Hadoop environment. When compared to Hadoop, Spark facilitates to develop faster applications with the help of keeping data in memory instead of hard disk. Despite a number of projects were developed on Hadoop, now this trend is shifting to Spark. In addition, the problem of anonymizing big data streams for real-time applications can be solved with Spark technology. Hence to sum up, Spark is the main technology facilitates developing both faster anonymization applications and big data stream anonymization solutions. In this study, anonymization techniques, big data technologies and privacy preserving big data publishing was reviewed and a big data anonymization model based on Spark was proposed for the first time. It is expected that the proposed model might help to researchers to solve big data privacy issues and also provide solutions for new generation privacy violations problems.

With the rapid development of Internet of Things (IoT) technologies, the increasing volume and diversity of sources of geospatial big data have created challenges in storing, managing, and processing data. In addition to the general characteristics of big data, the unique properties of spatial data make the handling of geospatial big data even more complicated. To facilitate users implementing geospatial big data applications in a MapReduce framework, several big data processing systems have extended the original Hadoop to support spatial properties. Most of those platforms, however, have included spatial functionalities by embedding them as a form of plug-in. Although offering a convenient way to add new features to an existing system, the plug-in has several limitations. In particular, while executing spatial and nonspatial operations by alternating between the existing system and the plug-in, additional read and write overheads have to be added to the workflow, significantly reducing performance efficiency. To address this issue, we have developed Marmot, a high-performance, geospatial big data processing system based on MapReduce. Marmot extends Hadoop at a low level to support seamless integration between spatial and nonspatial operations of a solid framework, allowing improved performance of geoprocessing workflow. This paper explains the overall architecture and data model of Marmot as well as the main algorithm for automatic construction of MapReduce jobs from a given spatial analysis task. To illustrate how Marmot transforms a sequence of operators for spatial analysis to map and reduce functions in a way to achieve better performance, this paper presents an example of spatial analysis retrieving the number of subway stations per city in Korea. This paper also experimentally demonstrates that Marmot generally outperforms SpatialHadoop, one of the top plug-in based spatial big data frameworks, particularly in dealing with complex and time-intensive queries involving spatial index.

Big data refers to a massive volume of data collected from heterogeneous data sources including data collected from Internet of Things (IoT) devices. Big data analytics is playing a crucial role in extracting patterns that would benefit efficient and effective decision making. Processing this massive volume of data poses several critical issues such as scalability, security and privacy. To preserve data privacy, numerous privacy-preserving data mining and publishing techniques exist. Data anonymization utilizing data mining techniques for preserving an individual’s privacy is a promising approach to prevent the data against identity disclosure. In this paper, a Parallel Clustering based Anonymization Algorithm (PCAA) is proposed, and the results prove that the algorithm is scalable and also achieves a better tradeoff between privacy and utility. The MapReduce framework is used to parallelize the anonymization process for handling a huge volume of data. The algorithm performs well in terms of classification accuracy, F-measure, and Kullback–Leibler divergence metrics. Moreover, the big data generated from heterogeneous data sources are efficiently protected to meet the ever-growing requirements of the application.

<span>Big data has remodeled the way organizations supervise, examine and leverage data in any industry. To safeguard sensitive data from public contraventions, several countries investigated this issue and carried out privacy protection mechanism. With the aid of quasi-identifiers privacy is not said to be preserved to a greater extent. This paper proposes a method called evolutionary tree-based quasi-identifier and federated gradient (ETQI-FD) for privacy preservations over big healthcare data. The first step involved in the ETQI-FD is learning quasi-identifiers. Learning quasi-identifiers by employing information loss function separately for categorical and numerical attributes accomplishes both the largest dissimilarities and partition without a comprehensive exploration between tuples of features or attributes. Next with the learnt quasi-identifiers, privacy preservation of data item is made by applying federated gradient arbitrary privacy preservation learning model. This model attains optimal balance between privacy and accuracy. In the federated gradient privacy preservation learning model, we evaluate the determinant of each attribute to the outputs. Then injecting Adaptive Lorentz noise to data attributes our ETQI-FD significantly minimizes the influence of noise on the final results and therefore contributing to privacy and accuracy. An experimental evaluation of ETQI-FD method achieves better accuracy and privacy than the existing methods.</span>

Big data and cloud computing (BDCloud) collectively offer a paradigm shift in the way businesses are now acquiring, using and managing information technology. With the fast growth of this paradigm, we argue that each and every CS and IT students should be equipped with foundation knowledge in this collective paradigm and should possess hand-on-experiences in managing big data applications in clouds to acquire skills that are necessary to meet current and future industry demands. This poster presents our research that proposes gradual and systematic integration of big data and cloud computing related topics into multiple core (required) courses of CS/IT curriculum. The poster, supported by a NSF grant, will be useful for CS/IT students and their instructors as it identifies big data and cloud computing related topics that are important to cover, finds a sequence of the prescribed topics that can be incorporated into existing core courses most effectively, and suggests specific core courses in which their coverage might find an appropriate context. The poster further identifies the major challenges this proposed intervention may encounter and provides a deeper analysis of them. Finally, the poster describes our experience of implementing one such course with proposed interventions during Fall of 2016 semester. The pre- post- test results that measure student opinion and understanding of big data and cloud computing topics are presented in the poster and demonstrate improved student interest and learning.

Big data needs to be kept private because of the increase in the amount of data. Data is generated from social networks, organizations and various other ways, which is known as big data. Big data requires large storage as well as high computational power. At every stage, the data needs to be protected. Privacy preservation plays an important role in keeping sensitive information protected and private from any attack. Data anonymization is one of the techniques to anonymize data to keep it private and protected, which includes suppression, generalization, and bucketization. It keeps personal and private data anonymous from being known by others. But when it is implemented on big data, these techniques cause a great loss of information and also fail in defense of the privacy of big data. Moreover, for the scenario of big data, the anonymization should not only focus on hiding but also on other aspects. This paper aims to provide a technique that uses slicing, suppression, and functional encryption together to achieve better privacy of big data with data anonymization.

Big data analytics gain significant interest over the traditional data‐processing methodologies that engage in extracting the hidden patterns and correlations from the massive data, termed as big data. With the aim of relieving the computational complexity the clustering method plays a significant role. With the knowledge of the clustering algorithms, the big data arriving from the distributed sources is processed using the MapReduce framework (MRF). The MRF possesses two functions, namely, map function and reduce function, such that the map function is based on the proposed Fractional Sparse Fuzzy C‐Means (FrSparse FCM) algorithm and reduce function is based on particle swarm optimisation‐based whale optimisation algorithm (P‐Whale). Initially, the optimal centroids are computed using the proposed algorithm in the mapper phase that is optimally tuned in the reducer phase, and it is clear that the proposed FrSparse FCM‐based MRF ensures the parallel processing of the big data. Experimentation is performed using the Skin data set and the localisation data set taken from the UCI machine learning repository, and the analysis is progressed using the metrics, such as accuracy and DB Index. The analysis proves that the proposed method acquired a maximum accuracy of 90.6012% and a minimum DB Index of 5.33.

Data anonymity, as an effective privacy protection method, has been widely used in real applications. High performance data anonymity algorithm is especially attractive for those massive data applications. In this paper, the authors propose a novel and efficient Datafly based data anonymity (Divide-Datafly) algorithm and the experimental results show that the proposed algorithm is not only more efficient than Datafly and Incognito, but also information loss less than KACA. Moreover, in order to improve the security of anonymous data, L-Divide-Datafly is presented that it combines Divide-Datafly and efficient distance based clustering. Experimental results show that L-Divide-Datafly achieves great performance both in execution time and Information loss.

The aim of the article is to develop theoretical foundations and generalize strategic approaches to staff development in the context of digital transformation and globalization processes, as well as to identify key vectors for the development and improvement of the human resources management system, taking into account modern challenges and opportunities of the digital economy. The methodological basis of the study is built on the synthesis of general scientific and special methods, supplemented by empirical tools for data collection and analysis, which guarantees the comprehensiveness, objectivity and reliability of the results obtained. The study provides a thorough analysis of the transformation of HR processes under the influence of digital technologies, in particular the use of artificial intelligence, big data processing systems, specialized HR platforms, VR/AR solutions and online educational environments. It is shown that these tools contribute to the modernization of recruiting, improvement of personnel assessment procedures, personalization of training and more efficient management of employee career growth. Emphasis is placed on the need to create a culture of continuous learning, develop digital competencies, stimulate motivation and form adaptive forms of employment that meet the challenges of the turbulent global market. It is substantiated that the integration of innovative HR technologies contributes to increasing productivity, optimizing training costs, reducing staff turnover and strengthening organizational resilience. It is concluded that in the modern business environment characterized by digital changes and globalization, strategic human resource development is becoming the main factor in ensuring competitive advantages of companies. The introduction of innovative technologies, in particular artificial intelligence algorithms, big data analytics and VR/AR tools, opens up new prospects for personalized learning, efficient recruitment and optimization of HR processes. At the same time, the spread of flexible employment models, such as remote and hybrid work, forms a new paradigm of labor relations that combines increasing productivity with ensuring a balance between the professional and personal needs of employees.

Cloud computing and big data: Technologies and applications

Big data refers to data sets whose size is beyond the capabilities of most current hardware and software technologies. The Apache Hadoop software library is a framework for distributed processing of large data sets, while HDFS is a distributed file system that provides high-throughput access to data-driven applications, and MapReduce is software framework for distributed computing of large data sets. Huge collections of raw data require fast and accurate mining processes in order to extract useful knowledge. One of the most popular techniques of data mining is the K-means clustering algorithm. In this study, the authors develop a distributed version of the K-means algorithm using the MapReduce framework on the Hadoop Distributed File System. The theoretical and experimental results of the technique prove its efficiency; thus, HDFS and MapReduce can apply to big data with very promising results.

Due to the rapid growth of network technology, huge volume and distinct data sent via networks is expanding constantly. The situation shows how complex and dense cyber attacks and hazards are developing. Due to the rapid advancement in network density, cyber security specialists find it difficult to monitor all network activity. Due to frequent and sophisticated cyber attacks, it is becoming more challenging to detect and identify abnormalities in network events. The use of deep learning provides a variety of tools and strategies for automated cyber-attack detection as well as quick attack-type prediction as well as evaluation. This work introduces a novel cyber-attack detection and mitigation process under the following phases including preprocessing, feature extraction via the Map Reduce framework that handles the big data, feature selection, attack detection and mitigation. The Improved Normalisation process is achieved on the preprocessing phase. The work is examined from a big data perspective; hence Map Reduce framework is utilised for this. As a result, the framework will manage the feature extraction process, where features including statistical features, raw features, improved correlation-based features, and info gain-based features will be extracted. Following feature extraction, the Improved Recursive Feature Elimination procedure is processed that selects the relevant features. The hybrid detection model, which combines Recurrent Neural Networks (RNN) Deep and Belief Networks (DBN) is used to detect the attacks. Once an attack has been detected, the attacker must be mitigated. To accomplish this, an improved BAIT-based mitigation procedure is used. The two datasets used in this work are, namely, Intrusion Detection Systems (IDS) 2018 Intrusion CSVs (CSE-CIC-IDS2018) and UNSW_NB15. Finally, the suggested model and the alternative methods are contrasted using a variety of measures such as accuracy, sensitivity, specificity, precision, FDR, FNR and FPR.

Earth observation (EO) data are remote sensing image data obtained by air-borne or space-borne sensors, which has the characters of distribution and heterogeneity. Because these data are owned by different space agencies. With Petabyte-scale data volumes of Earth Observation (EO) obtained, the EO data has been one of the most important part of big data. The characters of distribution and heterogeneity of big EO data storage and processing systems owned by different space agencies brings challenges to integrate and use these data. To meet the big EO data challenges, the new generation Spatial data infrastructure is needed to provide the solution and implementation technique to big EO data storage, access, process and analysis. The concept and architecture of Next-generation national EO Spatial Data Infrastructure (EO-SDI) is proposed in the paper. Key techniques are introduced to solve the complexity of distributed and heterogeneous EO data integration and on-demand processing. An interoperable national EO data system of systems is developed to integrate distributed EO archives of eight EO data centers.

Big Data- Is a Turnkey Solution