A Brief Introduction on Big Data 5Vs Characteristics and Hadoop Technology

Taking a byte out of big data

Nowadays a lot of massive data is stored and typically, the data itself contains a lot of non-trivial but useful information. Data mining techniques can be used to discover this information which can help the companies for decision-making. However, in real life applications, data is massive and is stored over distributed sites. One of my major research topics is to protect privacy over this kind of data. Previously, the important characteristics, issues and challenges related to management of the large amount of data has been explored. Various open source data analytics frameworks that deal with large amount of Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. Digital universe is flooded with huge amount of data generated by number of users worldwide. These data are of diverse in nature, come from various sources and in many forms. To keep with the desire to store and analyze ever larger volumes of complex data, relational databases vendors have delivered specialized analytical platforms that come in many shapes and sizes from software only to analytical services that run in third party hosted environments. In addition new technologies have emerged to address exploding volumes of complex data, including web traffic, social media content and machine generated data including sensor data, global positioning system data.Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data has become a prominent research field, especially when it comes to decision making and data analysis. However, Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges. Since Big data is a recent upcoming technology in the market which can bring huge benefits to the business organizations, it becomes necessary that various challenges and issues associated in bringing and adapting to this technology are brought into light. Another challenge is that data collection may not have enough accuracy which will lead to a non-consistent analysis that can critically affect the decision based on this analysis. Moreover, it is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is; current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Since data is growing at a huge speed making it difficult to handle such large amount of data (Exabyte). The main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties.If data is to be used to make accurate decisions in time it becomes necessary that it should be available in accurate, complete and timely manner. This makes the data management and governance process bit complex adding the necessity to make Data open and make it available to government agencies in standardized manner with standardized APIs, metadata and formats thus leading to better decision making, business intelligence and productivity improvements.This paper presents a discussion and evaluation for the most prominent techniques used in the processes of data collection and analysis in order to identify the privacy defects in them that affects the accuracy of big data. Depending on the results of this analysis, recommendations were provided for improving data collection and analysis techniques that will help to avoid if not all then most of the problems facing the use of big data in decision making. Keywords: Big Data, Big Data Challenges, Big Data Accuracy, Big Data Collection, Big Data Analytics.

In IT industry research communities and data scientists have observed that Big Data has challenged the legacy of solutions. ‘Big Data’ term used for any collection of data or data sets which is so large and complex and difficult to process and manage using traditional data processing applications and existing Relational Data Base Management Systems (RDBMSs). In Big Data; the most important challenges include analysis, capture, curation, search, sharing, storage, transfer, visualization and privacy. As the data increases in various dimensions with various features like structured, semi structured and unstructured with high velocity, high volume and high variety; the RDBMSs face another fold of challenges to be studied and analyzed. Due to the aforesaid limitations of RDBMSs, data scientists and information managers forced to rethink about alternative solutions for handling such data with 3Vs.Initially research study focused on to develop an intelligent base for decision makers so that alternative solutions for long term suitable solutions and handle the data and information with 3Vs can be designed. In this research attempts has been made to analyze the feature based capabilities of RDBMSs and then performance experimentation, observation and analysis has been done with Big Data handling tools and technologies. The features considered for scientific observation and analysis were resource consumption, execution time, on demand scalability, maximum data size, structure of the data, data visualization, and ease of deployment, cost and security. Finally the research provides a decision support metrics for decision makers in selecting the appropriate tool or technology based on the nature of data to be handled in the target organizations.

Addressing the policy challenges and opportunities of “Big data”

Big Data OverviewDriven by the need to generate business value, the enterprise has started to adopt Big Data as a solution, migrating from the classical databases and data stores which lack the flexibility and are not optimized enough [1].The changes in the environment make big data analytics attractive to all types of organizations, while the market conditions make it practical. The combination of simplified models for development, commoditization, a wider palette of data management tools, and low-cost utility computing has effectively lowered the barrier to entry. [2]. The concept addresses large volumes of complex data, rapid growing data sets that may come from different autonomous sources.In recent approaches, Big Data is characterized by principles known as the 4VVolume, Variety, Velocity and Veracity [3]. There are opinions about accepting other principles as Big Data characteristics, such as Value.Each day more businesses realize that Big Data is relevant as the applications generate large volumes of data generated automatically, from different data sources, centralized or autonomous. As traditional databases hit limitations when the need of analyzing this data, dedicated solutions must be considered.Important BigData Solutions:* Apache HBase/Hadoop is based on Google's BigTable distributed storage system, which runs on top of Hadoop as a distributed and scalable big data store. This means that HBase can leverage the distributed processing paradigm of the Hadoop Distributed File System (HDFS) and benefit from Hadoop's MapReduce programming model. It combines the scalability of Hadoop with real-time data access as a key/value store and deep analytic capabilities of Map Reduce [4].HBase allows to query for individual records as well as derive aggregate analytic reports across a massive amount of data. It can host large tables with billions of rows, millions of columns and run across a cluster of commodity hardware. HBase is composed of three types of servers in a master slave type of architecture. Region servers are responsible to serve data for reads and writes. When accessing data, clients communicate with HBase Region Servers directly. Region assignment, DDL (create, delete tables) operations are handled by the HBase Master process.* Apache Cassandra is a distributed database used for the administration and management of large amounts of structured data across multiple servers, while providing highly available service and no single point of failure. It provides features such as continuous availability, linear scale performance, data distribution across multiple data centers and cloud availability zones. Cassandra inherits its data architecture from Google's BigTable and it borrows its distribution mechanisms from Amazon's Dynamo.The nodes in a Cassandra cluster are completely symmetrical, all having identical responsibilities. Cassandra also employs consistent hashing to partition and replicate data. it has the capability of handling large amounts of data and thousands of concurrent users or operations per second across multiple data centers.Cassandra has a hierarchy of caching mechanisms and carefully orchestrated disk I/O which ensures speed and data safety. Write operations are sent first to a persistent commit log (ensuring a durable write), then to a write-back cache called a memTable; when the memTable fills, it is flushed to a sorted string table - SSTable - on disk. A Cassandra cluster is organized as a ring, and it uses a partitioning strategy to distribute data evenly.* Redis represents an in-memory data structure store used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperlogs and geospatial indexes with radius queries. Redis stores all data in RAM, allowing lightning fast reads and writes. It runs extremely efficiently in memory and handles high-velocity data, needing simple standard servers to deliver millions of operations per second with submillisecond latency. …

Machine learning for Big Data analytics in plants.

The term "big data" is becoming increasingly common these days. The amount of data generated is directly proportional to the amount of time spent on social media each day. The majority of users consider Twitter to be one of the most popular social networking platforms. The rise of social media has sparked an incredible amount of curiosity among those who use the internet nowadays. The information collected from these social networking sites may be put to a variety of uses, including forecasting, marketing, and the study of user sentiment. Twitter is a social media platform that is commonly used for making remarks in the form of brief status updates. A sentiment analysis may be performed on some or all of the millions of tweets that are received each year. Managing such a massive volume of unstructured data, on the other hand, is a laborious effort to do. To effectively manage large amounts of data, the analytics tools and models that are now on the market are insufficiently equipped and positioned. For this reason, it is essential to make use of a cloud storage solution for the applications of this kind. As a result, we have used Hadoop for the intelligent analysis as well as the storing of large amounts of data. In this article, we offer a system that does sentiment analysis on tweets using the Cloud.

In-Mapper combiner based MapReduce algorithm for processing of big climate data

Internet of Things (IoT) is nowadays increasingly becoming a worldwide network of interconnected devices uniquely addressable, via a standard communication protocol. Such devices generate a massive volume of heterogeneous data, which lead a system towards a major computational challenges, such as aggregation, storing, and processing. Also, a major problem arises when there is a need to extract useful information from this massive volume of data. Therefore, to address these needs, this paper proposes an architecture to analyze big data in the IoT. The basic concept involves the partitioning of dynamic data, i.e., big data with the complex magnitude is divided into subsets. These subsets are based on the theoretical model of data fusion, which works in the Hadoop processing server to enhance the computational efficiency. The proposed architecture is tested by analyzing healthcare data sets, mainly comprises of activities including walking, running, ECG. The feasibility and efficiency of the proposed architecture are implemented on Hadoop single node setup on UBUNTU 14.04 LTS core™i5 machine with 3.2 GHz processor and 4 GB memory. The results show that the proposed architecture efficiently analyze the massive volume of data with a maximum throughput.

In today's world, the increase in the volume and variety of data used in the process of military management leads to an increase in the requirements for their storage and processing. To effectively manage military operations, it is necessary to have information about various aspects of military activity, such as the location of military units, the availability of resources, the assessment of potential threats, and much more. Modern military operations require the processing and analysis of a large amount of data generated in real time. Data collected from sensors, satellites, probes, sensors, and other sources can be vast in volume and diverse in structure. The increase in the volume and heterogeneity of data used in the process of military management leads to an increase in the requirements for their storage and processing. This presents military organizations with the task of creating effective and reliable data storage and processing systems. The article considers the possibilities of using «Big Data» technology, which allow collecting, storing, processing and analyzing large volumes of various data in real time. The use of modern data management technologies can help increase the efficiency of military activities and ensure fast and accurate decision-making at various levels of management. This can be achieved through the use of artificial intelligence, machine learning and data analysis technologies that can help military structures perform calculations and forecasts, optimize the use of resources and increase the efficiency of solving various tasks. Also, an important component of the effective use of «Big Data» technologies in the military sphere is ensuring the protection of data from unauthorized access, as well as ensuring the reliability and stability of data storage and processing systems in the face of possible cyberattacks. The article analyzes the Hadoop system, which can be used to store, process and analyze «Big Data» in the military sphere.

Big data refers to a very large volume of data with possibly varied and complex structure. With growing data processing and data analytic techniques, big data provides significant benefits to organisations and individuals by improving productivity and enriching people's life. However, security and privacy are big concerns for big data applications. While a large quantity of data is collected, securely storing, processing and using the data are challenging. In this paper, we propose a four-factor framework for big data security and privacy in business information systems. The proposed framework addresses big data security and privacy issues in terms of collecting the right data, collecting the right amount of data, protecting the data in the right way, and using the data for the right purposes. We present a set of approaches and models for each of the four factors to improve big data security and privacy.

Cloud computing and big data: Technologies and applications

As cyber-attacks and network vulnerabilities get more sophisticated, optimizing protection systems utilizing advanced computational methodologies is critical. Big data approaches are a viable way to simulate network attacks and improve protection measures. But conventional approaches frequently can’t handle massive volumes of real-time data or quickly adjust to changing threats. The goal of the research is to create a large data-driven computer-aided network attack simulation and defense system that uses advanced Machine Learning (ML) techniques to optimize defensive techniques, improve threat detection, and increase system adaptability. A hybrid system that combines big data analytics with Hyperbolic Tangent Particle Swarm Optimized Decision Tree (HTPSO-DT) methods was presented to simulate and predict possible cyber-attacks. Big data refers to the methodologies for collecting, processing by min–max scaling, and extracting insights from diverse, high-volume, and high-velocity data sets using Discrete Wavelet Transform (DWT). The system simulates attack scenarios and optimizes defense responses using real-time network traffic data; behavior analysis and predictive modeling. The defense system adjusts by continuously learning from simulations and continually improving its techniques. The proposed method of HTPSO-DT has performed and achieved precision at 99.12%, recall at 99.15%, [Formula: see text]1 score at 99.17%, [Formula: see text]2 score at 99.09%, [Formula: see text]beta score at 98.95%, and ROC-AUU at 0.88. The method significantly improves attack detection accuracy and reduces defense response time. The suggested solution improves the effectiveness of network defense strategies by integrating big data and ML, allowing for real-time, adaptive protection.

Social media is being employed to build support for social, economic, and political justice (Selander et al. 2016; Vaast et al. 2014) in movements such as occupy Wall Street, violence against women's movement and global sustainability movement. These movements have used social media in ways that goes beyond simple communications. As social media allows people to produce and share user generated content, they enable a certain set of affordances of these technologies (Bharati et al 2015; Bharati et al 2014). The social media affordances, available to both collective and individual actors, translate into capabilities afforded to social movements (Tufecki 2014). Research on connective action has examined the effects of digital action repertoires on interaction and engagement such as in the Tea Party and Occupy movements (Agarwal et al. 2014; Selander et al. 2016). Social media can also facilitate mobilization of movement and participation by new volunteers and oftentimes provides a transnational character by diffusing actions beyond the virtual (Van Laer and Van Aelst 2010). Conversation, an essential part of social movements, shapes “social life by altering individual and collective understandings, by creating and transforming social ties, by generating cultural materials that are then available for subsequent social interchange, and by establishing, obliterating, or shifting commitments on the part of participants&x201D; (Tilly 2002, p. 122). In a personal interaction that involves repeated organized interactions between individuals, typically, leads to shared values and trust. The role of social media technologies in furthering this conversation has to be studied and its' influence on social movement ascertained. A few scholars have started to investigate social media affordances and capabilities, especially focused on discourse, during contentious collective action. Still the research has been limited to studying mechanisms of participation, development of a sense of collective identity, creation of community, and framing of political discourse (Farrell 2012; Garrett 2006). Social media data on social movements can involve impersonal “like&x201D; and “share&x201D; to more engaged conversations. Twitter, Instagram, Facebook, and YouTube offer a wide communicative and discourse reach in networks with text, image, audio, and video data. This social media based big data, consisting mostly of unstructured data, comes in the form of social media posts, digital pictures and videos. The symposium will discuss how formal organizational structures and practices might be integrated with social media capabilities to reinforce and enhance social movement organizations as leaders in social change movements. It will explore the characteristics of social media discourse and assess the dynamics of social movements and, subsequent, impact on real-world protests. The symposium will also demonstrate how discourse analysis can be applied visually in order to understand communication patterns. The panel symposium will focus on theoretical and methodological challenges of social media analytics, big data and social movements. Panelists will engage the audience in an interactive discussion on: 1) Theoretical challenges: a. How and why are social movement recruitment and engagement mechanisms being impacted as a result of social media? b. How do we advance theory on social media and social movements when we are overwhelmed with social media based big data? c. What approach should we undertake if big data analysis contradicts most theories on social media and social movements? d. How do we address the issue of generalizability of social media and social movement research when data collection was limited to one social media platform, albeit involving big data? 2) Methodological challenges: a. What methodological approaches have worked in the analysis of social media, big data and social movements? b. How can discourse analysis be applied visually in order to understand communication patterns evident in social media-based big data? c. How can we employ social media analytics to investigate image and video data? d. What combinations of qualitative and quantitative methodologies be employed for big data and social media analytics in the context of social movements? e. What are the limitations of quantitative data analysis techniques, such as structural equation modeling, because of an extremely large sample size? f. What are the limitations of qualitative data analysis techniques as they become extremely labor intensive and, maybe even, impractical because of big data?

Big data refers to a massive volume of data collected from heterogeneous data sources including data collected from Internet of Things (IoT) devices. Big data analytics is playing a crucial role in extracting patterns that would benefit efficient and effective decision making. Processing this massive volume of data poses several critical issues such as scalability, security and privacy. To preserve data privacy, numerous privacy-preserving data mining and publishing techniques exist. Data anonymization utilizing data mining techniques for preserving an individual’s privacy is a promising approach to prevent the data against identity disclosure. In this paper, a Parallel Clustering based Anonymization Algorithm (PCAA) is proposed, and the results prove that the algorithm is scalable and also achieves a better tradeoff between privacy and utility. The MapReduce framework is used to parallelize the anonymization process for handling a huge volume of data. The algorithm performs well in terms of classification accuracy, F-measure, and Kullback–Leibler divergence metrics. Moreover, the big data generated from heterogeneous data sources are efficiently protected to meet the ever-growing requirements of the application.