Globally distributed software defined storage (proposal)

Big Data OverviewDriven by the need to generate business value, the enterprise has started to adopt Big Data as a solution, migrating from the classical databases and data stores which lack the flexibility and are not optimized enough [1].The changes in the environment make big data analytics attractive to all types of organizations, while the market conditions make it practical. The combination of simplified models for development, commoditization, a wider palette of data management tools, and low-cost utility computing has effectively lowered the barrier to entry. [2]. The concept addresses large volumes of complex data, rapid growing data sets that may come from different autonomous sources.In recent approaches, Big Data is characterized by principles known as the 4VVolume, Variety, Velocity and Veracity [3]. There are opinions about accepting other principles as Big Data characteristics, such as Value.Each day more businesses realize that Big Data is relevant as the applications generate large volumes of data generated automatically, from different data sources, centralized or autonomous. As traditional databases hit limitations when the need of analyzing this data, dedicated solutions must be considered.Important BigData Solutions:* Apache HBase/Hadoop is based on Google's BigTable distributed storage system, which runs on top of Hadoop as a distributed and scalable big data store. This means that HBase can leverage the distributed processing paradigm of the Hadoop Distributed File System (HDFS) and benefit from Hadoop's MapReduce programming model. It combines the scalability of Hadoop with real-time data access as a key/value store and deep analytic capabilities of Map Reduce [4].HBase allows to query for individual records as well as derive aggregate analytic reports across a massive amount of data. It can host large tables with billions of rows, millions of columns and run across a cluster of commodity hardware. HBase is composed of three types of servers in a master slave type of architecture. Region servers are responsible to serve data for reads and writes. When accessing data, clients communicate with HBase Region Servers directly. Region assignment, DDL (create, delete tables) operations are handled by the HBase Master process.* Apache Cassandra is a distributed database used for the administration and management of large amounts of structured data across multiple servers, while providing highly available service and no single point of failure. It provides features such as continuous availability, linear scale performance, data distribution across multiple data centers and cloud availability zones. Cassandra inherits its data architecture from Google's BigTable and it borrows its distribution mechanisms from Amazon's Dynamo.The nodes in a Cassandra cluster are completely symmetrical, all having identical responsibilities. Cassandra also employs consistent hashing to partition and replicate data. it has the capability of handling large amounts of data and thousands of concurrent users or operations per second across multiple data centers.Cassandra has a hierarchy of caching mechanisms and carefully orchestrated disk I/O which ensures speed and data safety. Write operations are sent first to a persistent commit log (ensuring a durable write), then to a write-back cache called a memTable; when the memTable fills, it is flushed to a sorted string table - SSTable - on disk. A Cassandra cluster is organized as a ring, and it uses a partitioning strategy to distribute data evenly.* Redis represents an in-memory data structure store used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperlogs and geospatial indexes with radius queries. Redis stores all data in RAM, allowing lightning fast reads and writes. It runs extremely efficiently in memory and handles high-velocity data, needing simple standard servers to deliver millions of operations per second with submillisecond latency. …

In today's world, the increase in the volume and variety of data used in the process of military management leads to an increase in the requirements for their storage and processing. To effectively manage military operations, it is necessary to have information about various aspects of military activity, such as the location of military units, the availability of resources, the assessment of potential threats, and much more. Modern military operations require the processing and analysis of a large amount of data generated in real time. Data collected from sensors, satellites, probes, sensors, and other sources can be vast in volume and diverse in structure. The increase in the volume and heterogeneity of data used in the process of military management leads to an increase in the requirements for their storage and processing. This presents military organizations with the task of creating effective and reliable data storage and processing systems. The article considers the possibilities of using «Big Data» technology, which allow collecting, storing, processing and analyzing large volumes of various data in real time. The use of modern data management technologies can help increase the efficiency of military activities and ensure fast and accurate decision-making at various levels of management. This can be achieved through the use of artificial intelligence, machine learning and data analysis technologies that can help military structures perform calculations and forecasts, optimize the use of resources and increase the efficiency of solving various tasks. Also, an important component of the effective use of «Big Data» technologies in the military sphere is ensuring the protection of data from unauthorized access, as well as ensuring the reliability and stability of data storage and processing systems in the face of possible cyberattacks. The article analyzes the Hadoop system, which can be used to store, process and analyze «Big Data» in the military sphere.

With the rapid development of national economy, our demand for power resources has also increased rapidly. The traditional power grid can not be achieved operational status of equipment for effective monitoring and control, we will not be able to ensure the quality and quality of electricity service. Cloudbased build smart grid information platform enables gathering the parameter of grid operation state, greatly increasing the economic efficiency of power enterprises. Firstly, cloud computing and smart grid briefly described, and then pointed out that cloud computing applications in the smart grid information platform, and specialized of cloud-based smart grid information platform, in order for the power grid enterprises to enhance their level of service and provide effective reference. KEYWORD: cloud computing; smart power grids; information platform; Application 4th International Conference on Mechanical Materials and Manufacturing Engineering (MMME 2016) © 2016. The authors Published by Atlantis Press 202 3 THE MAIN PROBLEM IS STILL THE SMART GRID INFORMATION PLATFORM FACING The transition to a cloud computing data center. Our data center because of the different regions and the degree of information presents different characteristics, some data centers already have a basic cloud computing capabilities, but most data centers still remain in the initial level, to ensure the normal operation of the existing data center power under the premise, gradual transition to cloud computing data center in order to meet future requirements of the development of smart grid. Restricting data transmission problem. State Grid Corporation between provincial branches and there are a lot of data needs to be transmitted, will bring massive data transmission efficiency decreases, taking into account the load balancing device between the cloud node, thus building a wide area network channels to resolve data transmission problem. Security of data transmission. Data Center for Cloud Computing power companies to build internal private cloud, capable of self-maintenance and management, system wide area network data transmission services, according to the third security zone defined security partition, and install proprietary equipment production area and external public network isolation . 4 CLOUD COMPUTING APPLICATIONS IN THE SMART GRID INFORMATION POLICY PLATFORM Integrate and optimize the management of heterogeneous resources. Among the smart grid systems, data resources, information systems between various platforms and mutually dispersed, severe heterogeneity, thus leading to the integration of the quality of information sharing to increase difficulty, unable to effectively improve network management efficiency for the power distribution system electrical work quality management and marketing operations system maintenance upgrade will not be able to effectively meet demand (Jie Yang, 2014). Cloud computing technology enables the power system to effectively integrate the various business data and computing resources on the basis of information for business collaboration platform built on effective, able to satisfy the huge demand for smart grid information resource sharing. Cloud computing network virtualization and application virtualization technology, the server virtualization technology and other resources to achieve each service abstraction, and then realize the difference between the system and the software effectively eliminated by the appropriate method to achieve the effective integration of information resources, and then to optimize the management, greatly improving the efficiency of heterogeneous resources management applications. Distributed storage and management of large volume of data. After application of smart grid information platform construction, customer service requests varied, huge data volume information brought it to the storage and management of data provides a great challenge. Cloud computing technology in general face amount of data each smart grid system and unit collected using forms distributed storage of data to achieve the effective storage, greatly to ensure the security and reliability of data storage. GFS effective use of technology with each node, the master server is responsible for the effective management of the whole file system, the file system and complete the work of the metadata stored in the data block is the main server for data storage block sliced, and by saving three or more redundant backup storage. At the same time, client access and complete access to the main block server information. Fast Power System Parallel Computation and Treatment. Smart grids and information platform for the large volume of data to provide stable operation of the power grid data base, but more critical is the analysis and design of each system operating data, planning and decision-making, and even failures and transient stability, state estimation and intelligent a variety of computing and content analysis and decision-making process, its huge computing needs, general computing system can not reach its goal (Wang, Mengxue, 2015). Cloud computing technology through parallel computing technology has greatly enhanced parallel processing capability of the power system, which provides parallel programming model makes more simple and efficient parallel algorithms. MapReduce as Google made an important parallel programming model, and its role in large data sets in parallel computing plays a very prominent. Parallel processing job results by the client to achieve effective user submitted, via decomposition automated master role, the corresponding task scheduling to working nodes and nodes work completed by the specific calculations and perform the appropriate task. 5 SMART GRID INFORMATION PLATFORM CLOUD COMPUTING ANALYSIS Cloud Computing Smart Grid Information Platform Architecture. Smart grid technology information platform architecture cloud computing include the following several levels and aspects (Yan Cui, 2015): First, that the platform infrastructure layer is the integration of the various hardware resources and related management capabilities of virtualized using virtual technology to achieve the corresponding hardware devices such as computers and network storage devices abstraction, and then complete the automation and optimization of resource data and load management; secondly, the main platform for the integration of the various layers of software

Chapter 21 - Big Data Integration

The daily growing volume of electronic data poses challenges to traditional ways of organizing data storage, processing, and analysis. The feasibility of further research in this area is confirmed by the high demand for data storage and analytical data processing services. The article is devoted to the study of big data security issues. Despite earlier technological capabilities, the term or Big data, has only been actively used in the last decade. This term, originally coming from the exact sciences, programming, has caused and continues to cause a lot of controversy: what exactly is meant by Big data. In the course of the work, the specifics of Big data were described, the main characteristics of Big data were highlighted, and modern approaches to big data protection were analyzed. The role of Big data in the development of the digital economy and the need to address issues of their legal regulation are outlined. The following problems of Big data protection were formulated: limiting the speed of data access, organizing access via network protocols through General-purpose networks. A review of known developers of protection methods in these areas and recommendations for their application is made. Problems of legal regulation of Big data are revealed. The complexity of the problem, the need for multi-level protection and the use of modern methodological, theoretical and software developments from the first steps of working with Big data are shown.

The advent of information and communication technologies (ICT) ushers a cost-effective prospect to take care of large volumes of complex data, commonly known as “big data” in the supply chain operational environment. Big data is being generated today by web applications, social media, intelligent machines, sensors, mobile phones, and other smart handheld devices. Big data is characterized in terms of the velocity, volume, and variety with which it produces along the supply chain. This is due to recent advances in telecommunication networks along with centralized and decentralized data storage systems, which are processed thanks to modern digital computational capabilities. There is a growing interest in the use of this large volume of data and advanced analytics for diverse types of business problems in supply chain management (SCM). Such decision-support software applications employ pure mathematical techniques, artificial intelligence techniques, and sometimes uses both techniques to perform analytical operations that undercover relationships and patterns within supply chain generated big data. This chapter proposes a framework for the utilization of big data in SCM decision making. The framework is based on the SCOR (supply chain operations reference) model, which is endorsed by Supply Chain Council (SCC). The proposed framework is influenced by the enterprise potential of augmented reality and virtual reality in supply chain applications, and it identifies key categories of big data analytics applications for the key businesses of SCOR model. Finally, the chapter highlights research issues to extract insight from big data sources for enterprise decision making.

The new technology helps collect and store a considerable amount of information about consumers’ preferences and decisions in real-time. Thus, the unprecedented volume, speed and variety of primary data, Big Data, are available from all sources and individual consumers. The term “Big Data” is now widespread and accepted globally. The term Big Data has become increasingly known and used in many industries. Big Data is a new source of product development ideas, customer service, shelf location, distribution, dynamic pricing, etc. Big Data will have an impact on almost every area of marketing. Firms that do not develop the resources and capabilities to use Big Data capabilities effectively will be challenged to develop a sustainable competitive advantage to survive the Big Data revolution. Consumer analysis is at the heart of a Big Data revolution. Marketing specialists are beginning to recognize the importance of Big Data as the new capital and that access to Big Data offers a company new ways to differentiate its products. In this article, exploratory research has been conducted to highlight areas where Big Data allows organizations to fundamentally know about their business and translate it into better decision-making and improved performance in marketing and consumer behavior. Moreover, a global analysis (based on public data) was performed using large volumes of data in marketing and the main reasons they were stored and used.KeywordsBig DataMarketingBig Data’s growth

Public cloud infrastructure is widely used by enterprises to store and process big data. Cloud and its distributed computing phenomena not only provides scalable, available and affordable solution for storage and compute services but also raises security concerns. Many security solutions that came into existence encrypt data and allow accessing plaintext for data analytics in the confines of secure hardware. However, the fact remains that the large volumes of data is processed in distributed environment involving hundreds of commodity machines. There exist numerous communications between machines in MapReduce computing model. In the process, compromised MapReduce machines or functions are vulnerable to query based inference attacks on big data that lead to leakage of sensitive information. The main focus of this paper is to overcome the problem aforementioned. Towards this end, a methodology is proposed with an underlying algorithm for defeating query based inference attacks on big data in Hadoop. The proposed algorithm is known as Multi-Model Defence Against Query Based Inference Attacks (MMD-QBIA). A realistic attack model is considered for validating the effectiveness of the proposed methodology. Then an integrated framework for security and privacy to big data is evaluated. Cloudera Distribution Hadoop (CDH) is the environment used for empirical study. The experimental results revealed that the proposed solution prevents different kinds of query based inference attacks on big data besides security to big data in Hadoop MapReduce framework.

At this time big data, with large data volumes that continue to grow and become complex, is a challenge and opportunity for governments and organizations. This is said to be a challenge considering that the increase in data volume will produce a lot and variety of information, thus requiring efficient ways of storing and managing data. Big data is said to be an opportunity considering that the magnitude of the study results and proper information management will open up new opportunities to recognize the potential for improving services in various government and organizational sectors. Big data can open up opportunities to increase knowledge about various aspects of people's lives so that it can encourage the creation of new trends and innovations in various fields, improve and optimize operations, and determine future conditions so that governments and organizations can produce appropriate basic products. In the future, big data will change the way governments and organizations run their operations, creating challenges and opportunities for decision-makers. Even so, in the future big data can be utilized more optimally for the benefit of the wider community. In the future, big data has great potential to be used in many sectors to improve people's welfare. By using the right methods, big data can be very valuable in improving the welfare and quality of life of the wider community. Therefore, to be able to implement big data to improve the welfare and quality of life of the wider community, it is necessary to understand the challenges and opportunities of big data in the development of digital technology.

Exploiting the cloud storage hierarchy both within and across data-centers of different cloud providers empowers Internet applications to choose data centers (DCs) and storage services based on storage needs. However, using multiple storage services across multiple data centers brings a complex data placement problem that depends on a large number of factors including, e.g., desired goals, storage and network characteristics, and pricing policies. In addition, dynamics e.g., changing user locations and access patterns, make it impossible to determine the best data placement statically. In this paper, we present TripS, a lightweight system that considers both data center locations and storage tiers to determine the data placement for geo-distributed storage systems. Such systems make use of TripS by providing inputs including SLA, consistency model, fault tolerance, latency information, and cost information. With given inputs, TripS models and solves the data placement problem using mixed integer linear programming (MILP) to determine data placement. In addition, to adapt quickly to dynamics, we introduce the notion of Target Locale List (TLL), a pro-active approach to avoid expensive re-evaluation of the optimal placement. The TripS prototype is running on Wiera, a policy driven geo-distributed storage system, to show how a storage system can easily utilize TripS for data placement. We evaluate TripS/Wiera on multiple data centers of AWS and Azure. The results show that TripS/Wiera can reduce cost 14.96% ∼ 98.1% based on workloads in comparison with other works' approaches and can handle both short- and long-term dynamics to avoid SLA violations.

Computing in the cloud is one of the platforms that may be used to provide distributed computing resources. Supplying and managing cloud resources most effectively is referred to as resource management. A recent development in technology known as fog computing is an example of an expanded and dispersed infrastructure. This architecture maintains application processes between end devices and the network edge to provide more dependable and efficient services. These services include remote data storage, allowing customers to access their data from a distant location. Providing remote storage service is an advantageous function offered by cloud suppliers. On the other hand, the data stored in the cloud is geographically dispersed and kept in various data centers, significantly increasing the risk to users' privacy and security. One of the problems that might arise with privacy is when many data centers store the same information. Many cloud service providers check their customers' data using a Third-Party Auditor (TPA) to address concerns about client privacy and data integrity. Currently, most trusted TPAs only have one validator, making it impossible to expand the data integrity across several data centers. The various verifiers used by TPAs have been reduced in number in response to Man in the Cloud (MiTC) attacks. As a result, they cannot check and authenticate the integrity of data stored in several data centers. A unique Peer to Peer (P2P) authentication protocol with Certificate Authority (CA) and Data Storage Protocol is presented as a solution to the problem that has been outlined above to check for and go around any issues that may arise (DSP). The efficiency of the proposed protocol is demonstrated by the incorporation of TPAs and Certificate Authorities. The proposed protocol has been tested with a single user and a single storage server, as well as multiple storage servers in ownCloud with one backup server, two storage servers, three clients, and two TPAs. The NoSQL server in an organization's cloud is set up to save data to storage servers in the appropriate format. The Amanda backup server is used to back up the mirror copy of the stored data on the storage servers. Automated Validation of Internet Security Protocols with Data Fusion and Applications, or AVISPA for short, is a technology that may be used to verify data stored in the cloud. The findings make it abundantly evident that the suggested protocol is strong enough to guarantee the authenticity of data kept in several data centers.

Big data is the data set which contains variety of data which increases on a daily basis in an organisation and also it has large volume of data with high velocity. It is the complex form of data which is usually identified from new set of data sources. It is widely used in most of the companies as it will have latest data which is useful to solve critical issues in the business or an organisation. BIG data has such large volume of data that it cannot be handled by simple data handling software and need latest technology software to handle such large volumes of data. The amount of data present in big data is so complex and large in volume that it needs skilled professionals to manage the data and requires efficient efforts to maintain it. Small businesses cannot afford to implement BIG data in their business due to its complex nature and also the cost factor is higher as compared to traditional data software. The paper will explain the various categories of big data, its importance, strategies and implementation process.

The volume of data in the field of Earth data observation has increased considerably, especially with the emergence of new generations of satellites and models providing much more precise measures and thus voluminous data and files. One of the most traditional and popular data formats used in scientific and education communities (reference) is the NetCDF format. However, it was designed before the development of cloud storage and parallel processing in big data architectures. Alternative solutions, under open source or under proprietary licences, appeared in the past few years (See Rasdaman, Opendatacube). These data cubes are managing the storage and the services for an easy access to the data but they are also altering the input information applying conversions and/or reprojections to homogenize their internal data structure, introducing a bias in the scientific value of the data. The consequence is that it drives the users in a closed infrastructure, made of customized storage and access services. The objective of this study is to propose a light new open source solution which is able to store gridded datasets into a native big data format and make data available for parallel processing, analytics or artificial intelligence learning. There is a demand for developing a unique storage solution that would be opened to different users: Scientists, setting up their prototypes and models in their customized environment and qualifying their data to publish as Copernicus datatsets for instance; Operational teams, in charge of the daily processing of data which can be run in another environment, to ingest the product in an archive and make it available to end-users for additional model and data science processing. Data ingestion and storage are key factors to study to ensure good performances in further subsetting access services and parallel processing. Through typical end users’ use cases, four storage and services implementations are compared through benchmarks: Unidata's THREDDS Data Server (TDS) which is a traditional NetCDF data access service solution built on the NetCDF-Java, an extension of the THREDDS Data Server using object store, pangeo/Dask/Python ecosystem, and the alternative Hadoop/Spark/Parquet solution, driven by CLS technical and business requirements.

As we are living in the era of big data, huge volumes of a wide variety of complex data-which can be of different levels of veracity-are generated or collected at a high velocity from rich sources of data in various real-life applications. A rich source of these big data sources is the Internet of Things (IoT), which include a collection of sensors, smartphones and other mobile devices, wearable devices, as well as other "things" that are capable to operate within the existing Internet infrastructure. Embedded in these big data are valuable knowledge and useful information. Hence, the research problem of data mining from big IoT data have drawn attention of many researchers as it aims to discover implicit, previously unknown and potentially useful information and knowledge from the data. For instance, frequent pattern mining finds sets of frequently co-occurring items in the IoT domains. Associative classification discovers rules revealing relationships among items within the frequent patterns and their associations with the corresponding class labels. Induction based classification uses decision tree or random forest to learn from old big IoT for classifying or making predictions on new data. Over the past quarter of a century, many serial, distributed, parallel, and MapReduce-based (Hadook-based and Spark-based) big data mining algorithms have been proposed. These algorithms are run in local computers, distributed and parallel environments, clusters, grids, clouds and/or data centers. In this paper, we review some of these algorithms, discuss issues and research prospective in mining classification patterns from these big IoT data in fog. Our case study on a real-life application shows the feasibility on classifying real-life big IoT data over fog for urban analytics.

The amount of data has increased exponentially as a consequence of the availability of new data sources and the advances in data collection and storage. This data explosion was accompanied by the popularization of the Big Data term, addressing large volumes of data, with several degrees of complexity, often without structure and organization, which cannot be processed or analyzed using traditional processes or tools. Moving towards Big Data Warehouses (BDWs) brings new problems and implies the adoption of new logical data models and tools to query them. Hive is a DW system for Big Data contexts that organizes the data into tables, partitions and buckets. Several studies have been conducted to understand ways of optimizing its performance in data storage and processing, but few of them explore whether the way data is structured has any influence on how quickly Hive responds to queries. This paper investigates the role of data organization and modelling in the processing times of BDWs implemented in Hive, benchmarking multidimensional star schemas and fully denormalized tables with different Scale Factors (SFs), and analyzing the impact of adequate data partitioning in these two data modelling strategies.