Chapter 21 - Big Data Integration

The volume of the coming data in HEP is growing. The volume of the data to be held for a long time is growing as well. Large volume of data – big data – is distributed around the planet. The methods, approaches how to organize and manage the globally distributed data storage are required. The distributed storage has several examples for personal needs like own-cloud.org, pydio.com, seafile.com, sparkleshare.org. For enterprise-level there is a number of systems: SWIFT - distributed storage systems (part of Openstack), CEPH and the like which are mostly object storage. When several data center’s resources are integrated, the organization of data links becomes very important issue especially if several parallel data links between data centers are used. The situation in data centers and in data links may vary each hour. All that means each part of distributed data storage has to be able to rearrange usage of data links and storage servers in each data center. In addition, for each customer of distributed storage different requirements could appear. The above topics are planned to be discussed in data storage proposal.

Big data is the data set which contains variety of data which increases on a daily basis in an organisation and also it has large volume of data with high velocity. It is the complex form of data which is usually identified from new set of data sources. It is widely used in most of the companies as it will have latest data which is useful to solve critical issues in the business or an organisation. BIG data has such large volume of data that it cannot be handled by simple data handling software and need latest technology software to handle such large volumes of data. The amount of data present in big data is so complex and large in volume that it needs skilled professionals to manage the data and requires efficient efforts to maintain it. Small businesses cannot afford to implement BIG data in their business due to its complex nature and also the cost factor is higher as compared to traditional data software. The paper will explain the various categories of big data, its importance, strategies and implementation process.

Due to its historical nature, data warehouses require that large volumes of data need to be stored in their repositories. Some organizations are beginning to have problems to manage and analyze these huge volumes of data. This is due, in large part, to the relational databases which are the primary method of data storage in a data warehouse, and start underperforming, crumbling under the weight of the data stored. In opposition to these systems, arise the NoSQL databases that are associated with the storage of very large volumes of data inherent to the Big Data paradigm. Thus, this article focuses on the study of the feasibility and the implications of the adoption of a NoSQL database, within the data warehousing context. MongoDB was selected to represent the NoSQL systems in this investigation. In this paper will be explained the processes required to design the structure of a data warehouse and typically dimensional queries in the MongoDB system. The undertaken research culminates in the performance analysis of queries executed in a traditional data warehouse, based on the SQL Server system, and an equivalent data warehouse based on the MongoDB system.

The new technology helps collect and store a considerable amount of information about consumers’ preferences and decisions in real-time. Thus, the unprecedented volume, speed and variety of primary data, Big Data, are available from all sources and individual consumers. The term “Big Data” is now widespread and accepted globally. The term Big Data has become increasingly known and used in many industries. Big Data is a new source of product development ideas, customer service, shelf location, distribution, dynamic pricing, etc. Big Data will have an impact on almost every area of marketing. Firms that do not develop the resources and capabilities to use Big Data capabilities effectively will be challenged to develop a sustainable competitive advantage to survive the Big Data revolution. Consumer analysis is at the heart of a Big Data revolution. Marketing specialists are beginning to recognize the importance of Big Data as the new capital and that access to Big Data offers a company new ways to differentiate its products. In this article, exploratory research has been conducted to highlight areas where Big Data allows organizations to fundamentally know about their business and translate it into better decision-making and improved performance in marketing and consumer behavior. Moreover, a global analysis (based on public data) was performed using large volumes of data in marketing and the main reasons they were stored and used.KeywordsBig DataMarketingBig Data’s growth

Big Data is used to store huge volume of both structured and unstructured data which is so large and is hard to process using current / traditional database tools and software technologies. The goal of Big Data Storage Management is to ensure a high level of data quality and availability for business intellect and big data analytics applications. Graph database which is not most popular NoSQL database compare to relational database yet but it is a most powerful NoSQL database which can handle large volume of data in very efficient way. It is very difficult to manage large volume of data using traditional technology. Data retrieval time may be more as per database size gets increase. As solution of that NoSQL databases are available. This paper describe what is big data storage management, dimensions of big data, types of data, what is structured and unstructured data, what is NoSQL database, types of NoSQL database, basic structure of graph database, advantages, disadvantages and application area and comparison of various graph database.

Data quality is paramount in data-driven decision-making processes, especially when dealing with large volumes of data in environments like data warehouses and data lakes. These systems store vast amounts of raw and processed data from multiple sources, making data management and quality assurance complex yet critical. With the growing adoption of Artificial Intelligence (AI), new techniques and tools have emerged that can significantly enhance data quality. This paper discusses how AI can improve the quality of data within both data warehouses and data lakes by automating data cleansing, validation, anomaly detection, and ensuring consistency. It explores the benefits, challenges, and methodologies for integrating AI tools into these systems. Keywords: Data quality, AI in data quality, data warehouse, data lakes, big data, data processing, data cleansing, data profiling.

Big Data OverviewDriven by the need to generate business value, the enterprise has started to adopt Big Data as a solution, migrating from the classical databases and data stores which lack the flexibility and are not optimized enough [1].The changes in the environment make big data analytics attractive to all types of organizations, while the market conditions make it practical. The combination of simplified models for development, commoditization, a wider palette of data management tools, and low-cost utility computing has effectively lowered the barrier to entry. [2]. The concept addresses large volumes of complex data, rapid growing data sets that may come from different autonomous sources.In recent approaches, Big Data is characterized by principles known as the 4VVolume, Variety, Velocity and Veracity [3]. There are opinions about accepting other principles as Big Data characteristics, such as Value.Each day more businesses realize that Big Data is relevant as the applications generate large volumes of data generated automatically, from different data sources, centralized or autonomous. As traditional databases hit limitations when the need of analyzing this data, dedicated solutions must be considered.Important BigData Solutions:* Apache HBase/Hadoop is based on Google's BigTable distributed storage system, which runs on top of Hadoop as a distributed and scalable big data store. This means that HBase can leverage the distributed processing paradigm of the Hadoop Distributed File System (HDFS) and benefit from Hadoop's MapReduce programming model. It combines the scalability of Hadoop with real-time data access as a key/value store and deep analytic capabilities of Map Reduce [4].HBase allows to query for individual records as well as derive aggregate analytic reports across a massive amount of data. It can host large tables with billions of rows, millions of columns and run across a cluster of commodity hardware. HBase is composed of three types of servers in a master slave type of architecture. Region servers are responsible to serve data for reads and writes. When accessing data, clients communicate with HBase Region Servers directly. Region assignment, DDL (create, delete tables) operations are handled by the HBase Master process.* Apache Cassandra is a distributed database used for the administration and management of large amounts of structured data across multiple servers, while providing highly available service and no single point of failure. It provides features such as continuous availability, linear scale performance, data distribution across multiple data centers and cloud availability zones. Cassandra inherits its data architecture from Google's BigTable and it borrows its distribution mechanisms from Amazon's Dynamo.The nodes in a Cassandra cluster are completely symmetrical, all having identical responsibilities. Cassandra also employs consistent hashing to partition and replicate data. it has the capability of handling large amounts of data and thousands of concurrent users or operations per second across multiple data centers.Cassandra has a hierarchy of caching mechanisms and carefully orchestrated disk I/O which ensures speed and data safety. Write operations are sent first to a persistent commit log (ensuring a durable write), then to a write-back cache called a memTable; when the memTable fills, it is flushed to a sorted string table - SSTable - on disk. A Cassandra cluster is organized as a ring, and it uses a partitioning strategy to distribute data evenly.* Redis represents an in-memory data structure store used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperlogs and geospatial indexes with radius queries. Redis stores all data in RAM, allowing lightning fast reads and writes. It runs extremely efficiently in memory and handles high-velocity data, needing simple standard servers to deliver millions of operations per second with submillisecond latency. …

The advent of information and communication technologies (ICT) ushers a cost-effective prospect to take care of large volumes of complex data, commonly known as “big data” in the supply chain operational environment. Big data is being generated today by web applications, social media, intelligent machines, sensors, mobile phones, and other smart handheld devices. Big data is characterized in terms of the velocity, volume, and variety with which it produces along the supply chain. This is due to recent advances in telecommunication networks along with centralized and decentralized data storage systems, which are processed thanks to modern digital computational capabilities. There is a growing interest in the use of this large volume of data and advanced analytics for diverse types of business problems in supply chain management (SCM). Such decision-support software applications employ pure mathematical techniques, artificial intelligence techniques, and sometimes uses both techniques to perform analytical operations that undercover relationships and patterns within supply chain generated big data. This chapter proposes a framework for the utilization of big data in SCM decision making. The framework is based on the SCOR (supply chain operations reference) model, which is endorsed by Supply Chain Council (SCC). The proposed framework is influenced by the enterprise potential of augmented reality and virtual reality in supply chain applications, and it identifies key categories of big data analytics applications for the key businesses of SCOR model. Finally, the chapter highlights research issues to extract insight from big data sources for enterprise decision making.

The volume of data in the world is growing very fast and generated from verity of sources like social media, sensors airline industry or scientific data in different formats. Biggest challenge is how to infer meaningful insights from such a varietyful and big data along with concern of data storage and management of fast growing data. The size of the databases used in today’s enterprises has been growing at exponential rates day by day. Hence, industries requirement to quickly process and analyze the big data volumes for business decision making and customer insights has also grown exponentially. Data pouring from various sources may be can be structured or unstructured in nature. Structured data refers to a relatively well-organized information, which can be further inserted into traditional RDBMS. As Traditional RDBMS are efficient and easy queries by simple, straightforward search algorithms or SQL queries. In contrast to structured data, unstructured data can be considered as information, which does not, comes in a pre-defined data format, well organized data storage model, or cannot be stored well into relational tables. It is assumed to be fastest growing type of data, e.g. image, sensors data, web chats, social networking messaging data, video, documents, log files, and email data. There are many techniques and software available, which can process and provide efficient storage of unstructured data and help organization to perform analytics on unstructured data. Unstructured data does not well-organized and not stored in predefined manner e.g. logs, web chats. The variety and on ordered nature of data makes storage methods and structure makes execution a time and resource-consuming affair. Advancement into technology has open floodgates to push huge volume of unstructured type of data. Multimedia data is one of the example of unstructured big data, which spans all over the Internet. This needs high execution capability to extract useful information. Rapid processing of multimedia data such as video is important for e.g. criminal investigations, surveillance monitoring, news analysis, sports analytics domain, emotion extraction, etc. Hence, analysis of multimedia data in minimum timeframe is one of the latest research areas. Therefore, we have researched techniques for analyzing unstructured data to extract meaningful information hidden in the big data. In addition, we will describe about various techniques and software used to Manage, process unstructured big data in efficient manner, and increases the performance of complexity analysis.

As cloud computing continues to evolve, organizations are finding new ways to store the massive amounts of big data that are collected. Big data storage often require greater flexibility and scalability which can be provided by incorporating NoSQL technologies. NoSQL (Not Only SQL) is quickly becoming a popular approach to store large and unstructured data. This paper looks at the various classifications of NoSQL technologies as well as many of the notable characteristics of the technologies. The authors also describe some deficiencies of using NoSQL and give some explanation to why companies are adopting the technology. The paper concludes with suggestions for future research of NoSQL technologies and a content analysis of current articles in database management is provided in the appendix. Keywords: big data, cloud computing, document databases, MongoDB, NoSQL, Not Only SQL, unstructured database INTRODUCTION Since NoSQL database technologies entered the market in 1998, these technologies have challenged the widely-accepted relational structures and database management practices supported by traditional SQL databases. Despite this trend, the perception that NoSQL seeks to eliminate the use of relational databases is a common misconception. NoSQL is best considered with the acronym NOSQL - Not Only SQL - which more accurately represents an approach that combines non-relational databases with the use of relational ones. This approach seeks to leverage both NoSQL and SQL technologies in order to balance the demands of performance, scalability, and schema flexibility with data integrity and consistency (Purdue, 2014; Penchikala, 2013). The rise of web-scale applications creates performance, scalability, and schema flexibility demands that present significant challenges for traditional SQL / relational databases. Thus, a variety of new entrants have entered the market, each seeking to address specific challenges, including (Kent, 2012): * The need to store large volumes of data about users, objects and products * The need to provide near-real-time transaction processing * The need to provide high-speed analytics on large volumes of data (big data) * The need to support flexible data schemas for evolving business environments * The need to scale applications horizontally with predictable hardware and licensing cost structures Each NoSQL solution seeks to address one or more of the above issues, often by providing alternative approaches to long-established relational database norms. Traditional relational databases were not designed to cope with the scalability and flexibility challenges facing modern applications. Older databases were not designed to take advantage of the inexpensive storage and excess processing power available today. The defining characteristics of NoSQL technologies, as well as the relative advantages and disadvantages of NoSQL, and the subsequent influence on adoption of NoSQL technologies are each the subject of a small, yet growing body of research. This paper provides a literature review of that research and proposes potential areas where future research may focus. REVIEW OF LITERATURE NoSQL database technology is a relatively recent innovation and much of the research surrounding it is of an emerging nature. For the purpose of this literature review Galileo was primarily used for identifying the articles. Articles were selected that focused directly on NoSQL database technologies (unstructured / semi-structured databases, document databases, etc.) and articles that focused on relational databases. The contrasting articles provided the researchers with the ability to compare the features that are offered with a greater degree of variability in NoSQL databases (for example, an article on SQL scalability provided a foundation for comparing newer horizontal scaling techniques used in certain NoSQL databases). …

The rapid increase in the number of aerial and orbital Earth observation systems is generating a huge amount of remote sensing data that need to be readily transformed into useful information for policy and decision makers. This exposes an urgent demand for image interpretation tools that can deal efficiently with very large volumes of data. In this work, we introduce a set of methods that support distributed processing of georeferenced raster and vector data in a computer cluster, which may be a virtual cluster provided by cloud computing infrastructure services. The set of methods comprise a particular technique for indexing distributed georeferenced datasets, as well as strategies for distributing efficiently the processing of spatial context-aware operations. They provide the means for the development of scalable applications, capable of processing large volumes of geospatial data. We evaluated the proposed methods in a remote sensing image interpretation application, built on the MapReduce framework, and executed in a cloud computing infrastructure. The experimental results corroborate the capacity of the methods to support efficient handling of very large earth observation datasets.

Privacy preserving multi-party computation delegation for deep learning in cloud computing

Marine data acquisition and processing has generated new concepts in velocity analysis, amplitude analysis, and frequency analysis. This has been possible because of consistent surface conditions on marine data, development of direct detection of hydrocarbon concepts, and improved computing power. Exploitation of the correlation of time, amplitude, velocity, and frequency relations has been largely confined to time, amplitude, and velocity. Such basic seismic theories as dispersion, fluidity, and multiple energy indicate a strong relation between frequency and velocity. Experiments on synthetic data illustrates an additional interdependence between apparent velocity and frequency. Both the theoretical assumptions and the synthetic exercises are reenforced by experiments on seismic data. Overlays of continuous velocity analysis and continuous frequency analysis show a substantial correlation. Frequency analysis in a seismic section dimensionally compatible contour allows the correlation with velocity on large volumes of data. An example on Gulf Coast data indicates good potential for this type of analysis. INTRODUCTION The advent of marine seismic recording in conjunction with increased energy needs resulted in a very large volume of seismic data. It was also marine data which generated improved methods of rock velocity estimation. Direct detection of hydrocarbons motivated attention toward such neglected seismic observations as phase and frequency anomalies. Time relations continue to be the basis of interpretation of structure but stratigraphic exploration and direct detection of hydrocarbons brought amplitude to the fore as a meaningful and interpretable measure. Amplitude, while difficult on land data, is reasonably consistent on marine data due to the relatively steady surface conditions at sea. Analysis of both velocity and frequency began to receive more attention for the same reason. Interpretation techniques on seismic data broadened to include frequency and velocity considerations. Time, amplitude, velocity, and frequency are all directly related to the earth response. Anomalies in the seismic data usually will create anomalies in some of these basic parameters so that they are at least indirectly related to each other. Some formulas relating these measures are well known and much utilized. The best known such equation is perhaps the "moveout" equation directly relating time and velocity: (Available In Full Paper) The relation between frequency and velocity or frequency and amplitude is also direct but seldom used in data processing. However, any discussion of compressional wave theory must include frequency as a very basic parameter. Some existing seismic theory does relate velocity and frequency but actual measurement on real data is scant. It is proposed to consider their possible relationship in the context of geophysical theory and practical observation. VELOCITY DISPERSION One accepted theory, which directly relates frequency and velocity is dispersion. Residual attenuation of the seismic wavefront after distance and reflection factors are removed is proportional to frequency. The equation which governs this dependence of velocity on frequency according to O'Brien and Lucas, 1971, is: (Available In Full Paper) It is indicated by the equation that if the attenuation, Q, is known, the velocity at each frequency could be computed.

At this time big data, with large data volumes that continue to grow and become complex, is a challenge and opportunity for governments and organizations. This is said to be a challenge considering that the increase in data volume will produce a lot and variety of information, thus requiring efficient ways of storing and managing data. Big data is said to be an opportunity considering that the magnitude of the study results and proper information management will open up new opportunities to recognize the potential for improving services in various government and organizational sectors. Big data can open up opportunities to increase knowledge about various aspects of people's lives so that it can encourage the creation of new trends and innovations in various fields, improve and optimize operations, and determine future conditions so that governments and organizations can produce appropriate basic products. In the future, big data will change the way governments and organizations run their operations, creating challenges and opportunities for decision-makers. Even so, in the future big data can be utilized more optimally for the benefit of the wider community. In the future, big data has great potential to be used in many sectors to improve people's welfare. By using the right methods, big data can be very valuable in improving the welfare and quality of life of the wider community. Therefore, to be able to implement big data to improve the welfare and quality of life of the wider community, it is necessary to understand the challenges and opportunities of big data in the development of digital technology.

Teaching Digital Analytics with Google’s AEK Hairong Li (bio) Advertising education is in constant evolution to better meet the needs of future professionals in an ever-changing advertising business. With the proliferation of digital media, vast amounts of user data are cumulated as a byproduct of normal media consumption, and by appropriate privacy protection, they are a gold mine for consumer insights. When users visit a website, search for a product online, make a post on Facebook, buy a product online, or use a location-based service from a smartphone, they leave their digital footprints. Such real-time data have greatly benefited the operations of numerous websites and mobile apps, the services of major research firms such as Nielsen and comScore, and the business of a new generation of data aggregators, warehouses, and analytics. As a result, Big Data has played a greater role in the creation, placement, and evaluation of advertising, especially in the ways of consumer engagement. Consumer insights—through often-real-time digital analytics that show how consumers search for product information, what makes them like a product, what they say online, and where they buy a product—are shaping almost every aspect of advertising practice. Thus, the knowledge about and skills in collecting, processing, managing, analyzing, and reporting large volumes of consumer data have become a sought-after competency of advertising students. It is imperative to develop curricula and conduct research on Big Data-related topics such as computational advertising, social listening, programmatic buying, and advertising analytics, as some advertising programs in the US and other countries have already started. For example, a pre-conference on Big Data for advertising research and education was held at the American Academy of Advertising (AAA) conference in Atlanta, Georgia, on March 27, 2014, where expert panelists from industry and academia explored how Big Data approaches could be introduced in advertising research and education.1 During the first-of-its-kind pre-conference, panelists shared academic studies with Big Data methods, cases of using Big Data in advertising campaigns, possible topics of computational advertising courses, and perspectives on emerging issues. The large audience at the full-day pre-conference indicated a wide interest among the AAA members. As a follow-up, a pre-conference on computational advertising was planned for the AAA conference in Boston, Massachusetts, on March 23, 2017, with digital analytics as a focus for presentations and discussion.2 Early efforts indicate, however, that teaching digital analytics in the classroom is challenging, because shareable materials to build a fundamental understanding of analytics, including real world business data and the opportunity to perform a diverse range of analysis using an analytics tool, are not easily accessible to most advertising instructors and students. The Purpose Thus, the purpose of this article is to share our experience in developing and teaching a digital analytics course for undergraduate advertising majors. This online course covered the basic concepts of digital analytics and actual use of live data to generate consumer insights on key aspects of e-commerce. Digital analytics is a large and complex subject, and it includes web analytics, mobile analytics, and social analytics. This introductory course was designed to focus on web analytics, but it also covered parts of mobile analytics and social analytics that were directly related to web analytics. For example, traffic sources from mobile devices and social media platforms were explored, as they are becoming increasingly important in understanding multichannel users. The course objectives were for students to acquire knowledge about the mechanism of digital analytics, the ability to organize, interpret, and report data for consumer insights, and the skills to effectively use digital analytics tools to complete assigned tasks. This knowledge, together with these abilities and skills, should prepare students for an entry level data analyst position in advertising agencies, media outlets, and client companies. Students would get access to a real e-commerce site that was equipped with Google Analytics to practice with live data, using Google’s Analytics Education Kit (AEK), an education outreach program that is to be officially released in 2017. Analytics Education Kit (AEK) The course was facilitated with Google’s AEK. The AEK provides a number of resources to overcome...