Software Artifact Traceability in Big Data Systems

Big Data Systems are becoming increasingly complex and generally have very high operational costs. Cloud computing offers attractive solutions for managing large scale systems. However, one of the major bottlenecks in VM performance is virtualized I/O. Since Big Data applications and middleware rely heavily on high performance interconnects such as InfiniBand, the performance of virtualized InfiniBand interfaces is vital. Single Root I/O Virtualization (SR-IOV) is a hardware based approach which offers significant performance benefits as compared to software based I/O virtualization. With the increasing adoption of InfiniBand network for cloud computing, it is important to evaluate the performance benefits of SR-IOV for InfiniBand networks; especially to see the performance characteristics of Big Data applications and middleware under different scenarios. We characterize the main performance factors for different workloads through this study (such as map task scheduling, I/O, data replication, etc.). Our experimental evaluations show that the performance difference for a wide set of Big Data benchmarks and applications over SR-IOV with InfiniBand using RDMA-enabled Hadoop as compared to native InfiniBand network is just 5 -- 15%. In addition, with RDMA-enabled Hadoop, we see 20.9 -- 81.6% performance improvement for RDMA as compared to IPoIB.

Big Sensor Data Applications in Urban Environments

Cloud computing and big data: Technologies and applications

Fault tolerance is of great importance for big data systems. Although several software-based application-level techniques exist for fault security in big data systems, there is a potential research space at the hardware level. Big data needs to be processed inexpensively and efficiently, for which traditional hardware architectures are, although adequate, not optimum for this purpose. In this paper, we propose a hardware-level fault tolerance scheme for big data and cloud computing that can be used with the existing software-level fault tolerance for improving the overall performance of the systems. The proposed scheme uses the concurrent error detection (CED) method to detect hardware-level faults, with the help of Scalable Error Detecting Codes (SEDC) and its checker. SEDC is an all unidirectional error detection (AUED) technique capable of detecting multiple unidirectional errors. The SEDC scheme exploits data segmentation and parallel encoding features for assigning code words. Consequently, the SEDC scheme can be scaled to any binary data length “n” with constant latency and less complexity, compared to other AUED schemes, hence making it a perfect candidate for use in big data processing hardware. We also present a novel area, delay, and power efficient, scalable fault secure checker design based on SEDC. In order to show the effectiveness of our scheme, we (1) compared the cost of hardware-based fault tolerance with an existing software-based fault tolerance technique used in HDFS and (2) compared the performance of the proposed checker in terms of area, speed, and power dissipation with the famous Berger code and m-out-of-2m code checkers. The experimental results show that (1) the proposed SEDC-based hardware-level fault tolerance scheme significantly reduces the average cost associated with software-based fault tolerance in a big data application, and (2) the proposed fault secure checker outperforms the state-of-the-art checkers in terms of area, delay, and power dissipation.

The objective of this book is to introduce the basic concepts of big data computing and then to describe the total solution of big data problems using HPCC, an open-source computing platform. The book comprises 15 chapters broken into three parts. The first part, Big Data Technologies, includes introductions to big data concepts and techniques; big data analytics; and visualization and learning techniques. The second part, LexisNexis Risk Solution to Big Data, focuses on specific technologies and techniques developed at LexisNexis to solve critical problems that use big data analytics. It covers the open source High Performance Computing Cluster (HPCC Systems) platform and its architecture, as well as parallel data languages ECL and KEL, developed to effectively solve big data problems. The third part, Big Data Applications, describes various data intensive applications solved on HPCC Systems. It includes applications such as cyber security, social network analytics including fraud, Ebola spread modeling using big data analytics, unsupervised learning, and image classification. The book is intended for a wide variety of people including researchers, scientists, programmers, engineers, designers, developers, educators, and students. This book can also be beneficial for business managers, entrepreneurs, and investors.

This chapter revises the most important aspects in how computing infrastructures should be configured and intelligently managed to fulfill the most notably security aspects required by Big Data applications. One of them is privacy. It is a pertinent aspect to be addressed because users share more and more personal data and content through their devices and computers to social networks and public clouds. So, a secure framework to social networks is a very hot topic research. This last topic is addressed in one of the two sections of the current chapter with case studies. In addition, the traditional mechanisms to support security such as firewalls and demilitarized zones are not suitable to be applied in computing systems to support Big Data. SDN is an emergent management solution that could become a convenient mechanism to implement security in Big Data systems, as we show through a second case study at the end of the chapter. This also discusses current relevant work and identifies open issues.

Big data has evolved as a most challenging area in scientific study and research. It has drawn much attention during the last few years. It influences our modern society, business, government, healthcare, research and almost every discipline. In this data-driven era, where data is continuously acquired from a verity of sources for different purposes, the ability to make timely decisions based on available data is a very critical task. The massive data size, variety, velocity, accuracy and high dimensionality presents new challenges to big data. This paper attempts to present some challenges of big data. In addition, a study on the conceptual design of big data architecture presented on specific big data applications. In this paper, we presented research work on big data analytics techniques in the areas of text, sentiment, video, social media and predictive analytics. A comparative analysis on selected big data applications has been presented in detail.

Big data applications are becoming more popular across many different fields and civil engineering is no different. The benefits of big data lie in its potential to provide valuable insights into various large datasets. Big data applications can identify patterns and trends that were previously unknown, which can help them make informed decisions and develop effective strategies. In the case of civil engineering, this could be taking large datasets that have been produced in relation to pieces of infrastructure and using them to create more efficient management strategies. One of the issues with using big data is that if the input dataset is flawed then the output resulting from any big data analysis will be compromised. Therefore, data validation is used, which is the process of ensuring that data is accurate, complete, and consistent. The consequence of not undertaking data validation may be Inaccurate or inconsistent data which can lead to incorrect insights and decisions in big data applications. This paper explores the necessity of validating data before it is used in a big data application. It outlines some of the different methods used for validating data and provides an overview of the potential issues that may arise from data validation errors. A case study is then presented showing the process of validation on data collected from four bridges and provides recommendations for implementing data validation as part of a larger big data workflow. The results of the case study show that validation of data is an important step in the big data process both for confidence in the outputs and to make big data applications more useful and more common in the civil engineering field. The discussion and presented case study in this paper highlight the necessity of validating data. It has shown some of the potential issues that may arise from not undertaking data validation.

PurposeThe purpose of this paper is to see how critical and vital artificial intelligence (AI) and big data are in today’s world. Besides this, this paper also seeks to explore qualitative and theoretical perspectives to underscore the importance of AI and big data applications in multi-sectoral scenarios of businesses across the world. Moreover, this paper also aims at working out the scope of ontological communicative perspectives based on AI alongside emphasizing their relevance in business organizations that need to survive and sustain with a view to achieve their strategic goals.Design/methodology/approachThis paper attempts to explore the qualitative perspectives to build a direction for strategic management via addressing the following research questions concerned with assessing the scope of ontological communicative perspectives in AI relevant to business organizations; exploring benefits of big data combined with AI in modern businesses; and underscoring the importance of AI and big data applications in multi-sectoral scenarios of businesses in today’s world. Employing bibliometric analysis along with NVivo software to do sentiment analysis, this paper attempts to develop an understanding of what happens when AI and big data are combined in businesses.FindingsAI and big data have tremendous bearing on modern businesses. Because big data comprises enormous information of diverse sorts, AI-assisted machines, tools and devices help modern businesses process it quickly, efficiently and meaningfully. Therefore, business leaders and entrepreneurs need to focus heavily on ontological and communicative perspectives to deal with diverse range of challenges and problems particularly in the context of recent crises caused by COVID-19 pandemic.Research limitations/implicationsThere is hardly any arena of human activity wherein AI and big data are not relevant. The implication of this paper is that of combining both well so that we may find answers to the difficult and challenging multi-sectoral scenarios concerning not just businesses but life at large. Moreover, automated tools based on AI such as natural language processing and speech to text also facilitate meaningful communication at various levels not just in business organizations but other fields of human activities as well.Social implicationsThis paper has layered social implications, as it conceptually works out as to how strategically we may combine AI and big data to benefit modern business scenarios dealing with service providers, manufacturers, entrepreneurs, business leaders, customers and consumers. All the stakeholders are socio-culturally and contextually rooted/situated, and that is how this study becomes socially relevant.Originality/valueThis paper is an original piece of research and has been envisioned in view of the challenging business scenarios across the world today. This paper underscores the importance of strategically combining AI and big data, as they have enormous bearing on modern businesses. The insights arrived at in this paper have implications for business leaders and entrepreneurs across the globe who could focus more on ontological and communicative perspectives of AI combined with Big Data to deal with diverse range of challenges and problems that modern businesses have been facing particularly in recent times.

COVID-19, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), spread rapidly and affected most of the world since its outbreak in Wuhan, China, which presents a major challenge to the emergency response mechanism for sudden public health events and epidemic prevention and control in all countries. In the face of the severe situation of epidemic prevention and control and the arduous task of social management, the tremendous power of science and technology in prevention and control has emerged. The new generation of information technology, represented by big data and artificial intelligence (AI) technology, has been widely used in the prevention, diagnosis, treatment and management of COVID-19 as an important basic support. Although the technology has developed, there are still challenges with respect to epidemic surveillance, accurate prevention and control, effective diagnosis and treatment, and timely judgement. The prevention and control of sudden infectious diseases usually depend on the control of infection sources, interruption of transmission channels and vaccine development. Big data and AI are effective technologies to identify the source of infection and have an irreplaceable role in distinguishing close contacts and suspicious populations. Advanced computational analysis is beneficial to accelerate the speed of vaccine research and development and to improve the quality of vaccines. AI provides support in automatically processing relevant data from medical images and clinical features, tests and examination findings; predicting disease progression and prognosis; and even recommending treatment plans and strategies. This paper reviews the application of big data and AI in the COVID-19 prevention, diagnosis, treatment and management decisions in China to explain how to apply big data and AI technology to address the common problems in the COVID-19 pandemic. Although the findings regarding the application of big data and AI technologies in sudden public health events lack validation of repeatability and universality, current studies in China have shown that the application of big data and AI is feasible in response to the COVID-19 pandemic. These studies concluded that the application of big data and AI technology can contribute to prevention, diagnosis, treatment and management decision making regarding sudden public health events in the future.

"Big data" has become a major force of innovation across enterprises of all sizes. New platforms with increasingly more features for managing big datasets are being announced almost on a weekly basis. Yet, there is currently a lack of any means of comparability among such platforms. While the performance of traditional database systems is well understood and measured by long-established institutions such as the Transaction Processing Performance Council (TCP), there is neither a clear definition of the performance of big data systems nor a generally agreed upon metric for comparing these systems. In this article, we describe a community-based effort for defining a big data benchmark. Over the past year, a Big Data Benchmarking Community has become established in order to fill this void. The effort focuses on defining an end-to-end application-layer benchmark for measuring the performance of big data applications, with the ability to easily adapt the benchmark specification to evolving challenges in the big data space. This article describes the efforts that have been undertaken thus far toward the definition of a BigData Top100 List. While highlighting the major technical as well as organizational challenges, through this article, we also solicit community input into this process.

Querying big data is the cornerstone of the application of big data. From a database perspective, query is the function defined by domain, range and a specified semantics. For querying big data, the domain is the big data and becomes quite complicated with characteristics of large volume, heterogeneous types, strong timeliness, weak authenticity, etc. To describe and analyze querying big data theoretically, we propose the definitions of big data and big data system, which includes querying big data. Aforementioned characteristics of big data lead to a break-through in normal form qualification and Closed World Assumption (CWA) related to traditional database. Therefore, this paper also points out several challenges according to those characteristics of domain and analyzes them in detail using the first-order language. Since the complexity of domain makes traditionally tractable queries infeasible, this paper analyzes and summarizes the classification of queries on relational big data according to their structure and computational complexity.

Big data management and application specialty is an emerging major in our school of information and business management. Based on the Internet plus and big data era, we mainly study the application of big data analysis theory and methods in economic management and the management and management of big data. This paper aims to explore how to improve the innovation and practical ability of students in this major through practical teaching. This paper mainly elaborates from four aspects, namely, the basic principles of practical teaching, the path of students' ability improvement, the construction of the completed practical teaching system and the application of practical teaching in this major.

The chapter showcases the awareness of big data usage among librarians in Zimbabwe. The concept of big data is new, and librarians are building capacity to move with the current trends in librarianship. This chapter assists in pointing areas where big data can be applied in libraries. It also documents the challenges that are faced when using big data applications and proffer solutions that can be applied to deal with those challenges. It answers the question of whether it is practical to utilise big data in any type of library. A qualitative study was done where an online questionnaire was administered to twenty librarians in research institutions in Zimbabwe. The findings revealed that librarians are aware of the big data concept but are not utilising the tools and techniques in data mining and analysis. The authors recommend that capacity building should be done to equip librarians with the requisite skills.

Data is accumulating from almost all aspects of our everyday lives that it becomes huge and multi- structured and has hidden useful information. The challenges with Big Data include capture, curation, storage, search, sharing, transfer, analysis, and visualization. Big Data provides materials for mining hidden patterns to support innovation mostly by data mining. The interaction research with Big Data support methods for innovation is rare at present. Knowledge discovered by data mining is novel and quantitative. However, it still lacks a uniform knowledge management model to support the innovation process effectively. The fact shows that since there emergence, big data techniques are changing very fast. In this paper, developing and maintaining the information security system to illustrate how those big data systems evolve and also describes some key design factors and challenges for future big data systems. Proposed design generic systems that can provide near real-time analytic services for many netease applications, such as spam detection, game log analysis and social community mining. No solutions can address all big data problems, especially when data size keeps increasing, more complex user requirements need to handle, the emergence of new hardware violates the old design and the old system becomes too complicated for maintenance. We face a series of technical challenges that have not been well addressed by both academic community and industry.