Big Data and Analytics—A Journey Through Basic Concepts to Research Issues

A project titled “Establishing the advanced disaster reduction management system by fusion of real-time disaster simulation and big data assimilation,” was launched as Core Research for Evolutional Science and Technology (CREST) by the Japan Science and Technology Agency (JST). Intended to save as many lives as possible in future national crises involving earthquake and tsunami disasters, the project works on a disaster mitigation system of the big data era, based on cooperation of large-scale, high-resolution, real-time numerical simulations and assimilation of real-time observation data. The world’s most advanced specialists in disaster simulation, disaster management, mathematical science, and information science work together to create the world’s first analysis platform for real-time simulation and big data that effectively processes, analyzes, and assimilates data obtained through various observations. Based on quantitative data, the platform designs proactive measures and supports disaster operations immediately after disaster occurrence. The project was launched in 2014 and is working on the following issues at present.Sophistication and fusion of simulations and damage prediction models using observational big data: Development of a real-time simulation core system that predicts the time evolution of disaster effect by assimilating of location information, fire information, and building collapse information which are obtained from mobile terminals, satellite images, aerial images, and other new observation data in addition to sensing data obtained by the undersea high-density seismic observation network.Latent structure analysis and major disaster scenario creation based on a huge amount of simulation results: Development of an analysis and extraction method for the latent structure of a huge amount of disaster scenarios generated by simulation, and creation of severe scenarios with minimum “unexpectedness” by controlling disaster scenario explosion (an explosive increase in the number of predicted scenarios).Establishment of an earthquake and tsunami disaster mitigation big data analysis platform: Development of an earthquake and tsunami disaster mitigation big data analysis platform that realizes analyses of a huge number of disaster scenarios and increases in speed of data assimilation, and clarifies the requirements for operation of the platform as a disaster mitigation system.The project was launched in 2014 as a 5-year project. It consists of element technology development and system fusion, feasibility study as a next-generation disaster mitigation system (validation with/without introduction of the developed real-time simulation and big data analysis platform) in the affected areas of the Great East Japan Earthquake, and test operations in affected areas of the Tokyo metropolitan earthquake and the Nankai Trough earthquake.

While the potential benefits of Big Data adoption are significant, and some initial successes have already been realized, there remain many research and technical challenges that must be addressed to fully realize this potential. The Big Data processing, storage and analytics, of course, are major challenges that are most easily recognized. However, there are additional challenges related for instance to Big Data collection, integration, and quality enforcement. This paper proposes a hybrid approach to Big Data quality evaluation across the Big Data value chain. It consists of assessing first the quality of Big Data itself, which involve processes such as cleansing, filtering and approximation. Then, assessing the quality of process handling this Big Data, which involve for example processing and analytics process. We conduct a set of experiments to evaluate Quality of Data prior and after its pre-processing, and the Quality of the pre-processing and processing on a large dataset. Quality metrics have been measured to access three Big Data quality dimensions: accuracy, completeness, and consistency. The results proved that combination of data-driven and process-driven quality evaluation lead to improved quality enforcement across the Big Data value chain. Hence, we recorded high prediction accuracy and low processing time after we evaluate 6 well-known classification algorithms as part of processing and analytics phase of Big Data value chain.

Recent technologies such as Big Data could be used in order to help the improvement of other fields except for the telecommunication field. Due to the Health field, the Big Data technology could contribute with the aim to help for the purpose of analysis and management of the huge amounts of health data. The main objective of this research proposal is an analytical study of the technologies IoT, Cloud Computing (CC) and large-scale data (Big Data) to resolve various issues facing the health sector in relation to these technologies. The purpose of the research proposal is the collection of medical (e-health) big data in real time. The collection will be performed by sensor devices and actuators, which will wear patients who suffer from various ailments. Then is following, the transfer of these data through a network to a cloud server. Additionally, these data will be processed in the cloud which makes its analysis so as to become meaningless. By the analysis of these data is done the data mining. Finally, to address the various problems in the health sector, the transfer of the analyzed health data will be held by the devices of the relevant persons. Also, in our study, we will deal with the security of medical data which constitute personal data and must be protected.

Despite the enormous potential of big data, it is a relatively new issue for many companies, particularly for those of smaller size that looks at this as a challenge, is unattainable and only possible for companies with high financial capacity. However, open source software presents itself as an excellent alternative for these companies, which will allow them to exploit the high volume of data they have at their disposal. In this sense, this study presents a proposal for an architecture based exclusively on open source software that includes the entire value chain of big data, from data collection to data analysis. This architecture was tested considering three emerging scenarios in which big data become very relevant and challenging, namely for mobile analytics, network analytics, and mobile analytics.

As a result of the digital transformation that has led by different production sectors, the organizations were flooded by vast, various, and complex data, called Big Data. To manage their data assets, the adoption of the Big Data value chain (BDVC) was suitable for a value realization as well as a data-intensive decision-making. The manipulation and exploitation of these data gave rise to the concept of Big Data monetization. The marriage between the BDVC and data monetization remains an own and recent approach. This combination allows the organization's process to become entirely data-driven and expand the scope of its exchanges. Besides, advanced deployment models such as cloud computing will enable this combination to become more resilient by providing large amounts of computational and networking capabilities. Furthermore, it allows sharing data between different BDVCs to build an expandable ecosystem. In this contribution, we review the literature regarding the association between BDVC, data monetization, and cloud computing. Then, we propose a Big Data monetization-driven value chain model that relies on cloud computing and analytical capabilities. It allows monetizing both data and insight as a service under a collaborative concept.

Taking a byte out of big data

In today’s big data era, cleaning big data streams has become a challenging task because of the different formats of big data and the massive amount of big data which is being generated. Many studies have proposed different techniques to overcome these challenges, such as cleaning big data in real time. This systematic literature review presents recently developed techniques that have been used for the cleaning process and for each data cleaning issue. Following the PRISMA framework, four databases are searched, namely IEEE Xplore, ACM Library, Scopus, and Science Direct, to select relevant studies. After selecting the relevant studies, we identify the techniques that have been utilized to clean big data streams and the evaluation methods that have been used to examine their efficiency. Also, we define the cleaning issues that may appear during the cleaning process, namely missing values, duplicated data, outliers, and irrelevant data. Based on our study, the future directions of cleaning big data streams are identified.

Twitter has withstood the test of time as a successful social networking platform. In many circles globally, the majority of users choose Twitter when choosing a social media outlet for reliable scientific information and news. However, the Twitter application programming interface (API) limitations do not allow for low-cost data science options for academia. It becomes very expensive for academic researchers to gain the full potential of data analytics available from Twitter using a free API account. In this article, we present our big data analytics platform developed at our DaTALab at Lakehead University, Canada, that allows users to focus on their Twitter search criteria and gain access to large amounts of Twitter data at the touch of a button. The platform supports the collection of social media data and applies many filters for cleaning and further use for machine learning (ML) and artificial intelligence (AI)-based systems. Our focus has been primarily on healthcare-related research, which shows the strength of the presented platform. However, the platform itself is malleable to any topic of interest. Data collected and processed are suitable for further AI/ML analysis. We present our platform using a specific healthcare search topic to emphasize the power of our system for future research endeavors in the healthcare field.

Big data analytics platforms on cloud are becoming mainstream technology enabling cost-effective rapid deployment of customer's Big Data applications delivering quicker insights from their data. It is, therefore, even more imperative that we have high performant platform infrastructure and application at a reasonable cost. This is only possible if we make a transition from traditional approach to execute and measure performance by adopting new AI techniques such as Machine Learning (ML) & predictive approach to performance benchmarking for every application domain.

Businesses, government sectors, academia and society have been flooded with data which nowadays have become valuable economic assets. Data and Big Data Analytics (BDA) are vital for the succession of any organization. Big Data Analytics (BDA) delineates the approaches utilized to examine, process, analyze and expose hidden underlying patterns, interesting relations, and intelligence from huge datasets for different purposes. Despite the growth and widespread use of data analytics, the study on this research area is still at the nascent stage. The scenario is seen particularly for the developing countries like Malaysia. Previous literature has recognized data analytic development and its effect but has not yet provided a comprehensive review. This paper intends to conceptualize BDA manifestation, its application, and its challenges epistemology. This study employs a comprehensive review of past literature to serve the purpose. The findings of this paper will enhance the understanding of BDA conception epistemology (definition and application). This concept paper also intends to shed some light on BDA research areas by identifying the challenges surrounding BDA deployment. From the theoretical point of view, this paper contributes to providing BDA taxonomy based on the application in various industries. From the practical perspective, this paper contributes to BDA research field by identifying the obstacles that need to be managed by the BDA adopter for optimum utilization and decision making. From the government perspective, it contributes to Malaysian national agenda to becoming a leading regional BDA solution hub for all sectors and developed nation by the year 2025.

The use of large and diverse data in real time (called Big Data) affects many business processes and models. The tourism industry, characterized by manifold sub-sectors and players, provides a variety of starting points for Big Data usage. Examples are the optimization of transport offers using transaction data or a comprehensive analysis of destination trends based on social media posts. Big Data is a trending topic, however, the general discourse centres around potential ideas but fewer practical solutions. Based on a systematic literature analysis of initially 148 peer-reviewed journal articles, this article evaluates the current state of Big Data research within tourism. For this purpose, research articles centering around tourism-related Big Data were investigated according to the actual state of implementation of an IT solution, whether they truly grasp or represent Big Data in technological terms, and which added value they create for the tourism industry and research community. One key finding is that traditional data analysis is often wrongfully subsumed under the Big Data label. Further, the scientific literature predominantly discusses ideas or theoretical considerations, fewer tangible Big Data implementations, and fails to address and/or meet all requirements to be classified as Big Data. Only a minority of the presented solutions processes data in real time, whereas many rely on only one data source or structured data. Furthermore, most articles revolve around post-trip data analyses and are set to a destination context. In contrast, other tourism sectors as well as data interpretation and usage in pre-trip and on-trip phases are less represented. Lastly, this literature analysis provides an overview of true Big Data solutions already in operation and enables researchers to validly classify their own research activities in order to plan initiatives more specifically.

Objective: Aim of the present study was to literature review on the NoSQL Database for Big Data processing including the structural issues and the real-time data mining techniques to extract the estimated valuable information.Methods: We searched the Springer Link and IEEE Xplore online databases for articles published in English language during the last seven years (between January 2011 and December 2017). We specifically searched for two keywords (“NoSQL” and “Big Data”) to find the articles. The inclusion criteria were articles on the use of performance comparison on valuable information processing in the field of Big Data through NoSQL databases.Results: In the 18 selected articles, this review identified 8 articles which provided various suitable recommendations on NoSQL databases for specific area focus on the value chain of Big Data, 5 articles described the performance comparison of different NoSQL databases, 2 articles presented the background of basics characteristics data model for NoSQL, 1 article denoted the storage in respect of cloud computing and 2 articles focused the transactions of NoSQL.Conclusion: In this literature, we presented the NoSQL databases for Big Data processing including its transactional and structural issues. Additionally, we highlight research directions and challenges in relation to Big Data processing. Therefore, we believe that the information contained in this review will incredible support and guide the progress of the Big Data processing.

Cloud computing and big data: Technologies and applications

<p><strong>Background</strong></p> <p>Despite having the tools at our disposal to enable an adequate food supply for all people, inequities in food acquisition, distribution, and most importantly, food sovereignty, worsen food insecurity. The detrimental impact of climate change on food systems and mental health is further exacerbated by a lack of food sovereignty. We urgently require innovative solutions to enable food sovereignty, minimize food insecurity, and address climate change–related mental distress (ie, solastalgia). Indigenous communities have a wealth of Traditional Knowledge for climate change adaptation and preparedness to strengthen food systems. Traditional Knowledge combined with Western methods can revolutionize ethical data collection, engagement, and knowledge mobilization.</p> <p><strong>Objective</strong></p> <p>The Food Equity and Environmental Data Sovereignty (FEEDS) Project takes a participatory action, citizen science approach for early detection and warning of climate change impacts on food sovereignty, food security, and solastalgia. The aim of this project is to develop and implement a sustainable digital platform that enables real-time decision-making to mitigate climate change–related impacts on food systems and mental well-being.</p> <p><strong>Methods</strong></p> <p>Citizen science enables citizens to actively contribute to all aspects of the research process. The FEEDS Project is being implemented in five phases: participatory project planning, digital climate change platform customization, community-led evaluation, digital platform and project refinement, and integrated knowledge translation. The project is governed by a Citizen Scientist Advisory Council comprising Elders, Traditional Knowledge Keepers, key community decision makers, youth, and FEEDS Project researchers. The Council governs all phases of the project, including coconceptualizing a climate change platform, which consists of a smartphone app and a digital decision-making dashboard. Apart from capturing environmental and health-related big data (eg, weather, permafrost degradation, fire hazards, and human movement), the custom-built app uses artificial intelligence to engage and enable citizens to report on environmental hazards, changes in biodiversity or wildlife, and related food and mental health issues in their communities. The app provides citizens with valuable information to mitigate health-related risks and relays big data in real time to a digital dashboard.</p> <p><strong>Results</strong></p> <p>This project is currently in phase 1, with the subarctic Métis jurisdiction of Île-à-la-Crosse, Saskatchewan, Canada.</p> <p><strong>Conclusions</strong></p> <p>The FEEDS Project facilitates Indigenous Peoples’ self-determination, governance, and data sovereignty. All citizen data are anonymous and encrypted, and communities have ownership, access, control, and possession of their data. The digital dashboard system provides decision makers with real-time data, thereby increasing the capacity to self-govern. The participatory action research approach, combined with digital citizen science, advances the cocreation of knowledge and multidisciplinary collaboration in the digital age. Given the urgency of climate change, leveraging technology provides communities with tools to respond to existing and emerging crises in a timely manner, as well as scientific evidence regarding the urgency of current health and environmental issues.</p>

The unprecedented growing and accessibility of traffic data has materialized various opportunities to improve the quality of transportation related services. In this paper, we extend Sipresk, a big data analytic platform, to ingest real-time traffic data and provide online analytics for various stakeholders. We leverage extended Sipresk to detect and classify events in Ontario highways. More specifically, we designed and implemented a set of algorithms that detect four traffic patterns each of which detects a specific type of event. The platform reports the event specifications no later than 20 seconds that provides the opportunity to react promptly to incidents in highways. Also, we configure Sipresk to process traffic sensor data from January 2015 to June 2016 for detection and classification of events for every single day. In less than 2 minutes, Sipresk is able to process 24 hours of data and generate a list of events occurred during that specific date.