A Scalable Platform to Collect, Store, Visualize, and Analyze Big Data in Real Time

Taking a byte out of big data

A project titled “Establishing the advanced disaster reduction management system by fusion of real-time disaster simulation and big data assimilation,” was launched as Core Research for Evolutional Science and Technology (CREST) by the Japan Science and Technology Agency (JST). Intended to save as many lives as possible in future national crises involving earthquake and tsunami disasters, the project works on a disaster mitigation system of the big data era, based on cooperation of large-scale, high-resolution, real-time numerical simulations and assimilation of real-time observation data. The world’s most advanced specialists in disaster simulation, disaster management, mathematical science, and information science work together to create the world’s first analysis platform for real-time simulation and big data that effectively processes, analyzes, and assimilates data obtained through various observations. Based on quantitative data, the platform designs proactive measures and supports disaster operations immediately after disaster occurrence. The project was launched in 2014 and is working on the following issues at present.Sophistication and fusion of simulations and damage prediction models using observational big data: Development of a real-time simulation core system that predicts the time evolution of disaster effect by assimilating of location information, fire information, and building collapse information which are obtained from mobile terminals, satellite images, aerial images, and other new observation data in addition to sensing data obtained by the undersea high-density seismic observation network.Latent structure analysis and major disaster scenario creation based on a huge amount of simulation results: Development of an analysis and extraction method for the latent structure of a huge amount of disaster scenarios generated by simulation, and creation of severe scenarios with minimum “unexpectedness” by controlling disaster scenario explosion (an explosive increase in the number of predicted scenarios).Establishment of an earthquake and tsunami disaster mitigation big data analysis platform: Development of an earthquake and tsunami disaster mitigation big data analysis platform that realizes analyses of a huge number of disaster scenarios and increases in speed of data assimilation, and clarifies the requirements for operation of the platform as a disaster mitigation system.The project was launched in 2014 as a 5-year project. It consists of element technology development and system fusion, feasibility study as a next-generation disaster mitigation system (validation with/without introduction of the developed real-time simulation and big data analysis platform) in the affected areas of the Great East Japan Earthquake, and test operations in affected areas of the Tokyo metropolitan earthquake and the Nankai Trough earthquake.

Recent technologies such as Big Data could be used in order to help the improvement of other fields except for the telecommunication field. Due to the Health field, the Big Data technology could contribute with the aim to help for the purpose of analysis and management of the huge amounts of health data. The main objective of this research proposal is an analytical study of the technologies IoT, Cloud Computing (CC) and large-scale data (Big Data) to resolve various issues facing the health sector in relation to these technologies. The purpose of the research proposal is the collection of medical (e-health) big data in real time. The collection will be performed by sensor devices and actuators, which will wear patients who suffer from various ailments. Then is following, the transfer of these data through a network to a cloud server. Additionally, these data will be processed in the cloud which makes its analysis so as to become meaningless. By the analysis of these data is done the data mining. Finally, to address the various problems in the health sector, the transfer of the analyzed health data will be held by the devices of the relevant persons. Also, in our study, we will deal with the security of medical data which constitute personal data and must be protected.

Heterogeneous real-time spatial data management is a very active research domain nowadays. We are talking about a real-time spatial big data that process a large amount of heterogeneous data accessed simultaneously by two types of transactions update transactions and user transactions. In these applications, it is desirable to execute transactions within their deadlines using a real-time spatial data. But the real-time spatial big data can be overloaded and many transactions may miss their deadlines, or real-time spatial data can be violated. To address these problems, we proposed, as a first contribution, a new architecture called feedback control scheduling architecture for real-time spatial big data (FCSA-RTSBD) (Hamdi et al., 2015). Then, we propose, as a second contribution, two-shadow speculative concurrency control (SCC-2S) with priority and imprecise computation (SCC-2S-P-IC). Finally, a simulation study is shown to prove that our contributions can achieve a significant performance improvement using the TPC-DS (TPC, 2014) benchmark.

Big data refers to data so large, varied and generated at such an alarming rate that is too challenging for the conventional methods, tools, and technologies to handle it. Generating value out of it through analytics has started gaining paramount importance. Advanced analytics in the form of predictive and prescriptive analytics can scour through big data in real time or near real time to create valuable insights, which facilitate an organization in strategic decision making. The purpose of this paper is to review the emerging areas of big data and analytics, and is organized in two phases. The first phase covers taxonomy for classifying big data analytics (BDA), the big data value chain, and comparison of various platforms for BDA. The second phase discusses scope of research in BDA and some related work followed by a research proposal for developing a contextual model for BDA using advanced analytics.

Negative sentiments toward anterior cruciate ligament injury prevention outweigh positive awareness discussions on social media

Purpose With the remarkable capability to reach the public instantly, social media has become integral in sharing scholarly articles to measure public response. Since spamming by bots on social media can steer the conversation and present a false public interest in given research, affecting policies impacting the public’s lives in the real world, this topic warrants critical study and attention. Design/methodology/approach We used the Altmetric dataset in combination with data collected through the Twitter Application Programming Interface (API) and the Botometer API. We combined the data into an extensive dataset with academic articles, several features from the article and a label indicating whether the article had excessive bot activity on Twitter or not. We analyzed the data to see the possibility of bot activity based on different characteristics of the article. We also trained machine-learning models using this dataset to identify possible bot activity in any given article. Findings Our machine-learning models were capable of identifying possible bot activity in any academic article with an accuracy of 0.70. We also found that articles related to “Health and Human Science” are more prone to bot activity compared to other research areas. Without arguing the maliciousness of the bot activity, our work presents a tool to identify the presence of bot activity in the dissemination of an academic article and creates a baseline for future research in this direction. Research limitations/implications We considered the features available from the Altmetric dataset. It can be exciting research to extract additional features about the authors of the article, the location of the publication, international collaboration and other demographic features of the authors to see the relation of these features with bot activity. Practical implications Since public interest in scientific findings can shape the decisions of policymakers, it is essential to identify the possibility of bot activity in the dissemination of any given scholarly article. Without arguing whether the social bots are good or bad and without arguing about the validity of a scholarly article, our work proposes a tool to interpret the public interest in an article by identifying the possibility of bot activity toward an article. This work publishes the models and data generated through the study and provides a benchmark and guideline for future works in this direction. Originality/value While the majority of the existing research focuses on identifying and preventing bot activity on social media, our work is novel in predicting the possibility of bot activity in the dissemination of an academic article using Altmetric metadata for the article. Little work has been performed in this specific area, and the models developed from our research give policymakers and the public a tool to interpret and understand the public interest in a scientific publication with appropriate caution.

In today’s big data era, cleaning big data streams has become a challenging task because of the different formats of big data and the massive amount of big data which is being generated. Many studies have proposed different techniques to overcome these challenges, such as cleaning big data in real time. This systematic literature review presents recently developed techniques that have been used for the cleaning process and for each data cleaning issue. Following the PRISMA framework, four databases are searched, namely IEEE Xplore, ACM Library, Scopus, and Science Direct, to select relevant studies. After selecting the relevant studies, we identify the techniques that have been utilized to clean big data streams and the evaluation methods that have been used to examine their efficiency. Also, we define the cleaning issues that may appear during the cleaning process, namely missing values, duplicated data, outliers, and irrelevant data. Based on our study, the future directions of cleaning big data streams are identified.

A subset of artificial intelligence (AI) is computational intelligence (CI). A human-like intelligence may be achieved in human-made robots by applying CI to develop intelligent systems. Compared to conventional methods, CI is more resilient, simpler, more efficient in producing tractable answers. When doing statistical computation (data analytics) in the fields of healthcare, social media research, industry, business, and pattern identification, recent technical advancements in CI have shown highly positive results. Big data is sometimes referred to as data that is “high volume, high diversity, and high velocity.” Big data is the enormous collection of information that businesses have gathered from a wide range of sources, including mobile applications, radio-wave reading and remote sensing equipment, wireless sensors, smartphones and other multimedia devices, remote sensing and location tracking equipment, and other similar sources. Data analytics is the process of applying analytical methods and code languages to large amounts of data in order to draw meaningful and appropriate conclusions from it. The Internet of things (IoT) is becoming widely used in almost all facets of contemporary life as a result of innovation in next-generation wireless technology, enabling people to work and live more intelligently. These fields cover a wide spectrum of commercial, military, and civil applications, including smart cities, smart homes, healthcare, the environment, and transportation. This chapter focuses on the various applications of the advancements in Computational Intelligence, big data technologies and Internet of things in various fields such as healthcare, agriculture, industries, business, energy sector, and retail business.

<p><strong>Background</strong></p> <p>Despite having the tools at our disposal to enable an adequate food supply for all people, inequities in food acquisition, distribution, and most importantly, food sovereignty, worsen food insecurity. The detrimental impact of climate change on food systems and mental health is further exacerbated by a lack of food sovereignty. We urgently require innovative solutions to enable food sovereignty, minimize food insecurity, and address climate change–related mental distress (ie, solastalgia). Indigenous communities have a wealth of Traditional Knowledge for climate change adaptation and preparedness to strengthen food systems. Traditional Knowledge combined with Western methods can revolutionize ethical data collection, engagement, and knowledge mobilization.</p> <p><strong>Objective</strong></p> <p>The Food Equity and Environmental Data Sovereignty (FEEDS) Project takes a participatory action, citizen science approach for early detection and warning of climate change impacts on food sovereignty, food security, and solastalgia. The aim of this project is to develop and implement a sustainable digital platform that enables real-time decision-making to mitigate climate change–related impacts on food systems and mental well-being.</p> <p><strong>Methods</strong></p> <p>Citizen science enables citizens to actively contribute to all aspects of the research process. The FEEDS Project is being implemented in five phases: participatory project planning, digital climate change platform customization, community-led evaluation, digital platform and project refinement, and integrated knowledge translation. The project is governed by a Citizen Scientist Advisory Council comprising Elders, Traditional Knowledge Keepers, key community decision makers, youth, and FEEDS Project researchers. The Council governs all phases of the project, including coconceptualizing a climate change platform, which consists of a smartphone app and a digital decision-making dashboard. Apart from capturing environmental and health-related big data (eg, weather, permafrost degradation, fire hazards, and human movement), the custom-built app uses artificial intelligence to engage and enable citizens to report on environmental hazards, changes in biodiversity or wildlife, and related food and mental health issues in their communities. The app provides citizens with valuable information to mitigate health-related risks and relays big data in real time to a digital dashboard.</p> <p><strong>Results</strong></p> <p>This project is currently in phase 1, with the subarctic Métis jurisdiction of Île-à-la-Crosse, Saskatchewan, Canada.</p> <p><strong>Conclusions</strong></p> <p>The FEEDS Project facilitates Indigenous Peoples’ self-determination, governance, and data sovereignty. All citizen data are anonymous and encrypted, and communities have ownership, access, control, and possession of their data. The digital dashboard system provides decision makers with real-time data, thereby increasing the capacity to self-govern. The participatory action research approach, combined with digital citizen science, advances the cocreation of knowledge and multidisciplinary collaboration in the digital age. Given the urgency of climate change, leveraging technology provides communities with tools to respond to existing and emerging crises in a timely manner, as well as scientific evidence regarding the urgency of current health and environmental issues.</p>

During the COVID-19 pandemic, social media became increasingly relied upon for health information in Canada. By analyzing georeferenced tweets using natural language processing, we aimed to understand regional discussions and concerns about school closures, masking, vaccines, and lockdowns during the pandemic's first two years. Using Twitter's application programming interface, we collected English-language tweets with keywords related to COVID-19 posted between January 1, 2020 and February 22, 2022 from Canadian users. Out of all retained tweets, 2,851,951 (47.9%) were about vaccines, 1,344,008 (22.6%) about lockdowns, 1,011,909 (17%) about schooling, and 752,014 (12.6%) about masking. Tweets on schooling received the most engagement, with the highest rates of likes (17.3%), retweets (18.7%), replies (10%), and quotes (6.8%). The most common emotions expressed were trust, fear, and anticipation, with lockdown tweets showing greater fear and sadness. Overall, sentiment was negative, particularly regarding lockdowns in the Northwest Territories and Alberta. During the COVID-19 pandemic, Twitter became an essential tool for analyzing public sentiment regarding government actions. Users showed the most interest in vaccines, followed by lockdowns, schooling, and masking, with the highest engagement on schooling tweets. Our analysis of sentiment, emotion, and content revealed valuable insights into public beliefs about COVID-19 in Canada, highlighting regional differences and shifts in sentiment, particularly negative reactions to school closures as government recommendations evolved. Our study adds to the growing evidence supporting the use of natural language processing for real-time analysis of social media content to early identify public health concerns.

Colorectal cancer (CRC) is the second leading cause of cancer-related mortality worldwide. Social media platforms such as Twitter are extensively used to communicate about cancer care, yet little is known about the role of these online platforms in promoting early detection or sharing the lived experiences of patients with CRC. This study tracked Twitter discussions about CRC and characterized participating users to better understand public communication and perceptions of CRC during the COVID-19 pandemic. Tweets containing references to CRC were collected from January 2020 to April 2021 using Twitter's Application Programming Interface. Account metadata was used to predict user demographic information and classify users as either organizations, individuals, clinicians, or influencers. We compared the number of impressions across users and analyzed the content of tweets using natural language processing models to identify prominent topics of discussion. There were 72,229 unique CRC-related tweets by 31,170 users. Most users were male (66%) and older than 40 years (57%). Individuals accounted for most users (44%); organizations (35%); clinicians (19%); and influencers (2%). Influencers made the most median impressions (35,853). Organizations made the most overall impressions (1,067,189,613). Tweets contained the following topics: bereavement (20%), appeals for early detection (20%), research (17%), National Colorectal Cancer Awareness Month (15%), screening access (14%), and risk factors (14%). Discussions about CRC largely focused on bereavement and early detection. Online coverage of National Colorectal Cancer Awareness Month and personal experiences with CRC effectively stimulated goal-oriented tweets about early detection. Our findings suggest that although Twitter is commonly used for communicating about CRC, partnering with influencers may be an effective strategy for improving communication of future public health recommendations related to CRC.

Nowadays a lot of massive data is stored and typically, the data itself contains a lot of non-trivial but useful information. Data mining techniques can be used to discover this information which can help the companies for decision-making. However, in real life applications, data is massive and is stored over distributed sites. One of my major research topics is to protect privacy over this kind of data. Previously, the important characteristics, issues and challenges related to management of the large amount of data has been explored. Various open source data analytics frameworks that deal with large amount of Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. Digital universe is flooded with huge amount of data generated by number of users worldwide. These data are of diverse in nature, come from various sources and in many forms. To keep with the desire to store and analyze ever larger volumes of complex data, relational databases vendors have delivered specialized analytical platforms that come in many shapes and sizes from software only to analytical services that run in third party hosted environments. In addition new technologies have emerged to address exploding volumes of complex data, including web traffic, social media content and machine generated data including sensor data, global positioning system data.Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data has become a prominent research field, especially when it comes to decision making and data analysis. However, Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges. Since Big data is a recent upcoming technology in the market which can bring huge benefits to the business organizations, it becomes necessary that various challenges and issues associated in bringing and adapting to this technology are brought into light. Another challenge is that data collection may not have enough accuracy which will lead to a non-consistent analysis that can critically affect the decision based on this analysis. Moreover, it is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is; current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Since data is growing at a huge speed making it difficult to handle such large amount of data (Exabyte). The main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties.If data is to be used to make accurate decisions in time it becomes necessary that it should be available in accurate, complete and timely manner. This makes the data management and governance process bit complex adding the necessity to make Data open and make it available to government agencies in standardized manner with standardized APIs, metadata and formats thus leading to better decision making, business intelligence and productivity improvements.This paper presents a discussion and evaluation for the most prominent techniques used in the processes of data collection and analysis in order to identify the privacy defects in them that affects the accuracy of big data. Depending on the results of this analysis, recommendations were provided for improving data collection and analysis techniques that will help to avoid if not all then most of the problems facing the use of big data in decision making. Keywords: Big Data, Big Data Challenges, Big Data Accuracy, Big Data Collection, Big Data Analytics.

This chapter provides a brief overview of the growing role played by big data, data analytics, and artificial intelligence (AI) in the accounting profession. The term 'Big Data' along with other trending topics such as 'Data Analytics' and 'Artificial Intelligence (AI)' have become buzzwords in the accounting profession in recent years. The skills of accounting professionals have evolved as technology has made rapid advances from using pencil and paper to typewriters and calculators, and eventually spreadsheets and accounting software. Data analytics in accounting is a relatively new skill set that is growing significantly in all areas of the accounting profession. Accounting professionals who can discern patterns and trends in big data and translate them into compelling strategic narratives will find themselves at the center of the twenty-first-century business world. Therefore, accounting professionals can capitalize on numerous opportunities in this rapidly evolving, disruptive, but ultimately advantageous environment by embracing big data, data analytics, and artificial intelligence (AI) to stay ahead of the competition.

In a world of swift digital transformations, the global market is seeing notable structural shifts. This applies not only to novel technologies, but also to how brands engage with their consumers. Artificial intelligence (AI) and big data analytics are at the heart of this change. With these instruments, companies can analyze consumer behavior more profoundly and precisely, considering the cultural and behavioral traits of diverse audiences worldwide. This allows them to hone their marketing strategies and cultivate a strong emotional bond between the consumer and the brand. The article aims to identify and examine the opportunities that artificial intelligence and big data technologies offer to boost the efficacy of brand communication within the international market. The article reviews theoretical approaches to constructing brand communication in the global market, considering digital transformation, specifically concentrating on the concept and function of brand communication. The key models of brand communication are assessed. The potential of employing artificial intelligence to develop and manage brand communications is examined. The role of big data in making communication decisions, encompassing audience segmentation, analysis of consumer behavior, and content adjustment to cultural specificities is identified. Particular attention is given to the integration of AI and Big Data into a brand’s global communication plan. The pros and cons associated with incorporating AI and Big Data into brand communication strategies, including ethical and legal aspects, are outlined. The article suggests constructing brand communication centered on artificial intelligence and big data, and offers suggestions for brands on the effective application of digital technologies in communicating with consumers on a worldwide scale. Via artificial intelligence and data analytics, brands can ensure a consistent customer experience throughout all stages of interaction. This starts with social media and expands all the way to physical points of sale, which in turn helps to boost customer loyalty and satisfaction. Analytics of behavioral, demographic, and contextual data allows for precise segmentation, recognizing consumer needs, and predicting market trends. This produces the conditions for brands to more effectively tailor content to the cultural and regional attributes of their audience.