Big Data for flood management: Realising the benefits for developing countries

The thesis concerns the use of big data in Official Statistics; the aim is to bring some new experimental studies into the big data literature. In particular, the purpose is to evaluate if and how big data could be used in Official Statistics. Only a few experiments exists on the use of big data for statistical purposes; it is a challenge task and a lot of experimentation is needed to find out evidence and solutions to use big data for statistical purposes. The analysis performed in the thesis goes into two different directions: 1. combining a traditional data source with a big data source to verify the potential of the latter to replicate official results; 2. analyzing a big data source per se and then trying to combine with an Official Statistics source to identify common patterns. The thesis initially proposes a literature review of definitions of big data and experiments, in particular concerning the use of the new sources combined with traditional data sources. Then, three original studies have been performed: the first two concern mobility in Lombardy region using mobile phone data. They both refer to the same issue (mobility patterns), but they differ in the traditional data source used: Origin/Destination matrix in the first case, an integrated version of the O/D matrix in the second. The objective of these two studies is trying to put in a unique interpretative framework one traditional statistical source and one typical kind of big data in order to evaluate some informative potentialities of this approach. In particular, we wanted to check if the two sources show common patterns, to evaluate future uses of the big data source in Official Statistics. The third study shows the pilot that was carried out during the traineeship I had the opportunity to attend at Eurostat, in collaboration with the Task Force Big Data. It concerns the use of Wikipedia, free online encyclopedia, for Tourism Statistics. The aim is to evaluate the use of Wikipedia page views as a source of information for the identification of factors that drive tourism to an area and whether it is possible to predict tourism flows using these data. A final chapter proposes conclusions and future remarks on the use of big data in Official Statistics. Two of the studies (the first on mobility patterns and the one on Wikipedia) have been or are being published, in a shorter and revised version. The three experiments show some potential in the use of big data in Official Statistics. The study needs more in-depth analysis, many more experiments and considerations will be necessary before we can achieve some definitive and convincing approaches.

Taking a byte out of big data

Assessing the methodological quality of big data: an introduction to the Total Error framework The availability and use of big data sources is increasing exponentially. The variety of new and emerging data sources offers opportunities to complement, replace, improve or add to conventional data sources. Survey data are one kind of conventional data sources. In survey research, a framework to assess the accuracy of survey data already existed for quite some time. This framework is known as the Total Survey Error (TSE) framework. The philosophy behind this framework has only recently been universalized to (big) data in general in the form of the Total Error (TE) framework. This generic framework, which allows for assessing the accuracy of (big) data, is outlined in this article. Additionally, the TE framework is applied to big data sources that could be relevant for policing: police-registered crime data, Twitter data and mobile phone data.

Attention on “big data” spans nursing and the health sciences, and extends as well to engineering/computer sciences through to the liberal arts in professional literature. A current Google search (3 Nov 2016) of “big data” yields 288 million entries. A focused search of “big data and nursing” yields more than 3.9 million entries. Thus we ask, “Why big data? Why nursing?” The focus of this chapter is to provide an overview of why big data has emerged now and to make the case for how big data has the capacity to change health, healthcare systems, and nursing. This chapter lays a foundation for the chapters and case studies to follow that explore what data, knowledge, and transformation processes are needed to put information and knowledge into the hands of nursing wherever nurses are working. In this chapter we examine the big data sources within and beyond nursing and healthcare that can be collected and analyzed to improve nursing and patient, family and community health. This chapter entices the reader to examine “Why big data now?” and “Why big data in the future?” This chapter is meant to stir curiosity for “Why should I be knowledgeable?” Whether the reader’s role is in clinical practice, education, research, industry, or policy, the applied uses of big data analytics are empowering change at an exponential speed across all domains. Big data has the capacity to illuminate nursing’s discovery of new knowledge and best practices that are safe, effective and lead to improved outcomes including well-being of providers; it also can expand nursing’s vision and future possibilities through increasing awareness of what nursing doesn’t know. The importance of nursing’s lens on the new discoveries obtained through big data and data science is critical to the transformation of health and healthcare systems. This transformation completes the challenge of placing the person at the center of all care initiatives and actions.

The use of "Big Data" to explain fluctuations in the broader economy or guide the business decisions of a firm is now so commonplace that in some instances it has even begun to rival more traditional government statistics and business analytics. Big data sources can very often provide advantages when compared to these more traditional data sources, but with these advantages also comes the potential for pitfalls. We lay out a framework called SMALL that we have developed in order to help interested parties as they navigate the big data minefield. Based on a set of five questions, the SMALL framework should help users of big data spot concerns in their own work and that of others who rely on such data to draw conclusions with actionable public policy or business implications. To demonstrate, we provide several case studies that show a healthy dose of skepticism can be warranted when working with and interpreting these new big data sources.

Big data is a term that is used to describe data that is high volume, high velocity, and/or high variety; requires new technologies and techniques to capture, store, and analyze it; and is used to enhance decision making, provide insight and discovery, and support and optimize processes. With regard to the definition of big data, IBM Company uses volume, velocity, variety, value and veracity as 5Vs to summarize the concept of big data. There are different types of big data, for example, structured, semi-structured and un-structured data. The contents of big data can be text data, audio data, video data and still image and it indicates that the big data may have diverse data types as well as data qualities. Big data has variety of sources such as healthcare center, commercial system, industries, social media, telecommunication, transportation, sensor machines and others. In this paper, I reviewed three the most security challenging perspectives and I studied lack of concentrations in these areas by most research works. To confirm security in the big data platforms, it is critical to ascertain the data rendering points and their security techniques to safeguard the data in this pacing digital world. Then I envisage directions for the future research. In this paper, I have reviewed the big data sources and its security issues in the three directions such as data at rest, data at communication and data in process/use. Keywords: Big Data, Big Data Security, Big Data source, Attribute based encryption, storage path, Transport layer security, secure shell, Cloud service Provider DOI: 10.7176/CEIS/12-3-01 Publication date: November 30 th 2021

As we are living in the era of big data, huge volumes of a wide variety of complex data-which can be of different levels of veracity-are generated or collected at a high velocity from rich sources of data in various real-life applications. A rich source of these big data sources is the Internet of Things (IoT), which include a collection of sensors, smartphones and other mobile devices, wearable devices, as well as other "things" that are capable to operate within the existing Internet infrastructure. Embedded in these big data are valuable knowledge and useful information. Hence, the research problem of data mining from big IoT data have drawn attention of many researchers as it aims to discover implicit, previously unknown and potentially useful information and knowledge from the data. For instance, frequent pattern mining finds sets of frequently co-occurring items in the IoT domains. Associative classification discovers rules revealing relationships among items within the frequent patterns and their associations with the corresponding class labels. Induction based classification uses decision tree or random forest to learn from old big IoT for classifying or making predictions on new data. Over the past quarter of a century, many serial, distributed, parallel, and MapReduce-based (Hadook-based and Spark-based) big data mining algorithms have been proposed. These algorithms are run in local computers, distributed and parallel environments, clusters, grids, clouds and/or data centers. In this paper, we review some of these algorithms, discuss issues and research prospective in mining classification patterns from these big IoT data in fog. Our case study on a real-life application shows the feasibility on classifying real-life big IoT data over fog for urban analytics.

The smart grid (SG) allows integration of renewable energy sources, distributed generation (DG) and storage systems. This chapter builds on the concepts of data management and analytics in SG to build the foundation needed for data analytics to transform Big Data for high‐value action. Big data sources in SG generally fall into two main categories; electric utility data sources and supplementary data sources. The Big Data system will store, process, and mine information in an efficient manner to enhance different SG services. The chapter presents a list of the most common big data tools utilized by successful analytics developers to store, manage, and analyze big data. Big Data needs additional convincing methods to handle the massive amount of information in a limited time span. Any data exchange in SG must be effectively protected through specific “Privacy Concerns” which have potential privacy impacts of SG and smart meter systems.

Artificial intelligence and technological advancements are exceptionally influenced the entire society and mankind. Unprecendented and extensive use of social media, mobile phones and the internet has resulted in accumulation of huge amount of data. Most of this big data are available in unstructured form and it is beyond the capability of traditional systems to manage, maintain, supervise, keeping and analyse the data within a limited time span. Effective analysis and interpretation of health care data provides new insights in the condition of patients and suggest the most appropriate treatment opportunities. Discovery and invention of vital information in medical data helps the health care professionals to arrive at appropriate clinical decisions and improvement of quality of life in a variety of patients. In this article, we have discussed various issues and addressed them with the updated information on big data sources, big data management, big data processing and big data analysis through various tools and techniques. We have also analysed and interpreted the recent applications and advancements in artificial intelligence and big data in the health care technology and m-Health domain.

Obtaining indicators on the innovation activities of firms has been a challenge in economic research for a long time. The most frequently used indicators - R&D expenditures and patents - provide an incomplete picture as they represent inputs in the innovation process. Output measurement of innovation has strongly relied on survey data such as the Community Innovation Survey (CIS). However, this type of data suffers from several shortcomings typical of surveys, including incomplete coverage of the business sector, subjectivity concerns, low timeliness, and limited comparability across industries and firms. An alternative that has attracted growing interest is to use big data sources to collect innovation data at the firm level. This paper discusses recent attempts to use digital big data sources including websites and social media to generate firm-level innovation indicators. It summarises the main challenges of using big data and proposes practical guidelines for their use, including a research agenda that should be useful to practitioners as well as users of statistics derived from big data.

Chapter 16 - Big Data Analytics and Social Media in Disaster Management

Big data have been in use since the 1990s, which usually include some complex data sets whose sizes are beyond the ability of commonly used software to handle within a reasonable period of time. In recent years, big data analytics by providing personalized medicine and regulation analysis, providing clinical risk intervention and forecast analysis, reducing waste and nursing patients with external and internal variability, standardization of medical terminology and patient registration, and fragmentation of the solution, help to improve health care. This paper provides an overview of the contents of big data healthcare. We summarize some kinds of medical big data, including the electronic health records, the medical image data, the healthcare system big data, the health Internet of Things and healthcare informatics, the remote medical monitoring big data, the biomedical big data, and other sources of big data. Furthermore, we discuss some methods for handling different kinds of medical big data. Additionally, we analyze the privacy of medical big data and summarize some methods and technologies to protect privacy. Aiming at some special cases, we list some other analyses and methods for them. Most importantly, we discuss the potential challenges and future research directions related to big data healthcare.

Big data deals with data of very large data size, heterogeneous data types and from different sources. The data is very complex in nature and having growing data. Dealing with big data is one of the emerging areas of research which is expanding at a rapid rate in all domains of engineering and medical sciences. A major challenge imposes on the analysis of big data is originated from big data generation source, which generate data with very fast speed with varying data distribution due to which the classical methods are unable to process big data. This paper discusses the characteristics, challenges, and issues with big data mining. It also illustrates the examples taken from various fields like medical, finance, social networking sites, stock exchange, etc. to realize the application and importance of big data mining. This paper explains about the use of parallel computing in data mining security issues and how to deal with them. Furthermore, this paper also discusses challenges associated with big streaming data with concept drifts.

Network analytics in the big graph database era: A research framework and agenda for business and innovation

Nowadays, big data enables to discover many aspects in agriculture sector such as finding unknown crop patterns or predicting the price of products. However, these massive data are often complex and heterogeneous which includes both structured (e.g., farm information) and unstructured data (e.g., image data, sensor data). It is required new techniques and tools to extract and represent valuable information in the form of human understanding to improve decision making for enhancing farm management. In this paper, we propose a big data knowledge acquisition platform which consists of efficient knowledge acquisition techniques integrated with an intuitive visualization tool supporting decision making applications. Firstly, we deploy open source big data frameworks (e.g., Flume, Hive, HBase) to support developing of multiple methods for collecting and storing data. Secondly, we implement distributed machine learning techniques on Hadoop and Spark to acquire knowledge from big data sources. Finally, we provide a visualization tool on web interface which can display extracted knowledge in multiple views (e.g., charts, tables) to support decision making applications. Experiments with real datasets show that the proposed platform is efficient and effective to answer important questions in smart farming.