Big Data Knowledge Acquisition Platform for Smart Farming

Big data is a term that is used to describe data that is high volume, high velocity, and/or high variety; requires new technologies and techniques to capture, store, and analyze it; and is used to enhance decision making, provide insight and discovery, and support and optimize processes. With regard to the definition of big data, IBM Company uses volume, velocity, variety, value and veracity as 5Vs to summarize the concept of big data. There are different types of big data, for example, structured, semi-structured and un-structured data. The contents of big data can be text data, audio data, video data and still image and it indicates that the big data may have diverse data types as well as data qualities. Big data has variety of sources such as healthcare center, commercial system, industries, social media, telecommunication, transportation, sensor machines and others. In this paper, I reviewed three the most security challenging perspectives and I studied lack of concentrations in these areas by most research works. To confirm security in the big data platforms, it is critical to ascertain the data rendering points and their security techniques to safeguard the data in this pacing digital world. Then I envisage directions for the future research. In this paper, I have reviewed the big data sources and its security issues in the three directions such as data at rest, data at communication and data in process/use. Keywords: Big Data, Big Data Security, Big Data source, Attribute based encryption, storage path, Transport layer security, secure shell, Cloud service Provider DOI: 10.7176/CEIS/12-3-01 Publication date: November 30 th 2021

Taking a byte out of big data

The volume of data in the world is growing very fast and generated from verity of sources like social media, sensors airline industry or scientific data in different formats. Biggest challenge is how to infer meaningful insights from such a varietyful and big data along with concern of data storage and management of fast growing data. The size of the databases used in today’s enterprises has been growing at exponential rates day by day. Hence, industries requirement to quickly process and analyze the big data volumes for business decision making and customer insights has also grown exponentially. Data pouring from various sources may be can be structured or unstructured in nature. Structured data refers to a relatively well-organized information, which can be further inserted into traditional RDBMS. As Traditional RDBMS are efficient and easy queries by simple, straightforward search algorithms or SQL queries. In contrast to structured data, unstructured data can be considered as information, which does not, comes in a pre-defined data format, well organized data storage model, or cannot be stored well into relational tables. It is assumed to be fastest growing type of data, e.g. image, sensors data, web chats, social networking messaging data, video, documents, log files, and email data. There are many techniques and software available, which can process and provide efficient storage of unstructured data and help organization to perform analytics on unstructured data. Unstructured data does not well-organized and not stored in predefined manner e.g. logs, web chats. The variety and on ordered nature of data makes storage methods and structure makes execution a time and resource-consuming affair. Advancement into technology has open floodgates to push huge volume of unstructured type of data. Multimedia data is one of the example of unstructured big data, which spans all over the Internet. This needs high execution capability to extract useful information. Rapid processing of multimedia data such as video is important for e.g. criminal investigations, surveillance monitoring, news analysis, sports analytics domain, emotion extraction, etc. Hence, analysis of multimedia data in minimum timeframe is one of the latest research areas. Therefore, we have researched techniques for analyzing unstructured data to extract meaningful information hidden in the big data. In addition, we will describe about various techniques and software used to Manage, process unstructured big data in efficient manner, and increases the performance of complexity analysis.

The thesis concerns the use of big data in Official Statistics; the aim is to bring some new experimental studies into the big data literature. In particular, the purpose is to evaluate if and how big data could be used in Official Statistics. Only a few experiments exists on the use of big data for statistical purposes; it is a challenge task and a lot of experimentation is needed to find out evidence and solutions to use big data for statistical purposes. The analysis performed in the thesis goes into two different directions: 1. combining a traditional data source with a big data source to verify the potential of the latter to replicate official results; 2. analyzing a big data source per se and then trying to combine with an Official Statistics source to identify common patterns. The thesis initially proposes a literature review of definitions of big data and experiments, in particular concerning the use of the new sources combined with traditional data sources. Then, three original studies have been performed: the first two concern mobility in Lombardy region using mobile phone data. They both refer to the same issue (mobility patterns), but they differ in the traditional data source used: Origin/Destination matrix in the first case, an integrated version of the O/D matrix in the second. The objective of these two studies is trying to put in a unique interpretative framework one traditional statistical source and one typical kind of big data in order to evaluate some informative potentialities of this approach. In particular, we wanted to check if the two sources show common patterns, to evaluate future uses of the big data source in Official Statistics. The third study shows the pilot that was carried out during the traineeship I had the opportunity to attend at Eurostat, in collaboration with the Task Force Big Data. It concerns the use of Wikipedia, free online encyclopedia, for Tourism Statistics. The aim is to evaluate the use of Wikipedia page views as a source of information for the identification of factors that drive tourism to an area and whether it is possible to predict tourism flows using these data. A final chapter proposes conclusions and future remarks on the use of big data in Official Statistics. Two of the studies (the first on mobility patterns and the one on Wikipedia) have been or are being published, in a shorter and revised version. The three experiments show some potential in the use of big data in Official Statistics. The study needs more in-depth analysis, many more experiments and considerations will be necessary before we can achieve some definitive and convincing approaches.

Assessing the methodological quality of big data: an introduction to the Total Error framework The availability and use of big data sources is increasing exponentially. The variety of new and emerging data sources offers opportunities to complement, replace, improve or add to conventional data sources. Survey data are one kind of conventional data sources. In survey research, a framework to assess the accuracy of survey data already existed for quite some time. This framework is known as the Total Survey Error (TSE) framework. The philosophy behind this framework has only recently been universalized to (big) data in general in the form of the Total Error (TE) framework. This generic framework, which allows for assessing the accuracy of (big) data, is outlined in this article. Additionally, the TE framework is applied to big data sources that could be relevant for policing: police-registered crime data, Twitter data and mobile phone data.

The use of "Big Data" to explain fluctuations in the broader economy or guide the business decisions of a firm is now so commonplace that in some instances it has even begun to rival more traditional government statistics and business analytics. Big data sources can very often provide advantages when compared to these more traditional data sources, but with these advantages also comes the potential for pitfalls. We lay out a framework called SMALL that we have developed in order to help interested parties as they navigate the big data minefield. Based on a set of five questions, the SMALL framework should help users of big data spot concerns in their own work and that of others who rely on such data to draw conclusions with actionable public policy or business implications. To demonstrate, we provide several case studies that show a healthy dose of skepticism can be warranted when working with and interpreting these new big data sources.

Attention on “big data” spans nursing and the health sciences, and extends as well to engineering/computer sciences through to the liberal arts in professional literature. A current Google search (3 Nov 2016) of “big data” yields 288 million entries. A focused search of “big data and nursing” yields more than 3.9 million entries. Thus we ask, “Why big data? Why nursing?” The focus of this chapter is to provide an overview of why big data has emerged now and to make the case for how big data has the capacity to change health, healthcare systems, and nursing. This chapter lays a foundation for the chapters and case studies to follow that explore what data, knowledge, and transformation processes are needed to put information and knowledge into the hands of nursing wherever nurses are working. In this chapter we examine the big data sources within and beyond nursing and healthcare that can be collected and analyzed to improve nursing and patient, family and community health. This chapter entices the reader to examine “Why big data now?” and “Why big data in the future?” This chapter is meant to stir curiosity for “Why should I be knowledgeable?” Whether the reader’s role is in clinical practice, education, research, industry, or policy, the applied uses of big data analytics are empowering change at an exponential speed across all domains. Big data has the capacity to illuminate nursing’s discovery of new knowledge and best practices that are safe, effective and lead to improved outcomes including well-being of providers; it also can expand nursing’s vision and future possibilities through increasing awareness of what nursing doesn’t know. The importance of nursing’s lens on the new discoveries obtained through big data and data science is critical to the transformation of health and healthcare systems. This transformation completes the challenge of placing the person at the center of all care initiatives and actions.

<p>Agenda 2030 goal 11 commits towards making disaster risk reduction an integral part of sustainable social and economic development. Flooding poses some of the most serious challenges in front of developing nations by hitting hardest to the most vulnerable. Focussing on the urban poor, frequently at highest risk are characterised by inadequate housing, lack of services and infrastructure with high population growth and spatial expansion in dense, lower quality urban structures. Use of big data from within these low-quality urban settlement areas can be a useful step forward in generating information to have a better understanding of their vulnerabilities. Big data for resilience is a recent field of research which offers tremendous potential for increasing disaster resilience especially in the context of social resilience. This research focusses to unleash the unrealised opportunities of big data through the differential social and economic frames that can contribute towards better-targeted information generation in disaster management. The scoping study aims to contribute to the understanding of the potential of big data in developing particularly in low-income countries to empower the vulnerable population against natural hazards such as floods. Recognising the potential of providing real-time and long-term information for emergency management in flood-affected large urban settlements this research concentrates on flood hazard and use of remotely sensed data (NASA, TRMM, LANDSAT) as the big data source for quick disaster response (and recovery) in targeted areas. The research question for the scoping study is: Can big data source provide real-time and long- term information to improve emergency disaster management in urban settlements against floods in developing countries?  Previous research has identified several potentials that big data has on faster response to the affected population but few attempts have been made to integrate the factors to develop an aggregated conceptual output . An international review of multi-discipline research, grey literature, grass-root projects, and emerging online social discourse will appraise the concepts and scope of big data to highlight the four objectives of the research and answer the specific questions around existing and future potentials of big data, operationalising and capacity building by agencies, risk associated and prospects of maximising impact. The research proposes a concept design for undertaking a thematic review of existing secondary data sources which will  be used to provide a holistic picture of how big data can support in resilience through technological change within the specific scope of social and environmental contexts of developing countries. The implications of the study lie in the system integration and understanding of the socio-economics, political, legal and ethical contexts essential for investment decision making for strategic impact and resilience-building in developing nations.</p>

Big Data analytics is one of the great challenges for Learning Machine (LM) algorithms because most real-life applications involve a massive information or big data knowledge base. By contrast, an Artificial Intelligent (AI) system with a data knowledge base should be able to compute the result in an accurate and fast manner. This study focused on the challenges and solutions of using with Big Data. Data processing is a mandatory step to transform unstructured Big Data into a meaningful and optimized data set in any LM module. However, an optimized data set must be deployed to support a distributed processing and real-time application. This work also reviewed the technologies currently used in Big Data analysis and LM computation and emphasized that the viability of using different solutions for certain applications could increase LM performance. The new development, especially in cloud computing and data transaction speed, offers significant advantages to the practical use of AI applications.

Advancement of the search process for digital heritage by utilizing artificial intelligence algorithms

Big data have been in use since the 1990s, which usually include some complex data sets whose sizes are beyond the ability of commonly used software to handle within a reasonable period of time. In recent years, big data analytics by providing personalized medicine and regulation analysis, providing clinical risk intervention and forecast analysis, reducing waste and nursing patients with external and internal variability, standardization of medical terminology and patient registration, and fragmentation of the solution, help to improve health care. This paper provides an overview of the contents of big data healthcare. We summarize some kinds of medical big data, including the electronic health records, the medical image data, the healthcare system big data, the health Internet of Things and healthcare informatics, the remote medical monitoring big data, the biomedical big data, and other sources of big data. Furthermore, we discuss some methods for handling different kinds of medical big data. Additionally, we analyze the privacy of medical big data and summarize some methods and technologies to protect privacy. Aiming at some special cases, we list some other analyses and methods for them. Most importantly, we discuss the potential challenges and future research directions related to big data healthcare.

The smart grid (SG) allows integration of renewable energy sources, distributed generation (DG) and storage systems. This chapter builds on the concepts of data management and analytics in SG to build the foundation needed for data analytics to transform Big Data for high‐value action. Big data sources in SG generally fall into two main categories; electric utility data sources and supplementary data sources. The Big Data system will store, process, and mine information in an efficient manner to enhance different SG services. The chapter presents a list of the most common big data tools utilized by successful analytics developers to store, manage, and analyze big data. Big Data needs additional convincing methods to handle the massive amount of information in a limited time span. Any data exchange in SG must be effectively protected through specific “Privacy Concerns” which have potential privacy impacts of SG and smart meter systems.

Artificial intelligence and technological advancements are exceptionally influenced the entire society and mankind. Unprecendented and extensive use of social media, mobile phones and the internet has resulted in accumulation of huge amount of data. Most of this big data are available in unstructured form and it is beyond the capability of traditional systems to manage, maintain, supervise, keeping and analyse the data within a limited time span. Effective analysis and interpretation of health care data provides new insights in the condition of patients and suggest the most appropriate treatment opportunities. Discovery and invention of vital information in medical data helps the health care professionals to arrive at appropriate clinical decisions and improvement of quality of life in a variety of patients. In this article, we have discussed various issues and addressed them with the updated information on big data sources, big data management, big data processing and big data analysis through various tools and techniques. We have also analysed and interpreted the recent applications and advancements in artificial intelligence and big data in the health care technology and m-Health domain.

As we are living in the era of big data, huge volumes of a wide variety of complex data-which can be of different levels of veracity-are generated or collected at a high velocity from rich sources of data in various real-life applications. A rich source of these big data sources is the Internet of Things (IoT), which include a collection of sensors, smartphones and other mobile devices, wearable devices, as well as other "things" that are capable to operate within the existing Internet infrastructure. Embedded in these big data are valuable knowledge and useful information. Hence, the research problem of data mining from big IoT data have drawn attention of many researchers as it aims to discover implicit, previously unknown and potentially useful information and knowledge from the data. For instance, frequent pattern mining finds sets of frequently co-occurring items in the IoT domains. Associative classification discovers rules revealing relationships among items within the frequent patterns and their associations with the corresponding class labels. Induction based classification uses decision tree or random forest to learn from old big IoT for classifying or making predictions on new data. Over the past quarter of a century, many serial, distributed, parallel, and MapReduce-based (Hadook-based and Spark-based) big data mining algorithms have been proposed. These algorithms are run in local computers, distributed and parallel environments, clusters, grids, clouds and/or data centers. In this paper, we review some of these algorithms, discuss issues and research prospective in mining classification patterns from these big IoT data in fog. Our case study on a real-life application shows the feasibility on classifying real-life big IoT data over fog for urban analytics.

Obtaining indicators on the innovation activities of firms has been a challenge in economic research for a long time. The most frequently used indicators - R&D expenditures and patents - provide an incomplete picture as they represent inputs in the innovation process. Output measurement of innovation has strongly relied on survey data such as the Community Innovation Survey (CIS). However, this type of data suffers from several shortcomings typical of surveys, including incomplete coverage of the business sector, subjectivity concerns, low timeliness, and limited comparability across industries and firms. An alternative that has attracted growing interest is to use big data sources to collect innovation data at the firm level. This paper discusses recent attempts to use digital big data sources including websites and social media to generate firm-level innovation indicators. It summarises the main challenges of using big data and proposes practical guidelines for their use, including a research agenda that should be useful to practitioners as well as users of statistics derived from big data.