An agricultural data storage mechanism based on HBase

Biodiversity research is seeing unprecedented global collaboration with initiatives such as the Earth BioGenome Project, an effort to sequence all known eukaryotic life, and Genesys, a global database for sharing crop genetic resources. However, as in other disciplines, public funding and policy for scientific research in agriculture tend to follow national borders even when science and its collaborations do not. In addition, agriculture is similar to biomedicine in having significant private investment in research and development where competition could inhibit sharing. It would seem that significant challenges lie ahead for making progress on ambitious global initiatives at least where agricultural samples, collections, and data are concerned. In this talk we will review several activities at the United States Department of Agriculture that illustrate how policy and infrastructure can overcome difficulties. For example, recent policies for openness of publicly-funded research products and adoption of FAIR data principles even for private or proprietary data hold promise and have elevated the importance of data infrastructure. The US Department of Agriculture's Agricultural Research Service (ARS) launched its Ag100Pest contribution to the Earth BioGenome Project, including the use of the i5K Workspace@NAL platform for its sequenced and annotated genomes. The GRIN Global platform supports not just USDA germplasm data management but a growing network of plant and animal researchers and collections around the world. The Ag Data Commons provides standardized metadata and machine-readable data dictionaries to the publicly accessible products of these and other USDA-funded efforts. It is teaming with the ARS high performance computing system SCINet to explore cost-effective public access to big data storage for agricultural data and models. Finally, many of these efforts extend and contribute back to widely used open source software systems. While challenges remain in coordinating and sustaining these efforts with international stakeholders, engagement with groups like AgBioData, the Research Data Alliance (RDA) Interest Group on Agricultural Data, and the Global Open Data for Agriculture and Nutrition coalition will continue to bear fruit (pun intended). We seek similar engagement with the broader biodiversity data community in order to ensure that policy and infrastructure investments result in maximum mutual benefit.

With the continuous advancement of digital informationization, the agricultural field has naturally entered the era of big data. At present, big data technology has achieved remarkable results in the Internet industry, but it is still in its infancy in the agricultural field. Due to the complexity of the agricultural field itself and the obvious diversity and heterogeneity of data, the agricultural Internet of Things in the big data environment is facing many difficulties and challenges, especially the collection and storage of massive data. The purpose of this article is to study the application of agricultural big data. This article briefly introduces the background of the development of agricultural big data and the significance of developing agricultural big data, and summarizes the meaning of big data. At the same time, this article summarizes and analyzes the major applications of agricultural big data in production process management, agricultural resource management, and agricultural ecology environmental management, agricultural products and food safety management. In the experimental part, this paper designed and implemented an agricultural big data system platform based on Hadoop. This article simulated the query of 10, 100 and 1000 records respectively for 10,000, 100,000, 1 million and 10 million records, and the average response time was 56ms. The experimental results show that the performance of the agricultural big data platform designed in this paper can meet the needs.

Continued population growth and land intensification put increasing pressure on agricultural production and point to a need for a ‘step change’ in agriculture to meet the demand. Advances in digital technology—often encapsulated in the term ‘big data’—are increasingly assumed to be the way this challenge will be met. For this to be achieved, it is necessary to understand the ways that farmers and other industry stakeholders perceive big data and how big data might change the industry. It is also necessary to address emerging moral and ethical questions about access, cost, scale and support, which will determine whether farms will be able to be ‘big data enabled’. We conducted a discourse analysis of 26 interviews with stakeholders in the grains industry in Australia. Two main discourses were identified: (1) big data as a technology that will significantly benefit a few larger farms or businesses—Big Data is for Big Farming—and conversely (2) big data as a way for every farmer to benefit—Big Data is for Everyone. We relate these findings and the literature on adoption of technology and social studies in agriculture to the potential of farmers to embrace big data, from basic concerns about network infrastructure through to more complex issues of data collection and storage. The study highlights that there are key questions and issues that need to be addressed in further development of digital technology and big data in agriculture, specifically around trust, equity, distribution of benefits and access. This is the first study of big data in agriculture that takes a discourse analysis approach and thus interrogates the status quo and the prevailing norms and values driving decisions with impacts on both farmers and wider society.

There are nearly 50 million AIS data generated every day in the Yangtze River Basin. Efficient storage and query of AIS data become the key problem of smart maritime administration. To overcome this problem, a distributed storage and query scheme based on Hadoop platform is proposed. Firstly, the original AIS message is parsed, the abnormal data is removed and the parsed data is pushed to the Kafka message queue. Secondly, the messages in the queue are pushed to different servers to complete the storage of real-time cache, historical trajectory and regional information. Finally, the query interface is provided to realize the real-time query of ship status, trajectory and regional ship in a certain period of time. The real applications show that compared with the traditional relational database, the solution can meet the requirements of real-time query and has high scalability.

As cloud computing continues to evolve, organizations are finding new ways to store the massive amounts of big data that are collected. Big data storage often require greater flexibility and scalability which can be provided by incorporating NoSQL technologies. NoSQL (Not Only SQL) is quickly becoming a popular approach to store large and unstructured data. This paper looks at the various classifications of NoSQL technologies as well as many of the notable characteristics of the technologies. The authors also describe some deficiencies of using NoSQL and give some explanation to why companies are adopting the technology. The paper concludes with suggestions for future research of NoSQL technologies and a content analysis of current articles in database management is provided in the appendix. Keywords: big data, cloud computing, document databases, MongoDB, NoSQL, Not Only SQL, unstructured database INTRODUCTION Since NoSQL database technologies entered the market in 1998, these technologies have challenged the widely-accepted relational structures and database management practices supported by traditional SQL databases. Despite this trend, the perception that NoSQL seeks to eliminate the use of relational databases is a common misconception. NoSQL is best considered with the acronym NOSQL - Not Only SQL - which more accurately represents an approach that combines non-relational databases with the use of relational ones. This approach seeks to leverage both NoSQL and SQL technologies in order to balance the demands of performance, scalability, and schema flexibility with data integrity and consistency (Purdue, 2014; Penchikala, 2013). The rise of web-scale applications creates performance, scalability, and schema flexibility demands that present significant challenges for traditional SQL / relational databases. Thus, a variety of new entrants have entered the market, each seeking to address specific challenges, including (Kent, 2012): * The need to store large volumes of data about users, objects and products * The need to provide near-real-time transaction processing * The need to provide high-speed analytics on large volumes of data (big data) * The need to support flexible data schemas for evolving business environments * The need to scale applications horizontally with predictable hardware and licensing cost structures Each NoSQL solution seeks to address one or more of the above issues, often by providing alternative approaches to long-established relational database norms. Traditional relational databases were not designed to cope with the scalability and flexibility challenges facing modern applications. Older databases were not designed to take advantage of the inexpensive storage and excess processing power available today. The defining characteristics of NoSQL technologies, as well as the relative advantages and disadvantages of NoSQL, and the subsequent influence on adoption of NoSQL technologies are each the subject of a small, yet growing body of research. This paper provides a literature review of that research and proposes potential areas where future research may focus. REVIEW OF LITERATURE NoSQL database technology is a relatively recent innovation and much of the research surrounding it is of an emerging nature. For the purpose of this literature review Galileo was primarily used for identifying the articles. Articles were selected that focused directly on NoSQL database technologies (unstructured / semi-structured databases, document databases, etc.) and articles that focused on relational databases. The contrasting articles provided the researchers with the ability to compare the features that are offered with a greater degree of variability in NoSQL databases (for example, an article on SQL scalability provided a foundation for comparing newer horizontal scaling techniques used in certain NoSQL databases). …

Ministry of Agriculture, Republic of Indonesia provide Indonesia agricultural commodities data which presented in the simple tabular format. It is a challenging task to analyze this growing data and generate data summarization. This study was conducted to implement online analytical processing (OLAP) application to provide an analytical tool for agricultural commodities data. The OLAP application is built using RESTful website architecture with Node.Js and Angular.Js. The back-end side uses Node.Js as a server and PostgreSQL as data storage. In front-end side, we use HTML5, Wijmo, and Angular.Js for visualizing data summary in the form of crosstab table and chart. OLAP operations are used in this application to perform analysis of some feature of agricultural commodities data. The features include crop production, productivity, and harvested area.

This paper designs a cloud platform for agricultural products e-commerce information service based on big data, relying on the Internet of Things, big data, cloud computing and other technologies. The data acquisition layer mainly uses IoT perception to collect data. The system completes data conversion, processing, and analysis in the big data center, and realizes the application through the visual display of the application layer. The development system uses ASP.Net technology and uses C# as the programming language. This article uses Silverer database as the background database to build an agricultural product e-commerce platform. The research shows that this paper applies big data technology to the field of agricultural product information service to realize real-time sharing of agricultural product big data. The system is conducive to the effective and full utilization of agricultural product data resources.

In the last few decades, a novel branch of intelligent computation algorithms has been inspired by a swarm intelligence theory which imitates the behavior of animals. These algorithms are successfully applied to solve many kinds of simple and complex problems in several fields such as optimization problems, pattern recognition, image processing, features section, and task scheduling in cloud and parallel computing. Cloud computing has not only become the preferred environment for several companies, but also helps others to overcome many server-related issues by utilizing characteristics such as reliability, flexibility, high scalability, and security. Therefore, many intelligent computation algorithms are used to improve this environment. In this chapter, an overview of swarm intelligence for solving the scheduling problems of tasks in cloud computing is presented, including particle swarm optimization, cat optimization algorithm, artificial bee colony, lion optimization algorithm, whale optimization algorithm, bat algorithm, gray wolf optimizer, cuckoo search algorithm, hybrid swarm algorithms, and multi-objective swarm optimization. All these algorithms are described and presented with their achievements in solving task scheduling issues in cloud computing.

Agriculture is facing and causing many challenges such as increased GHG emissions, food security issues, water scarcity. In addition, the growing world population creates an increasing demand for food production, which requires all countries of the world to join hands and cooperate to achieve food security. Agriculture data collection and analysis using emergent technologies such as IoT devices, cloud services, and big data is in the early development stages; however, it has been adopted and experimented with in many regions worldwide. These technologies demonstrated the ability to improve crop productivity and quality, minimize resource consumption, and reduce the impact on the environment. This study aims to establish the first phase in ongoing research on “AGRO Big Data: Toward Smart farming in the Kingdom of Bahrain” by designing a conceptual IoT-based framework that employs cloud services and big data tools. The kingdom of Bahrain is a small island that suffers from water scarcity and harsh soil conditions. It is considered a non-agriculture country and highly dependent on imported crops. The proposed framework will identify different agriculture stakeholders (users) in Bahrain and data domains such as soil sensors and weather stations. It will suggest using a mobile application (AgroBahrain) to collect and disseminate farm-related data along with the adoption of Google cloud services for the storage and transformation of agriculture big data. The framework is divided into two modules: 1- data collection and storage module, 2- data diffusion module. It concentrates on plants and crops data, excluding cattle and livestock. The framework is the first to build a conceptual process of collection, storage, and transformation of agriculture data in Bahrain. It is aligned with the kingdom's agriculture strategy developed early 2010. Moreover, the framework paves the road towards advancing the research in different agriculture-related fields, such as soil and crop quality.

The existing quality supervision system of agricultural products is facing challenges such as quality inspection data fraud, data privacy disclosure, and low credibility. In order to solve these problems, this paper designs and implements a Secure Edge-Cloud-BlockChain Integrated Agricultural Product Safety Supervision System based on blockchain and cryptography technology (SECBCAPSS), which enables secure, reliable, and efficient storage of detection data as well as trustworthy traceability. First, quality inspection entities and regulatory entities are deployed as blockchain nodes to achieve transparent supervision and reliable traceability of agricultural product data. Then, a two-stage encryption algorithm is designed and implemented: the SM4 algorithm is used to protect the key data of the inspection record, and the encryption scheme based on the Ciphertext-Policy Attribute-Based Encryption (CP-ABE) is used to protect the SM4 key and the detection result certificate. Then, a reliable agricultural product quality detection data storage mechanism is designed to ensure the consistency between cloud storage data and blockchain storage data. The experimental results show that the system function and performance meet the actual needs, can resist malicious data access attacks, and effectively reduce the data storage burden of the blockchain.

Strict Timed Causal Consistency as a Hybrid Consistency Model in the Cloud Environment

In the recent era of data science, AI, and machine learning the amount of data has increased exponentially, and future growth is predicted to be even greater. For the last 4 decades RDBMS is way for data storage, but with the rise of cloud computing online applications have taken place of the stand-alone applications, while distributed servers have taken the place of traditional dedicated servers. Cloud computing has become very efficient because of the lower cost, ability to scale up and down, flexibility, and pay-as-you-go model. This is the major change after the birth of the Internet. Cloud computing has revolutionized the way data is stored and accessed. Cloud databases are an integral part of the cloud computing paradigm, offering scalability, flexibility, and cost-efficiency for managing large volumes of data. Cloud databases such as Amazon RDS, IBM DB2 and MongoDB are becoming popular. Cloud databases find the restrictions of older relational databases. Data storage, retrieval, modification, and analysis are essentially the functions of cloud databases. Cloud databases are mostly used by data-intensive applications including data warehousing, data mining, and business intelligence. Applications for supply chain management, online e-commerce, and transactional data management, including banking and airline reservations, also benefit from cloud storage. This research paper reviews the concept of cloud databases, exploring their features, it examines the architecture of cloud databases, the comparison between traditional Relational Database Management Systems (RDBMS) and NoSQL databases, and popular implementations in the cloud environment. Additionally, the challenges faced in their development and deployment.

With the rapid development of national economy, our demand for power resources has also increased rapidly. The traditional power grid can not be achieved operational status of equipment for effective monitoring and control, we will not be able to ensure the quality and quality of electricity service. Cloudbased build smart grid information platform enables gathering the parameter of grid operation state, greatly increasing the economic efficiency of power enterprises. Firstly, cloud computing and smart grid briefly described, and then pointed out that cloud computing applications in the smart grid information platform, and specialized of cloud-based smart grid information platform, in order for the power grid enterprises to enhance their level of service and provide effective reference. KEYWORD: cloud computing; smart power grids; information platform; Application 4th International Conference on Mechanical Materials and Manufacturing Engineering (MMME 2016) © 2016. The authors Published by Atlantis Press 202 3 THE MAIN PROBLEM IS STILL THE SMART GRID INFORMATION PLATFORM FACING The transition to a cloud computing data center. Our data center because of the different regions and the degree of information presents different characteristics, some data centers already have a basic cloud computing capabilities, but most data centers still remain in the initial level, to ensure the normal operation of the existing data center power under the premise, gradual transition to cloud computing data center in order to meet future requirements of the development of smart grid. Restricting data transmission problem. State Grid Corporation between provincial branches and there are a lot of data needs to be transmitted, will bring massive data transmission efficiency decreases, taking into account the load balancing device between the cloud node, thus building a wide area network channels to resolve data transmission problem. Security of data transmission. Data Center for Cloud Computing power companies to build internal private cloud, capable of self-maintenance and management, system wide area network data transmission services, according to the third security zone defined security partition, and install proprietary equipment production area and external public network isolation . 4 CLOUD COMPUTING APPLICATIONS IN THE SMART GRID INFORMATION POLICY PLATFORM Integrate and optimize the management of heterogeneous resources. Among the smart grid systems, data resources, information systems between various platforms and mutually dispersed, severe heterogeneity, thus leading to the integration of the quality of information sharing to increase difficulty, unable to effectively improve network management efficiency for the power distribution system electrical work quality management and marketing operations system maintenance upgrade will not be able to effectively meet demand (Jie Yang, 2014). Cloud computing technology enables the power system to effectively integrate the various business data and computing resources on the basis of information for business collaboration platform built on effective, able to satisfy the huge demand for smart grid information resource sharing. Cloud computing network virtualization and application virtualization technology, the server virtualization technology and other resources to achieve each service abstraction, and then realize the difference between the system and the software effectively eliminated by the appropriate method to achieve the effective integration of information resources, and then to optimize the management, greatly improving the efficiency of heterogeneous resources management applications. Distributed storage and management of large volume of data. After application of smart grid information platform construction, customer service requests varied, huge data volume information brought it to the storage and management of data provides a great challenge. Cloud computing technology in general face amount of data each smart grid system and unit collected using forms distributed storage of data to achieve the effective storage, greatly to ensure the security and reliability of data storage. GFS effective use of technology with each node, the master server is responsible for the effective management of the whole file system, the file system and complete the work of the metadata stored in the data block is the main server for data storage block sliced, and by saving three or more redundant backup storage. At the same time, client access and complete access to the main block server information. Fast Power System Parallel Computation and Treatment. Smart grids and information platform for the large volume of data to provide stable operation of the power grid data base, but more critical is the analysis and design of each system operating data, planning and decision-making, and even failures and transient stability, state estimation and intelligent a variety of computing and content analysis and decision-making process, its huge computing needs, general computing system can not reach its goal (Wang, Mengxue, 2015). Cloud computing technology through parallel computing technology has greatly enhanced parallel processing capability of the power system, which provides parallel programming model makes more simple and efficient parallel algorithms. MapReduce as Google made an important parallel programming model, and its role in large data sets in parallel computing plays a very prominent. Parallel processing job results by the client to achieve effective user submitted, via decomposition automated master role, the corresponding task scheduling to working nodes and nodes work completed by the specific calculations and perform the appropriate task. 5 SMART GRID INFORMATION PLATFORM CLOUD COMPUTING ANALYSIS Cloud Computing Smart Grid Information Platform Architecture. Smart grid technology information platform architecture cloud computing include the following several levels and aspects (Yan Cui, 2015): First, that the platform infrastructure layer is the integration of the various hardware resources and related management capabilities of virtualized using virtual technology to achieve the corresponding hardware devices such as computers and network storage devices abstraction, and then complete the automation and optimization of resource data and load management; secondly, the main platform for the integration of the various layers of software

The agricultural sector holds the greatest reduction potential to limit adverse effects of reactive nitrogen in the environment. Assessing the negative impacts of excessive release of reactive nitrogen into the biosphere requires spatially explicit information to capture e.g. hot spots of nitrogen surplus, nitrogen use efficiency, the impact on sensitive ecosystems or on ground/drinking water quality. The agricultural economic model CAPRI is one of the main tools applied by the European Commission for the ex-ante analysis of the impact of agricultural policies and agro-environmental legislation at regional level (NUTS2) in Europe. CAPRI builds on long-term time series of regional, national and international agricultural statistics (e.g. crop and livestock production, fertilizer use), market and trade data. Inputs (e.g. inorganic fertilizer) in CAPRI are explicitly linked to production which delivers the basis for connecting environmental indicators (e.g. nitrogen surplus) directly to individual activities. To provide the link between the agro-economic model CAPRI and the impact assessment of nitrogen use in agriculture on the environment at higher spatial resolution, we developed a procedure to disaggregate CAPRI regional data and provide maps of nitrogen input/output, crop and livestock production for the time series 2000 – 2018 at the level of Farm Structure Units (FSU) for 26 EU member states and the UK. The FSU are built by the spatial intersection of a 10 x 10 km2 INSPIRE compliant grid, the CAPRI NUTS2 region borders and the soil mapping units of the Harmonized World Soil Data Base (FAO/IIASA/ISRIC/ISS-CAS/JRC, 2009), having a median area of 12 km2 (minimum 1 km2, maximum 100 km2). The disaggregation procedure for 36 crop types and 18 livestock categories from the regional level to the FSU is driven by information from the gridded Farm Structure Survey (FSS, 2010) crop and livestock data at 10 x 10 km2 resolution, CORINE (2018) non-agricultural land cover shares, altitude and slope constraints for individual crops / crop classes derived from LUCAS survey data (https://ec.europa.eu/eurostat/web/lucas). Nitrogen inputs from mineral fertilizer and manure are disaggregated from the regional level to the FSU following the crops’ requirements. N input from atmospheric deposition and N supply by biological fixation is taken into account at FSU level. N from mineralization of soil organic matter could not be taken into account due to lack of data. N losses from volatilization and surface run-off, N removal by harvest, N in crop residues complete the N flow data set in agricultural areas at FSU level.

The Food and Agriculture Organization of the United Nations (FAO) predicts that the global populationwill rise to 8 billion people by 2025 and 9.6 billion by 2050. To face challenges of food production there is need to develop methods to produce more output with the limited available natural resources. Climate-smart agriculture is an approach for guiding actions required to help stakeholders around the world and identify and develop strategies in order to make agriculture more productive and sustainable. Climate change will increase the vulnerability of agricultural production systems unless scientists and farmers reorient their present approaches toward making them climate-smart or climate-resilient. Use of cloud computing technology in agriculture sector has wider opportunity in the overall development of India. Cloud computing service provider may also offer services like Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS) with affordable cost. Cloud computing has been used for storage of agriculture data by government and private agencies. Use of IOT along with cloud computing can help the farmers a lot to increase the production by providing the correct communication between objects and charging according to the usage of service. Internet of things (IoT) will be beneficial to connect the devices and collect and distribute the information. IOT and cloud computing is a technology which tends to connect various objects in the world to the internet. Use of technology is constantly increasing to improve food production and commercial activities.