Benchmarking Database Systems for the Requirements of Sensor Readings

Most of the domains of science are facing an explosion in the data that they have to collect and process in order to conduct research. This is true for both: scientific domains dealing with experimental data, e.g. biology, sociology, astronomy etc, but also scientific domains dealing with simulation data, e.g. seismology, physics etc. To maximize the potential outcome of scientific data analysis, respective data management applications need to fulfil the following coarse tasks: fast on-demand data processing, and effective storage and consolidation of diverse data collections. These two tasks are in general hard to realize because of: (a) the big data size, (b) the diversity of data formats, (c) their conceptual dependencies, (d) disperse data locations, and (e) intensive and systematic nature of scientific queries. We present the characteristics of big scientific data collections and their necessities in terms of data management. Based on this discussion, we discuss the structure of a framework for the processing and consolidation of heterogeneous scientific data collections. Such a framework aims to mediate between the user and a set of available data management technologies, such as relational DBMSs, key-value stores and column stores, in order to efficiently direct data management operations (insertions, updates) and especially requests (queries) to the appropriate data management application. The framework aims to distribute, dissect, and schedule data management actions, as well as integrate results, in a way that reduces response time. This entails the accommodation of methods for the selective parallelism and serialization depending on partial results and response times. Also, this entails the accommodation of methods for the gradual alteration of data formats and storage, e.g. storage of semi-structured data or raw data in files into relational databases. Furthermore, we discuss the processing of scientific query bulks or workflows with the possibility to retrieve early partial results and calibrate query parameters.

The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, Key-value stores can effectively support large scale operation, but do not natively support multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present MD-HBase, a scalable data management system for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a Key-value store. The underlying Key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. We present the design of MD-HBase that builds two standard index structuresâ€"the K-d tree and the Quad treeâ€"over a range partitioned Key-value store. Our prototype implementation using HBase, a standard open-source Key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multidimensional range queries and nearest neighbor queries in real-time with response times as low as hundreds of milliseconds.

Abstract: The growing demand for intelligent energy management systems has led to the development of Internet of Things (IoT)-based smart meters that provide real-time monitoring, remote control, and data analysis. This paper presents the design and implementation of a smart energy meter that utilizes Wi-Fi (ESP32), Bluetooth (HC-05), and GSM (SIM800L) technologies to monitor energy consumption and control connected appliances remotely. The system leverages an energy meter for measurement, a microcontroller for processing, a relay module for control, and a user-friendly communication framework. Experimental results validate the effectiveness of the system in real-world scenarios, with applications in residential, commercial, and industrial energy monitoring. The proposed system also supports integration with cloud platforms and mobile applications to enhance accessibility and improve decision-making. Keywords: IoT, Smart Energy Meter, ESP32, GSM, Bluetooth, Energy Monitoring, Home Automation, Blynk, Real-time Monitoring, Remote Control

We describe an online algorithm to simplify large volumes of location and sensor data on the source mobile device, by eliminating redundant data points and saving important ones. Our approach is to use topological persistence to identify large scale sharp features of a data stream. We show that for one-dimensional data streams such as trajectories, simplification based on topologically persistent features can be maintained online, such that each new data-point is processed in O(1) time. Our method extends to multi-resolution simplifications, where it identifies larger scale features that represent more important elements of data, and naturally eliminates noise and small deviations. The multi-resolution simplification is also maintained online in real time, at cost of O(1) per input point. Therefore it is lightweight and suitable for use in embedded sensors and mobile phones. The method can be applied to more general data streams such as sensor data to produce similar simplifications. Our experiments on real data show that this approach when applied to the curvature function of trajectory or sensor data produces compact simplifications with low approximation errors comparable to existing offline methods.

Purpose – The purpose of this paper is to study business models (BMs) for smart meters (SMs) and discuss related issues in the French institutional context. Because SM introduce deregulation on both the demand and supply sides, the authors argue that they represent an opportunity to “unlock” the system by enabling feedback to consumers. The authors discuss the empirical findings from the TICELEC (Technologies de l’Information pour une Consommation Electrique – Information Technology for Sustainable Electricity Consumption Behaviors) project which is an experimental initiative to measure potential energy savings through the implementation of SM, and to test behavioral change. Design/methodology/approach – The empirical data are from the TICELEC project and refer to a municipality in southern France. The project was designed to show the qualitative changes deriving from a new technology, and the quantitative changes in the form of real reductions in residential electricity consumption in the short and medium terms. The authors discuss these changes and their potential replication, and examine the nature of the feedback provided to users and the implications for SM BMs for France and for smart cities more generally. Findings – The authors suggest that the opportunities provided by SM have to be compared with other kinds of intervention such as self-monitoring procedures. The results show that any intervention is important for moderating the sole impact of SM. The findings on the importance of changes to “energy habits” relate mainly to “curtailment” and “low efficiency” behaviors, which represent less costly changes. The lessons learned for BM developments linked to SM include incentive systems, smart tariffs, and technologies to increase potential behavior changes and energy savings in this field. Research limitations/implications – The authors’ analysis of the content of behavioral change shows that curtailment behavior and low-efficiency behavior remain dominant when SMs are implemented. Promoting high-efficiency behaviors is always difficult for reasons of cost. Thus, SM should be combined with other measures such as incentives systems, e.g. “smart tariffication,” and new services to increase their impact. Practical implications – A proper combination of smart tariffs and SMs to reduce peaks in demand would appear to be critical to boost SM development. It will also be important to integrate SMs with smart grids to improve energy efficiency and exploit renewables and energy storage in electricity networks. Social implications – SMs are important but any interventions that motivate households to change their energy habits also help in the French context. SMs enable households to try to reduce their energy consumption but they are not the solution. Originality/value – There are no detailed results published for France. Utilities such as Electricite Reseau Distribution France, have introduced R & D programs oriented to the deployment of SM which have been tested since 2009 (e.g. see the local LINKY meter projects in Lyon and Touraine). The empirical data are from the TICELEC project and refer to a municipality in southern France. The project was designed to show the qualitative changes deriving from a new technology, and the quantitative changes in the form of real reductions in residential electricity consumption in the short and medium terms. The authors discuss these changes and their potential replication, and examine the nature of the feedback provided to users and the implications for SM BMs for France and for smart cities more generally.

Various key-value (KV) stores are widely employed for data management to support Internet services as they offer higher efficiency, scalability, and availability than relational database systems. The log-structured merge tree (LSM-tree) based KV stores have attracted growing attention because they can eliminate random writes and maintain acceptable read performance. Recently, as the price per unit capacity of NAND flash decreases, solid state disks (SSDs) have been extensively adopted in enterprise-scale data centers to provide high I/O bandwidth and low access latency. However, it is inefficient to naively combine LSM-tree-based KV stores with SSDs, as the high parallelism enabled within the SSD cannot be fully exploited. Current LSM-tree-based KV stores are designed without assuming SSD's multi-channel architecture.To address this inadequacy, we propose LOCS, a system equipped with a customized SSD design, which exposes its internal flash channels to applications, to work with the LSM-tree-based KV store, specifically LevelDB in this work. We extend LevelDB to explicitly leverage the multiple channels of an SSD to exploit its abundant parallelism. In addition, we optimize scheduling and dispatching polices for concurrent I/O requests to further improve the efficiency of data access. Compared with the scenario where a stock LevelDB runs on a conventional SSD, the throughput of storage system can be improved by more than 4X after applying all proposed optimization techniques.

As cloud computing continues to evolve, organizations are finding new ways to store the massive amounts of big data that are collected. Big data storage often require greater flexibility and scalability which can be provided by incorporating NoSQL technologies. NoSQL (Not Only SQL) is quickly becoming a popular approach to store large and unstructured data. This paper looks at the various classifications of NoSQL technologies as well as many of the notable characteristics of the technologies. The authors also describe some deficiencies of using NoSQL and give some explanation to why companies are adopting the technology. The paper concludes with suggestions for future research of NoSQL technologies and a content analysis of current articles in database management is provided in the appendix. Keywords: big data, cloud computing, document databases, MongoDB, NoSQL, Not Only SQL, unstructured database INTRODUCTION Since NoSQL database technologies entered the market in 1998, these technologies have challenged the widely-accepted relational structures and database management practices supported by traditional SQL databases. Despite this trend, the perception that NoSQL seeks to eliminate the use of relational databases is a common misconception. NoSQL is best considered with the acronym NOSQL - Not Only SQL - which more accurately represents an approach that combines non-relational databases with the use of relational ones. This approach seeks to leverage both NoSQL and SQL technologies in order to balance the demands of performance, scalability, and schema flexibility with data integrity and consistency (Purdue, 2014; Penchikala, 2013). The rise of web-scale applications creates performance, scalability, and schema flexibility demands that present significant challenges for traditional SQL / relational databases. Thus, a variety of new entrants have entered the market, each seeking to address specific challenges, including (Kent, 2012): * The need to store large volumes of data about users, objects and products * The need to provide near-real-time transaction processing * The need to provide high-speed analytics on large volumes of data (big data) * The need to support flexible data schemas for evolving business environments * The need to scale applications horizontally with predictable hardware and licensing cost structures Each NoSQL solution seeks to address one or more of the above issues, often by providing alternative approaches to long-established relational database norms. Traditional relational databases were not designed to cope with the scalability and flexibility challenges facing modern applications. Older databases were not designed to take advantage of the inexpensive storage and excess processing power available today. The defining characteristics of NoSQL technologies, as well as the relative advantages and disadvantages of NoSQL, and the subsequent influence on adoption of NoSQL technologies are each the subject of a small, yet growing body of research. This paper provides a literature review of that research and proposes potential areas where future research may focus. REVIEW OF LITERATURE NoSQL database technology is a relatively recent innovation and much of the research surrounding it is of an emerging nature. For the purpose of this literature review Galileo was primarily used for identifying the articles. Articles were selected that focused directly on NoSQL database technologies (unstructured / semi-structured databases, document databases, etc.) and articles that focused on relational databases. The contrasting articles provided the researchers with the ability to compare the features that are offered with a greater degree of variability in NoSQL databases (for example, an article on SQL scalability provided a foundation for comparing newer horizontal scaling techniques used in certain NoSQL databases). …

The argument of SQL vs NoSQL always requires the answer It depends. If the participants in the argument about which database is better or more appropriate in a given situation have no background in any NoSQL database, then it is difficult to make informed decisions. In general, NoSQL databases arrange themselves into four groups: key-value stores, columnar databases, graph databases, and document databases. This paper describes a project to use gamification to encourage members of a busy corporate IT department to learn about the four groups of NoSQLs and give them brief hands-on experiences with one of each of the four categories: Redis, Cassandra, Neo4j, and MongoDB. The encouragement was done by using a combination of physical and virtual treasure hunts. Eight fun treasure hunts were created (two for each group of NoSQL) where participants in the office either worked individually or teamed up with other participants to find data hidden in each of the different types of NoSQL database. This data would then lead to a physical treasure box full of chocolates which was hidden in the office common area.

During data modelling, NoSQL databases are usually considered schema-less databases. They do not support rigid schema as traditional databases. Instead, offer flexibility and support dynamic schema, which means schema design can be changed at run time to support analytical tasks. NoSQL databases allow storing similar information with multiple schemas, which facilitates data evolution by eliminating restrictions on the schema. It raises the question of whether the schema matters in NoSQL databases, given that explicit schema declaration is not required before data storage. Furthermore, if the schema does matter, what is its potential impact on application performance? This paper examines these questions by taking three types of NoSQL databases: Document store, Column store, and Key-Value store. All possible schema structure alternatives are designed using a sample Entity Relationship (ER) model. Then, a set of common queries are performed on various schema alternatives to analyse the impact of query performance in terms of query response time, speedup factor, read and write Latency, and database size. The results suggest that the scheme alternatives affect the database performance in different NoSQL models.

The urge to manage daily energy consumption has now become a worldwide priority. The Internet of Things (IoT)-based smart meter has been widely used to monitor energy consumption in many households. LoRaWAN connectivity is confirmed to be one of the most reliable connectivity to be used as a network system for IoT use cases. It can fulfill the needs of low power, long-range, efficient, and affordable IoT technology in utility energy monitoring. LoRaWAN-based smart electric meters are designed in rural areas for monitoring and managing electric energy consumption. The smart electric meter system consists of smart kWh meters, LoRa gateway, network system, and dashboard for energy usage monitoring. The system design process consists of feature design, architecture design, data flow design, LoRa gateway site planning, and platform setup. The implementation can be concluded that LoRa can be used for smart electric meters in rural areas. A gateway can reach a maximum distance of 1.58 km, with an RSSI of -99.40 dBm ±4.56 dBm. LoRa gateway availability can reach 100% by implementing gateway protection systems. The LoRaWAN-based smart electricity meter has also proven to be costoptimized, efficiently reducing monthly costs to 92.12%.

The Sensor Web provides an interoperable way to discover, task, and retrieve sensor data over the network. The approach to knowledge discovery from the Sensor Web needs to explore data ingestion, domain models, and data processing. This chapter leverages recent advances in autonomous control and the Semantic Web technologies to present a systematic approach in which the Sensor Web and Earth science models are annotated with semantic metadata and chained together in an autonomous way as a service chain to simulate the process of Sensor Web mining based on semantic inference. This approach significantly advances sensor data exploration with semantics in an interoperable way to improve the accuracy and timeliness of monitoring and predicting rapidly changing Earth phenomena.

Novel occupancy detection method based on convolutional neural network model using PIR sensor and smart meter data

Databases play an important role in Information and Communication Technologies and data has no significance unless it is converted into meaningful information. Relational or SQL disk databases where data is structured in tables have served many applications over the years. The increasing need for faster data access in real-time systems has changed the traditional Disk-based database into an In-memory database where data is manipulated and stored directly in the main memory thereby reducing disk input/output operations. With the emergence of IoT and Big Data, NoSQL databases now represent an alternative to traditional relational databases in terms of simpler design and faster operations on large data volumes. This work compares three types of databases namely Relational On-disk, Relational In-memory and NoSQL On-disk/In-memory in terms of Insert, Select, Update and Delete operations together with Indexes, Stored procedures and Triggers in an IoT environment. A plant environment has been monitored in real-time using sensor data such as Temperature/Humidy, Moisture and Water level. Data was stored in different databases and the time taken to execute various operations were recorded. Experiments using the water level parameter demonstrate there is no specific database that provides the best performance for all data operations.

Sensor networks are increasingly becoming one of the main sources of Big Data on the Web. However, the observations that they produce are made available with heterogeneous schemas, vocabularies and data formats, making it difficult to share and reuse these data for other purposes than those for which they were originally set up. In this thesis we address these challenges, considering how we can transform streaming raw data to rich ontology-based information that is accessible through continuous queries for streaming data. Our main contribution is an ontology-based approach for providing data access and query capabilities to streaming data sources, allowing users to express their needs at a conceptual level, independent of implementation and language-specific details. We introduce novel query rewriting and data translation techniques that rely on mapping definitions relating streaming data models to ontological concepts. Specific contributions include: • The syntax and semantics of the SPARQLStream query language for ontologybased data access, and a query rewriting approach for transforming SPARQLStream queries into streaming algebra expressions. • The design of an ontology-based streaming data access engine that can internally reuse an existing data stream engine, complex event processor or sensor middleware, using R2RML mappings for defining relationships between streaming data models and ontology concepts. Concerning the sensor metadata of such streaming data sources, we have investigated how we can use raw measurements to characterize streaming data, producing enriched data descriptions in terms of ontological models. Our specific contributions are: • A representation of sensor data time series that captures gradient information that is useful to characterize types of sensor data. • A method for classifying sensor data time series and determining the type of data, using data mining techniques, and a method for extracting semantic sensor metadata features from the time series.

Electricity grids around the world are undergoing a fundamental transformation, thanks to the modernization of electricity distribution systems including smart meter deployment and applications. Data generated by smart meters provides a wealth of information that can help better understand and optimize the operation of electricity networks. This paper proposes a novel Hybrid Load Profile Clustering (HLPC) algorithm to identify patterns of electricity consumption of users from large volumes of residential electricity consumption interval data. The HLPC algorithm is in particular advantageous in detecting `spike' patterns of consumption among other various residential consumption patterns. Experimental results about the HLPC algorithm are presented using interval data from 600 smart meters in Victoria, Australia, to demonstrate that the proposed methodology outperforms standard clustering algorithms.