Smart Meter Data Analysis Using Big Data Tools

In Smart Grids the Information and Communication Technologies (ICT) could be used to better manage both consumption and production of electricity. The increasing presence of renewable energy sources in production and the permeation of novel consumption types (e.g. Plug-in Hybrid Electric Vehicles (PHEV)) will obviously cause the increase the fluctuation of electrical energy. One possible solution to these problems is development of novel methods for investigating electrical power consumption data series. As the existing learning algorithms of pattern classification are suitable for discovering internal structures of large datasets, it is important to generate a training/testing/validation learning database from existing measurements (e.g. from smart meters), actually via segmentation and labeling by hand. In this paper we propose a novel method for the automatic segmentation with a predefined confidence level. The algorithm is based on the generalized Jensen-Shannon divergence (JSD), and it estimates the change-points (CPTs) in electrical power consumption data. Both the method and some recent results in segmenting one household's power consumption data are presented in this paper.

Extremely randomised trees machine learning model for electricity theft detection

Data classification of users’ electricity consumption provides an in-depth analysis for users’ electricity consumption status, which plays a vital role in the management and distribution of electric energy. So, some data classification methods have been proposed to solve the classification problem of electricity consumption data. However, plaintext-based data classification may bring about the privacy leakage of electricity consumption data. In this paper, we propose a privacy-preserving classification scheme for electricity consumption data under fog computing-based smart metering system, which is based on convolutional neural network (CNN) model with fully homomorphic method (CKKS). The target of our proposed scheme is to solve the leakage problem of private electricity consumption data during the classification procedure. In our scheme, an improved K-means-based labeling algorithm is constructed to process historical electricity consumption data, which is used as the sample data to train the CNN classification model by cloud server. Also, the fog nodes are only permitted to obtain the related ciphertext parameters of the trained CNN model, and perform the classification of ciphertext-based electricity consumption data generated by fully homomorphic method. Based on the classical testing data, the experimental results show that our proposed classification scheme can provide the high classification accuracy of electricity data while protecting the privacy of electricity data.

A shape-based clustering method for pattern recognition of residential electricity consumption

Relevance. The increasing electricity consumption in the private residential sector, driven in part by the growing use of electric heating, is leading to higher loads on 0.4 kV power transmission lines. Traditional standardized load profiles do not always reflect modern consumption patterns and conditions, which creates risks of inaccurate assessments of the electrical grid’s capacity and necessitates more precise modeling of grid operating conditions.Purpose. To develop approaches for classifying consumers and identifying statistically significant patterns in electricity consumption in private residential areas for subsequent calculation of grid operating conditions.Methods. The analysis was based on half-hourly electricity consumption data from 42 private houses, collected via an Automated Meter Reading and Management System (AMRMS). The data was cleaned using the three-sigma rule to remove gaps and outliers, and heat maps were used to identify non-representative consumers. The statistical significance of differences was determined using analysis of variance (ANOVA) and Tukey’s test. Based on median consumption values, consumer groups were formed (low and high electricity consumption). Data processing and visualization were performed using MS Excel, Python (Pandas, NumPy, SciPy libraries), and the Statistica software package.Results. The analysis confirmed statistically significant differences in electricity consumption between most of the houses (F = 2065.4, p < 0.001). Tukey’s test showed that within each group, homes exhibited relatively stable energy consumption values, while intergroup comparisons revealed substantial variations in electricity usage. As a result of the study, two consumer types were identified: "low" and "high" consumption groups. The high-consumption group exhibited distinct evening peaks (18:00–22:00), whereas the low-consumption group had a more evenly distributed load profile.Conclusion. The application of statistical analysis methods to electricity consumption data enabled the simplification of household classification into two main groups and the development of typical consumption profiles. These results were integrated into the LineCapacity software, facilitating grid operation calculations and reducing the risk of misjudging the available power transmission capacity. A promising research direction is planned, focusing on expanding the dataset on residential electricity consumption. This will allow for the consideration of seasonal factors and the development of simulation modeling mechanisms for various consumer groups.

With the development of IoT technology, smart grid has gradually replaced the traditional grid. Smart grid is convenient and fast. It can provide real-time residential electricity monitoring and forecasting, give users better electricity guidance and save a lot of labor costs. Smart meters send customers’ electricity consumption data to the gateway, which aggregates the data and then sends it to the electricity consumption control center. But in this process, there will be a security problem of leakage of customer’s electricity consumption data. Most of the current user data privacy protection collection schemes use homomorphic encryption and randomization techniques. However, some of these schemes require a trusted third-party entity, and some may cause significant computational overhead. Due to the limited computational resources of smart meters, these techniques may be impractical. In this paper, we propose a local differential privacy data aggregation protection scheme based on the idea of grouping perturbation of electricity consumption data according to data domains. Experiments show that our scheme can provide statistical estimates of electricity in the region while satisfying the privacy protection of customers’ electricity consumption data. Moreover, our scheme has small computational and communication overheads, which can meet the application requirements in practical scenarios.

The enhancement of the intelligent construction of the power grid and widespread popularity of smart meters enable large amounts of electrical energy consumption data to be collected and analyzed. Based on the data, the energy provider gives a guiding price in the future periods to users. It encourages users to be more economical and smarter in the process of using electricity. By applying the social welfare model to equate demand and supply in every time interval, we gain the optimal prices and generation capacity. Nevertheless, the truth is that there is a great gap between the consumers’ booked electrical energy consumption and the optimal generation capacity, causing the power system overload and even outage. This article puts forward a novel automatic process control strategy in order to monitor the gap between the consumers’ booked electrical energy consumption and optimal generation capacity by using statistical method to predict the future one. When the predicted value exceeds the boundary, the energy provider adopts the changeable electricity price to stimulate consumers to adjust their electrical energy demands so that it can have smoothly actual electrical energy consumption. Our adjustment method is data-driven exponential function-based adjustment. Case study results show that the strategy can obtain small adjustment times, stable actual consumption load, and controllable prediction errors. Different from the linear monitoring and adjustment strategy, our approach obtains almost the same adjustment frequency, less standard deviation of residuals, and higher total social welfare and energy provider profit.

A data mining-based framework for the identification of daily electricity usage patterns and anomaly detection in building electricity consumption data

The relationship and causality direction between electricity consumption and economic growth is an important issue in the fields of energy economics and policies towards energy use. Extensive literatures has discussed the issue, but the array of findings provides anything but consensus on either the existence of relations or direction of causality between the variables. This study extends research in this area by studying the long-run and causal relations between economic growth, electricity consumption, labour and capital based on the neo-classical one sector aggregate production technology mode using data of electricity consumption and real GDP for ASEAN from the year 1983 to 2012. The analysis is conducted using advanced panel estimation approaches and found no causality in the short run while in the long-run, the results indicate that there are bidirectional relationship among variables. This study provides supplementary evidences of relationship between electricity consumption and economic growth in ASEAN.

Since the reform and opening-up, under the new economic situation and policy, the rapid growth of power demand in Beijing is threatening the sustainable development of China’s economy and environment. To recognize the driving factors of electricity consumption growth and offer policy implications, based on the data of electricity consumption, the Gross Domestic Product (GDP) and the resident population in Beijing from 1990 to 2021, this research used the Kaya-equation and logarithmic mean divisia index (LMDI) model to decompose the growth of power demand in Beijing into the quantitative contribution of each driving factor from the perspective of industrial electricity consumption and residential electricity consumption. The results of the decomposition analysis show that, as far as industrial electricity consumption is concerned, the contribution rates of economic growth, electricity consumption intensity and output value structure to industrial electricity growth are 234.26%, −109.01% and −25.25%, respectively, which shows that economic growth is the primary driving force promoting the growth of industrial electricity demand. Power consumption intensity is the main reason for restraining the growth of industrial power demand, the growth rate is sliding and the contribution of the industrial structure is relatively small; as far as residential power consumption is concerned, the contribution rates of per capita power consumption and population size to residential power growth are 68.13% and 31.87%, respectively, which indicates that per capita power consumption is the main factor promoting the growth of residential power demand, followed by the total population. The study results show that the consumption of electric power would increase if Beijing’s economy and urbanization keep developing, and optimizing the industry structure, improving the efficiency of electric energy utilization and adopting clean power energy are the main approaches to making Beijing’s consumption of electric power decrease.

Improving energy use in a commercial building has become the subject of great importance in organizations worldwide. Improving energy usage refers to the efforts to reduce energy consumption. Reducing energy consumption in commercial buildings can be accomplished through continuous supervision using appropriate managerial techniques. Commercial companies are required to use energy more efficiently and participate in energy improvement. This study seeks to improve electrical energy consumption in commercial buildings by Analysing the electrical data consumption and identifying the factors that contribute to high consumption using Six Sigma DMAIC (Define-Measure- Analyse-Improve-Control) problem solving methodology. A case study was used to validate the DMAIC framework. Two years of electrical consumption data of a case study done from January 2018 to December 2019 was collected and analysed. The study revealed an average increment in energy consumption of 3.9 %. The outcomes using statistical Pareto chart showed that the boiler is the highest significant energy user in the building with 38.3% due; followed by the kitchen with 24.2 %, followed by DB A and lifts with 20,1 % and the rest with 17.37 %. After the campaign of DMAIC, there was a reduction of 6 % in boiler consumption which was 2.3 % reduction of total consumption of the month for the building. Therefore, the study successfully demonstrates how Six Sigma DMAIC methodology can be applied to improve electrical consumption in a commercial building and reduce its related costs.

Electricity grids around the world are undergoing a fundamental transformation, thanks to the modernization of electricity distribution systems including smart meter deployment and applications. Data generated by smart meters provides a wealth of information that can help better understand and optimize the operation of electricity networks. This paper proposes a novel Hybrid Load Profile Clustering (HLPC) algorithm to identify patterns of electricity consumption of users from large volumes of residential electricity consumption interval data. The HLPC algorithm is in particular advantageous in detecting `spike' patterns of consumption among other various residential consumption patterns. Experimental results about the HLPC algorithm are presented using interval data from 600 smart meters in Victoria, Australia, to demonstrate that the proposed methodology outperforms standard clustering algorithms.

Reliable energy analysis of buildings relies heavily on high-quality data leading to proper indicators. Previous studies have highlighted the importance of data quality in analyzing energy usage in residential and non-residential buildings in order to transform declarations to actions, optimise energy efficiency policies and monitor progress and failures in countries. Collected data must adhere to national and international standards for energy performance in buildings. This study aims to provide practical guidelines for effectively collecting and preparing data suitable for evaluating energy performance in Egyptian higher-education (HE) buildings. The guidelines are developed based on a comprehensive case study, considering data availability in typical educational facilities. Architectural and civil engineering drawings, construction specifications, and occupancy details are accessible. However, actual monthly electrical and natural gas consumption data for individual buildings are lacking. To address this, the study proposes the creation of detailed datasheets for each building, encompassing all energy sources and their electrical and power specifications, such as equipment, machinery, and HVAC systems. These datasheets were utilized to calculate energy consumption and energy usage indicators (EUI). The findings demonstrate that the datasheets enable adequate assessment of energy usage in various spaces within educational buildings, including staff rooms, lecture halls, and laboratories. This facilitates the identification of areas in need of targeted energy efficiency improvements. Notably, the study reveals that electricity consumption in the Faculty of Engineering building is significantly influenced by PCs, laboratories, lighting, and air conditioning.

Modeling and disaggregating hourly electricity consumption in Norwegian dwellings based on smart meter data

Exploring the transformation and upgrading of China’s economy using electricity consumption data: A VAR–VEC based model