From Crisis to Control: NannyML's Role in Accurate Energy Demand Forecasting

Do not index

Canonical URL

Introduction

Energy demand forecasting models assist in providing stable and reliable energy supplies. This ensures that power plants generate the correct amount of electricity to meet consumer needs, optimizing resource use and minimizing costs [1]. However, forecasting models’ performance can degrade over time, becoming less accurate due to changes in consumption patterns, technological advancements, or economic shifts. When these models fail, the consequences can be severe, from power shortages to skyrocketing costs [2].

This is where machine learning (ML) model monitoring becomes crucial. Monitoring for energy demand forecasts can help detect when the model’s performance drops, allowing for timely interventions to prevent downstream consequences. NannyML is a great solution for continuous model monitoring, which assists in alerting data scientists only when there has been a drop in the performance of energy demand models.

This blog will discuss the importance of energy demand forecast monitoring and examine some of the biggest challenges in continually predicting reliable energy demand. Additionally, this blog will demonstrate how NannyML can help overcome these issues to ensure a swifter restoration of model performance.

Why Energy Demand Forecasts Matter: Key Users and Their Needs

Many stakeholders rely on energy demand forecasting to drive operational efficiency and strategic planning. The table below details how these forecasts are applied across users [3,4,5]. With different core needs and scales of impact - it is essential that energy demand forecasts remain accurate.

Table depicting the key users and an example use case for their energy demand forecasting needs.

Previous Failure in Energy Demand Forecasting Models

In the unprecedented heatwave in August 2020, California experienced rolling blackouts affecting millions of residents. The extreme temperatures caused a surge in electricity demand, well beyond the predicted energy demand level, pushing the state's power grid to its limits. The California Independent System Operator (Cal ISO) declared a Stage 3 emergency when power reserves fell below critical levels, resulting in load interruptions for 4 million people. Additionally, state authorities implemented emergency measures, including using temporary power generators, and urged residents to be conservative in their energy use [6].

The blackouts highlighted California's ongoing issues managing its electric grid during extreme weather conditions. As climate change-induced heat waves become more prevalent, covariate shift in weather data to more extreme temperatures become more of a threat [7]. Subsequently, monitoring energy demand models’ performance is necessary to ensure continuously accurate predictions and allow earlier disaster mitigation strategies.

Massive climate change-induced temperature peak in 2020 [6].

NannyML Solutions to Common Challenges in Energy Demand Forecasting Monitoring

Now that we have seen how covariate shifts in extreme temperatures can impact our energy demand forecasting models - let’s unpack additional problem areas and possible solutions to ensure our models continue providing representative energy demand predictions.

Delayed Energy Demand Target Issues

Accessing energy demand ground truth values can take between a few weeks to a few months based on the forecasts' long time horizons and the collection of energy demand targets from multiple data sources [8,9]. This means that underperforming models left unmonitored or indiscriminately retrained risk impairing their underlying business cases. Additionally, waiting to conduct a deep dive into model performance only after targets become available can result in missed opportunities to address performance issues proactively.

NannyML Approaches for Delayed Energy Demand

NannyML is uniquely positioned to address these multifaceted challenges in energy demand forecasting. Since NannyML enables performance estimation without target access, proactive interventions for many performance drop causes can be taken before targets arrive. This capability is crucial for data quality issues and covariate shift resolution, as both root causes can be addressed before targets arrive.

Since demand forecasting models tend to be regression-based, the direct loss estimation (DLE) algorithm would be used to estimate the model performance. This algorithm works by training an additional model called the nanny model. The nanny model learns to estimate the loss or error associated with the original model's predictions.

After your demand forecasting model generates predictions, the nanny model uses the original features and the predictions from the primary model to estimate the errors. This estimation process enables you to monitor your model's performance in real-time without immediate access to the actual energy demand figures.

Once targets arrive, they can be used to determine how accurate the NannyML estimates have been and whether appropriate action was taken to improve model performance.

Direct loss estimation (DLE) estimated and realized performance alignment over levels of poor model performance.

Segment Aggregation in Energy Demand Forecasting

Another factor determining the prediction's reliability is the forecast segment size. Forecasting energy demand on a larger aggregated segment ( entire regions or countries) tends to be more accurate due to the averaging effects of large data sizes that smooth out anomalies and fluctuations. When aggregating over a large population or area, individual variations in energy consumption tend to cancel each other out, leading to more stable and predictable patterns [10].

However, forecasting on more granular segments, such as individual households or specific localities, presents significant challenges. The variability in consumption patterns is much higher at this level due to factors like individual behavior, local weather conditions, and specific appliance usage, making predictions more volatile and less reliable [11]. Historically, failures in accurately predicting energy demand at more minor scales have resulted in costly imbalances and inefficiencies in energy distribution.

Energy demand forecasting noise fluctuations across customer scales [10], where dem is energy demand forecasted, and tod is the time of day.

Addressing Segment Aggregation with NannyML

NannyML's robust segmentation solutions help maintain prediction reliability across scales and geographical locations. Segmentation allows users to group input data into different levels of granularity and then test model performance metrics. The ability enables users to identify whether a model's issues stem from a specific area or are more widespread, facilitating a more targeted intervention. The segmentation insight is also vital for preventing inappropriate model transplants, where a model trained in one location is applied to another without considering contextual differences. Subsequently, NannyML’s segmentation toolset allows for a nuanced analysis of model performance drops.

Data Quality Issues in Input Data

High-quality input data is crucial for the reliability of energy-demand ML models. Missing, incorrect, outdated, or noisy data can confuse models, making it difficult to identify genuine trends.

Another challenge is smart meters. While smart meters have revolutionized energy demand forecasting by providing high-resolution, real-time data, they also can introduce data quality challenges such as missing data intervals. These missing intervals create gaps in the dataset and skew analysis if not adequately addressed. Smart meters can also produce erroneous data, such as zeros or spikes, due to technical glitches or meter malfunctions, distorting consumption patterns and leading to inaccurate predictions. Additionally, issues like data duplication and incorrect timestamps can complicate the analysis further [13]. Therefore, robust data validation and monitoring solutions are essential to detect and rectify these issues, ensuring that the data collected remains reliable and actionable for accurate energy demand forecasting models.

NannyML Solutions to Data Quality

Using the NannyML suite of data quality checkers to identify and fill gaps and detect and correct anomalies like unexpected spikes or zeros ensures a high standard of data quality for energy demand forecasting models. Additionally, earlier degraded data quality identification can lead to fixing the data pipeline upstream and adding additional checks for anomaly-riddled smart meter data for data quality issues. This ensures accurate and reliable energy demand forecasts, supporting better decision-making and efficient energy distribution while reducing the risk of financial losses from inaccurate predictions.

Seasonality and Covariate Shift in Key Energy Demand Inputs

Covariate shift can significantly affect energy demand forecasting models by altering the distribution of input features that the model relies on for making predictions. Covariate shift is defined as a change in the distribution of input features, , of a model, while or the relationship between input and target remains unchanged. Covariate shift detection can be further broken down into univariate and multivariate drift. Univariate shifts occur when only a single input feature experiences a distribution change, while multivariate shift refers to a change in the joint distributions of some or all features.

In the context of energy demand forecasting, extreme weather can cause a covariate shift, as seen by the Californian blackouts. Additionally, factors that impact consumers' ability to afford electricity, such as economic recessions and heightened energy prices due to changes in energy regulation, can also affect the distribution of these key inputs to energy demand models [14].

Covariate Shift Solved

NannyML has both univariate and multivariate drift detectors, enabling data scientists to understand how drift affects their energy demand forecasting models. This could be as basic as looking at the changing Jensen-Shannon distances between the selected reference and monitoring datasets in the univariate case. In the multivariate case, data reconstruction with PCA enables data scientists to quickly determine the scale of the joint feature drift based on the size of the reconstruction error from the PCA components, as explained in the blog Don’t Drift Away with Your Data: Monitoring Data Drift from Setup to Cloud.

💡

Datasets: Reference and Monitored

NannyML uses two datasets for its algorithms: a reference and a monitored dataset.

The reference dataset has observations from when the model experienced acceptable performance levels. If the model has only recently been put into production, the reference dataset could be the test dataset. Targets are usually available here.

The monitored dataset is a data subset whereby the observations are for an analysis period you want the NannyML algorithms to evaluate.

Data Chunk

A data chunk is a NannyML term for a data sample. All NannyML algorithms work on the data chunk level.

Depiction of an alerted covariate shift in the model enables a swift resolution before targets are available.

Seasonality is another very important factor in energy demand forecasts, with winter and summer energy demands having vastly different distributions [12]—especially when taking into account the differences in consumer habits for various geographical locations (i.e., for the same summer temperature, there may be an increase in demand in one suburb over another due to air conditioning culture in the neighborhood subpopulation) [11]. While seasonality in and of itself is not a covariate shift, as the historical cyclical pattern can be predicted - how segments and datasets are constructed for the NannyML tools needs to ensure that the seasonality is reflected to prevent false alerts for covariate shift. Chunks that are too small will not capture the effects of seasonality. Additionally, reference and monitoring datasets must be chosen to capture seasonality.

Demonstration of the seasonal variation of energy demand from Myst AI [9].

Altered Consumer Habits and Concept Drift Possibility

Concept drift refers to the phenomenon where the input distribution remains unchanged, but the conditional distribution of the target given the original inputs changes. Mathematically, this is represented as a change in , where is the input features and is the target, while , the input distribution remains constant.

Concept drift in energy demand forecasting models could stem from structural changes in the energy sector. These changes include new energy production methods, shifts in consumer behavior (such as efforts to reduce energy usage), and the introduction of new consumer products. For example, electric vehicles require an increased electricity supply for charging, which introduces novel energy consumption patterns not present in historical data [14], while more energy-efficient products lower household energy demand for the same use time. These factors disrupt existing demand patterns, leading to less accurate forecasts.

Handling Concept Drift, the NannyML Way

NannyML enables the detection and resolution of concept drift in energy demand forecasting models through the Reverse Concept Drift (RCD) algorithm. This algorithm allows concept drift to act as a model retraining trigger, as discussed in the blog post Using Concept Drift as a Model Retraining Trigger. This proactive retraining approach helps prevent model performance degradation and ensures that energy demand forecasts remain reliable and responsive to ongoing changes in the energy sector.

Missing out on performance improvement due to a concept drift.

Even if no observable drops in realized performance exist, running concept drift detection is still a good idea. This is because estimated performance might improve due to data drifting to regions further away from the class boundary, making predicting easier. However, this increase in estimated performance is unmatched and lost in the realized performance, indicating that concept drift is likely occurring. Concept drift detection can subsequently reveal ways of accessing these performance gains that are not immediately apparent, allowing for timely retraining. Wojtek Kuberski, one of NannyML’s co-founders, further explores this concept in his webinar Root Cause Analysis for ML Model Failure.

Conclusion

Monitoring energy demand forecasting models is crucial for maintaining their accuracy and reliability. These predictions are essential for optimizing energy resources, reducing costs, and ensuring power grid stability. However, these models' performance can degrade as external factors evolve, leading to inefficiencies and financial losses. NannyML offers a comprehensive solution with advanced tools for monitoring energy demand ML models in production. This ensures that forecasting models remain accurate and relevant over time.

Explore NannyML's toolset to safeguard your predictive models and maximize the efficiency of your energy management strategies. Sign up for your free trial of the NannyML cloud today or organize a call with a founder to learn how to apply NannyML to your industry.

References

Toggle for the comprehensive reference list

[1] L. Suganthi and A. A. Samuel, ‘Energy models for demand forecasting—A review’, Renew. Sustain. Energy Rev., vol. 16, no. 2, pp. 1223–1240, Feb. 2012, doi: 10.1016/j.rser.2011.08.014.

[2] A. Aziz, D. Mahmood, M. S. Qureshi, M. B. Qureshi, and K. Kim, ‘AI-based peak power demand forecasting model focusing on economic and climate features’, Front. Energy Res., vol. 12, p. 1328891, Jul. 2024, doi: 10.3389/fenrg.2024.1328891.

[3] ‘Balancing Act: Understanding Energy Grid Demand & Supply Management’, Fusion for Business. Accessed: Aug. 16, 2024. [Online]. Available: https://fusionforbusiness.co.uk/balancing-act-understanding-energy-grid-demand-supply-management/

[4] ‘Balancing the system’, Energy UK. Accessed: Aug. 16, 2024. [Online]. Available: https://www.energy-uk.org.uk/insights/balancing-the-system/

[5] J. P. Carvallo, P. H. Larsen, A. H. Sanstad, and C. A. Goldman, ‘Long term load forecasting accuracy in electric utility integrated resource planning’, Energy Policy, vol. 119, pp. 410–422, Aug. 2018, doi: 10.1016/j.enpol.2018.04.060.

[6] CAISO. Final Root Cause Analysis: Mid-August Extreme Heat Wave (Jan. 2021), http://www.caiso.com/Documents/Final-Root-Cause-Analysis-Mid-August-2020-Extreme-Heat-Wave.pdf.

[7] S. Wilburn, ‘Navigating the Heat: How Extreme Weather is Impacting Our Electric Grid’, Verde Watts. Accessed: Aug. 16, 2024. [Online]. Available: https://www.verdewatts.com/extreme-weather-is-impacting-our-electric-grid/

[8] M. Gaur, S. Makonin, I. V. Bajic, and A. Majumdar, ‘Performance Evaluation of Techniques for Identifying Abnormal Energy Consumption in Buildings’, IEEE Access, vol. 7, pp. 62721–62733, 2019, doi: 10.1109/ACCESS.2019.2915641.

[9] E. Boyle, Work on Climate. PAW Climate 2022 - Myst AI: How to build accurate electricity demand forecasts, (Jul. 31, 2022). Accessed: Aug. 16, 2024. [Youtube Video]. Available: https://www.youtube.com/watch?v=Kw5Qhqs9VcE

[10] M. Fasiolo, University Of Bristol. Forecasting electricity demand, (07 2021). Accessed: Aug. 12, 2024. [YouTube Video]. Available: https://www.youtube.com/watch?v=4Z0vpt7u7MM

[11] K. Gajowniczek and T. Ząbkowski, ‘Electricity forecasting on the individual household level enhanced based on activity patterns’, PLOS ONE, vol. 12, no. 4, p. e0174098, Apr. 2017, doi: 10.1371/journal.pone.0174098.

[12] D. Cawthorne, A. R. De Queiroz, H. Eshraghi, A. Sankarasubramanian, and J. F. DeCarolis, ‘The Role of Temperature Variability on Seasonal Electricity Demand in the Southern US’, Front. Sustain. Cities, vol. 3, p. 644789, Jun. 2021, doi: 10.3389/frsc.2021.644789.

[13] J. Shishido, ‘Smart Meter Data Quality Insights’, EnerNOC Utility Solutions, 2012. Accessed: Aug. 16, 2024. [Online]. Available: https://www.aceee.org/files/proceedings/2012/data/papers/0193-000375.pdf

[14] L. Cozzi et al., ‘The energy world is set to change significantly by 2030, based on today’s policy settings alone’, Directorate of Sustainability, Technology and Outlooks and the International Energy Agency, Oct. 2023. Accessed: Aug. 16, 2024. [Online]. Available: https://www.iea.org/news/the-energy-world-is-set-to-change-significantly-by-2030-based-on-today-s-policy-settings-alone