Keep your Model Performance Breezy: Wind Turbine Energy Model Monitoring

Do not index

Canonical URL

Introduction

Europe’s energy future is far from secure. Geopolitical tensions, the need to import 60% of its energy [1], and unstable fossil fuel markets make Europe’s autonomous and robust transition to renewable energy urgent.

Yet the rollout of renewable energy has been challenging, with availability, especially solar and wind resources, often being irregular in Europe [2]. Additionally, it can be complex to integrate this renewably produced electricity into the existing grid infrastructure [1].

Machine learning (ML) models have been used to help alleviate some of these challenges through optimized forecasting, such as solar energy generation prediction models [3] and dynamic performance models of renewables for better grid stability [4].

Yet what happens if the performance of these renewable energy forecasting models drops undetected? Incorrect renewable energy generation forecasts can lead to the grid receiving inadequate power, which could lead to blackouts if capacity cannot be rerouted to other electricity-generating means. On the flip side, an oversupply of unforeseen energy can cause significant damage to the grid infrastructure. Additionally, inaccurate renewable energy generation forecasts can result in wasting renewable energy, which in turn can increase costs for energy utility companies. Holistically, this could result in decreased public confidence in greener energy generation and the slowing down of the transition toward cleaner energy [5].

So, how can we ensure that our ML renewable energy models for energy generation continue to provide accurate predictions?

In this blog, we will discuss how NannyML’s advanced ML monitoring toolset can be used for ML model monitoring within the renewable energy space. Specifically, we will examine monitoring a model predicting energy output from a wind turbine farm and common scenarios that could result in a sub-performing model. This analysis will conclude with a mini-tutorial depicting how NannyML could be used for wind turbine energy generation (WTEG) models.

Wind Turbine Energy Generation Models (WTEG)

WTEG models are important tools for understanding the amount of wind energy to be supplied to the grid. This helps grid operators and transmission operators ensure a continuous electricity supply regardless of weather conditions. Additionally, WTEG helps wind farms with electricity pricing by balancing wind energy supply with consumer demands [6,7]. Yet these models have many feature inputs that are subject to changing conditions, which can cause a model’s performance to degrade.

Real-World Impact of Failing Energy Generation Model

The Texas power grid is a deregulated system in which different companies generate energy to feed into the grid without a central parent company owning the power plants, transmission lines, or distribution networks. The majority of Texas’s power comes from natural gas. Yet wind energy is the second-largest energy source, with just under a quarter of Texas’s energy needs met by wind [8].

Energy sources feeding the Texas Grid during 2021 power grid failure. Image by Texas Tribute [8].

In the extreme Texan winter of February 2021, a severe winter storm caused massive electricity and grid deficits. Over 4.5 million homes were without power, resulting in over 100 deaths and over $130 billion in property damage in Texas [9].

While there were multiple factors contributing to the blackouts experienced, forecasting model failure provided a significant contribution to the grid failure. This was due to:

Wind turbine energy generation models failing to account for the storm’s severity or weather data features’ shifts, thus incorrectly predicting output power levels [10].

Energy demand forecasting for consumers being too low, with the Electric Reliability Council of Texas’s (ERCOT) model underestimating demand by 14% (9600MW) [10]. This was once more likely the result of changing input features such as heating device use and temperature data shifts.

While it is impossible to know how much of a difference model monitoring would have made to this disaster, having continuously reliable predictions independent of weather conditions in the future could assist in faster disaster response and mitigation. When it comes to essential services, like electricity generation, we simply cannot afford to have inaccurate predictions.

What are the common failure modes of wind turbine energy generation models?

Wind turbines are highly sophisticated structures that must operate under various extreme weather conditions, from sudden, unforeseen gusts of wind to gale force intensity [11].

Wind turbine power generation and transport illustration. From Wikimedia Commons

Vibrational effects, changes in the mechanical efficiencies, and maintenance of the wind turbine’s components affect the power prediction from WTEG models. Yet, these considerations are beyond the scope of this article. Only environmental changes of commonly measured and frequently used input features for these WTEG models are considered.

It should additionally be noted that wind turbines are physics-driven systems. This means that WTEG models are constrained to behave according to the governing physics-based relationships. These models, therefore, tend to be robust against concept drift performance degradations. However, in cases of extreme humidity or temperature, the turbines' sensors and electronics could measure these WTEG feature inputs differently than previously [12], which could lead to a measurement system malfunction-induced concept drift.

Yet, covariate shifts and data quality-based performance drops can be detected, linked back to their root cause, and subsequently resolved without the need to wait for actual wind power targets. These performance degradation causes will subsequently be discussed further.

Wind Power Contributors

Wind turbines generate power by converting the kinetic energy in wind to electricity via a generator connected to the turbine blades. Standard turbines require a wind speed of around 3.5m/s to start moving, with optimal power generated at speeds of 11m/s. Wind speeds above 25m/s will typically trigger the wind turbine's control system to apply a braking force to slow the turbine down and prevent excessive speeds, which can damage the structural integrity of the turbine and exceed voltage thresholds [13].

Output power vs speed of a typical wind turbine. Image by author, with inspiration from [14].

The power produced by a wind turbine is directly proportional to the cube of the wind speed (), so accurately monitoring wind speed is essential for accurate WTEG model predictions [15]. Additionally, the power produced by a wind turbine is directly proportional to air density () [16].

Power equation of a wind turbine. Image by author.

Accuracy and Missing Values of Monitored Components

Wind turbine nacelles typically have a sensor called an anemometer that monitors the wind speed in real-time. Since even a 5% error in wind speed measurements will lead to a 15% error in predicted power, the cumulative effects of inaccurate wind speed measurements have a massive impact on a WTEG model’s power prediction [17].

For air density monitoring - temperature, pressure, relative humidity, and dew point temperature sensors are typically installed on a metrological mast near the wind turbine site [18,19]. Cooler temperatures, higher pressures, and lower humidities or drier air would all contribute to a larger air density and, subsequently, a higher wind turbine power output.

Any malfunctioning, defects, aging, or improper maintenance of any of the sensors could result in missing or nonsensical values [20,21]. Sensors can also experience calibration drift due to possible software updates [21] or extreme environmental conditions outside of their operational range. Bugs in the sensor software or data corruption during the sensor value storage process [22] are also possible.

Covariate Shift of Key Features

Wind speeds can be predicted ahead of time for use in WTEG forecasting models. However, phase (timing) errors can arise. These errors occur when wind speed changes are predicted in models after a steep wind change has manifested [23]. If the original training data didn’t include instances of these phase changes, a covariate shift may result.

Wind gusts can additionally affect WTEG model predictions. Gusts are short-term increases in wind speed in the atmosphere that increase the fatigue load on turbines, which affects generated power and causes fluctuations in grid voltage [24]. These non-linear effects once more make covariate shifts more likely, making the wind speed no longer capable of accurately representing power output.

Climate change is one factor that has significantly shifted the expected feature distributions in terms of air density-related features. Over recent years, heat waves have become more prevalent, with some of the hottest global temperatures recorded [25]. Additionally, humidity has increased globally due to warmer air capable of holding more water vapor [26]. Both these factors result in less dense air, yet historical datasets may not capture this spread of lower air density values.

WTEG Model Monitoring

We will demonstrate NannyML’s complete workflow to unpack how the toolset can benefit WTEG model monitoring, even when actual power targets are delayed.

For this example, we will use the Wind Power Generation Data from Kaggle from Location 1. This dataset covers most of the discussed features, such as temperature, relative humidity, dewpoint temperature and wind guts, speed and direction, and various heights.

Following standard preprocessing of the input features, a synthetic issue is added to the dataset. Let’s see if the NannyML toolset can help us identify this issue!

Depiction of the dataset features. Image by Author.

🚧

NannyML Terminology

A data chunk refers to a sample of data. All NannyML algorithms work at the data chunk level, typically derived from a specific time period.

NannyML additionally uses two distinct datasets: the reference dataset and the monitored dataset.

The reference dataset is collated from a period of acceptable model performance. Depending on how long the model has been in production, this dataset could be the test dataset or a benchmark dataset.

The monitored dataset consists of a subset of data with observations from a specific analysis period that you want NannyML to assess.

Performance Estimations

To predict wind turbine power forecasts, a light gradient-boosting machine (LGMB) regression model is selected and trained. For regression models, NannyML’s direct loss estimation (DLE) performance monitor is used for performance estimation. The mean absolute error (MAE) metric is selected for performance degradation alerts because it is interpretable and aligns with the industry standard WTEG model evaluation metrics [27,28,29]. The alert threshold is set to trigger if the MAE falls two standard deviations above or below the mean. These dynamic thresholds are ideal where data presents seasonal variation, as in the case of temperature, humidity, and other features in the WTEG model [30].

DLE performance estimation graph produced by the WTEG model. Image by author.

Following the DLE analysis with a 2-month-sized chunk (1300 observations), the last 3 chunks from February 2022 to June 2022 present performance alerts. Despite the intuitive understanding that a lower MAE indicates a better-performing model, triggering an alert for a sudden drop in MAE and proceeding with the investigation is a good strategy.

📖

Examples for Justifying Investigating a Lower MAE 1. A sudden MAE drop below historical levels could indicate a covariate shift to a region that makes it easier to discriminate between the different power levels, resulting in a genuine improvement in performance. Subsequent issue resolution steps would not be required here. 2. An unusually low and unexpected MAE may result from a data quality issue in the production environment, bugs in the model implementation or data pipeline leading to non-representative predictions. 3. For NannyML’s monitoring, a drastic shift in MAE away from historic levels could also indicate an inappropriately chosen monitoring dataset. If the monitoring process is only capturing a subset of predictions, an abnormally low MAE could imply that the sampling method is biased towards easy-to-predict cases.

Root Cause Analysis

Following an extensive root cause analysis, the data reconstruction with PCA reveals that, indeed, a multivariate data drift has occurred over the region where the performance alert occurs.

Data Reconstruction with PCA for multivariate drift detection with relevant alerts. Image by author.

Investigating further with NannyML’s univariate drift detection methods reveals that the windspeed at 10m and 100m had shifted over the period when the performance alert was noted. Both these shifts seem to indicate that the data distribution shifted to an easier region for the model to make predictions.

Univariate data drift detected in windspeed_100m and windspeed_10m features. Image by author. — Univariate data drift detected in `windspeed_100m` and `windspeed_10m` features. Image by author.

To verify that a shift to an easier prediction region occurred, we plot the spread of the MAE vs wind speed and histograms capturing the spread of the wind speed and power in both the reference data set (where DLE learns how to estimate performance degradation) and in the last 3 chunks of the monitoring data set where the performance degradation occurs.

The wind speed in the last 3 monitoring chunks indicates a shift to the most extreme higher and lower values (when compared to the reference set). Additionally, the distribution in MAE for these values in the monitoring set is biased towards lower MAE values with fewer orange and red points than the reference MAE plot. This furthers our confidence that the wind data has shifted to an easier prediction region. Subsequently, no issue resolution step is required.

Reference dataset depiction of MAE spread, wind speed distribution, and power distribution. Image by author.

Last 3 chunks of the monitoring dataset (where alerts occurred) with the MAE spread depiction, wind speed distribution, and power distribution. Image by author.

Target Availability

Once the actual power targets become available, we are able to plot the estimated MAE against the realized MAE. This allows us to understand how the DLE estimated performance compares to reality and whether our issue resolution was necessary and appropriate.

In this case, DLE correctly predicted the performance change, as realized MAE dropped below historically recorded values. Using the NannyML toolset enabled us to understand this change and feel confident not proceeding with unnecessary issue resolution steps.

Estimated vs realized performance on MAE metric. Image by author.

📖

The existence of a covariate shift does not always correlate with a drop in performance. This is why it is important to monitor the estimated performance first to understand if intervention is required.

Did NannyML detect the synthetic issue correctly?

Yes! A covariate shift was indeed induced by redistributing the observations according to the windspeed_100m data value. The highest and lowest wind speeds were moved into the monitored dataset.

This biased redistribution is one of the preferred methods for synthetically producing a covariate shift. Other methods, such as changing variable values or adding new rows, have a higher likelihood of also injecting concept drift into the dataset.

A depiction of the change in the dataset distribution with the monitoring dataset before (left) and after (right) the injection of covariate shift. Image by author.

Conclusion

In summary, this blog emphasizes the need for monitoring machine learning models responsible for renewable energy forecasting to ensure grid stability and public confidence in green energy. While ML models are crucial for optimizing energy predictions, their potential failures can lead to significant consequences. The real-world example of the Texas power grid failure underscores the importance of reliable predictions for disaster prevention.

NannyML's advanced monitoring toolset provides a solution for maintaining the accuracy of energy generation models. By continuously monitoring model performance and identifying common failure modes, stakeholders can mitigate risks associated with inaccurate forecasts. Additionally, linking the cause of performance changes to the root of the change is important for targeted issue resolutions that are both required and suitable.

To find out more about how to use NannyML in your industry, book a call with one of the NannyML founders!

Acknowledgment

This article greatly benefited from Jessica Fischer's industry insights. Jessica and I have been friends since January 2017, when we both headed to Peru through WindAid Institute’s volunteering program to build and install wind turbines in underprivileged Peruvian villages.

Any inaccuracies in the blog post, however, are the author's alone.

Author and Jessica Fischer on site in Playa Blanca, Peru, installing a wind turbine built during the WindAid Institute program. Picture from author’s collection.

References

Toggle for the comprehensive reference list

[1] Mario, ‘Four challenges of the energy crisis for the EU’s strategic’, Eur. Parliam. Res. Serv., Apr. 2023.

[2] ‘Irregular Renewable Generation Must Be Addressed In Europe’, Power Gen Advancement. Accessed: Jul. 14, 2024. [Online]. Available: https://www.powergenadvancement.com/news/irregular-renewable-generation-must-be-addressed-in-europe/

[3] A. Talwariya, P. Singh, J. H. Jobanputra, and M. L. Kolhe, ‘Machine learning based renewable energy generation and energy consumption forecasting’, Energy Sources Part Recovery Util. Environ. Eff., vol. 45, no. 2, pp. 3266–3278, Jun. 2023, doi: 10.1080/15567036.2023.2194257.

[4] R. Khan, ‘Integrating Renewables and Machine Learning for Improved Grid Stability’, University of Houston. Accessed: Jul. 15, 2024. [Online]. Available: https://uh.edu/news-events/stories/2024/march/03122024-li-improving-grid-stability.php

[5] A. A. Berraies, A. Tzanetos, and M. Blondin, ‘Machine Learning to Facilitate the Integration of Renewable Energies into the Grid’, in Handbook of Smart Energy Systems, M. Fathi, E. Zio, and P. M. Pardalos, Eds., Cham: Springer International Publishing, 2022, pp. 1–23. doi: 10.1007/978-3-030-72322-4_65-1.

[6] F. Scheurich et al., ‘Time-Series Based Surrogate Model For Wind Farm Performance Prediction’, J. Phys. Conf. Ser., vol. 2767, no. 9, p. 092001, Jun. 2024, doi: 10.1088/1742-6596/2767/9/092001.

[7] V. Sohoni, S. C. Gupta, and R. K. Nema, ‘A Critical Review on Wind Turbine Power Curve Modelling Techniques and Their Applications in Wind Based Energy Systems’, J. Energy, vol. 2016, pp. 1–18, 2016, doi: 10.1155/2016/8519785.

[8] Cai, E. Douglas, and M. Ferman, ‘How Texas’ power grid failed in 2021 — and who’s responsible for preventing a repeat’, Texas Tribune. Accessed: Jul. 17, 2024. [Online]. Available: https://www.texastribune.org/2022/02/15/texas-power-grid-winter-storm-2021/

[9] J. W. Busby et al., ‘Cascading risks: Understanding the 2021 winter blackout in Texas’, Energy Res. Soc. Sci., vol. 77, p. 102106, Jul. 2021, doi: 10.1016/j.erss.2021.102106.

[10] University of Texas at Austin, ‘The Timeline and Events of the February 2021 Texas Electric Grid Blackouts’, Jul. 2021.

[11] ‘Edgewise Vibrations’, Wind Power Lab. Accessed: Jul. 16, 2024. [Online]. Available: https://windpowerlab.com/edgewise-vibrations/#:~:text=Wind turbines (WTs) are complex,lead to blade-tower impacts.

[12] ‘The Impact of Temperature and Humidity on Electronics’. Accessed: Aug. 06, 2024. [Online]. Available: https://www.rikasensor.com/the-impact-of-temperature-and-humidity-on-electronics.html

[13] ‘What is a Wind Turbine?’, Acciona. Accessed: Jul. 16, 2024. [Online]. Available: https://www.acciona.com/renewable-energy/wind-energy/wind-turbines/?_adin=01678711439

[14] S. Cole, ‘Wind Turbine Power Curve’, The Roundup. Accessed: Jul. 29, 2024. [Online]. Available: https://theroundup.org/wind-turbine-power-curve/

[15] Masterclass by Michael Courtney on Wind Measurements (part I), (Jun. 15, 2023). Accessed: Jul. 16, 2024. [Online Video]. Available: https://www.youtube.com/watch?v=3xzH5-6p2Qw&list=PL1lvXOhY32ajrmRvEnKbZ3zbrQU8wbb76&index=9

[16] C. Jung and D. Schindler, ‘The role of air density in wind energy assessment – A case study from Germany’, Energy, vol. 171, pp. 385–392, Mar. 2019, doi: 10.1016/j.energy.2019.01.041.

[17] Astolfi, F. De Caro, M. Pasetti, F. Castellani, A. Vaccaro, and A. Flammini, ‘Assessing the effects of anemometer systematic errors on wind generators performance by data-driven techniques’, Sustain. Energy Grids Netw., vol. 39, p. 101417, Sep. 2024, doi: 10.1016/j.segan.2024.101417.

[18] E. Weyer and L. Sloka, ‘Wind Energy and Air Density Measurement’, WindTech International. Accessed: Jul. 23, 2024. [Online]. Available: https://www.windtech-international.com/editorial-features/wind-energy-and-air-density-measurement

[19] M. L. Thøgersen, ‘WindPRO / ENERGY: Modelling of the Variation of Air Density with Altitude through Pressure, Humidity and Temperature’, May 2005. Accessed: Jul. 23, 2024. [Online]. Available: https://www.emd-international.com/files/windpro/WindPRO_AirDensity.pdf

[20] P. Svickova, ‘Wind Measurement Error and Financial Impact’, Barani. Accessed: Jul. 21, 2024. [Online]. Available: https://www.baranidesign.com/faq-articles/2020/2/20/wind-measurement-error-and-financial-impact

[21] Leahy, C. Gallagher, P. O’Donovan, and D. T. J. O’Sullivan, ‘Issues with Data Quality for Wind Turbine Condition Monitoring and Reliability Analyses’, Energies, vol. 12, no. 2, p. 201, Jan. 2019, doi: 10.3390/en12020201.

[22] ‘Wind Turbine Failures: Causes, Consequences, and Impact on Energy Output’. Accessed: Jul. 23, 2024. [Online]. Available: https://www.enlitia.com/resources-blog-post/wind-turbine-failures-causes-consequences-and-impact-on-energy-output

[23] Masterclass by Gregor Giebel - Forecasting Wind Power, (Jun. 15, 2023). Accessed: Jul. 16, 2024. [YouTube]. Available: https://www.youtube.com/watch?v=j-5xq3ZyMk0

[24] Zhou, N. Cherukuru, X. Sun, and R. Calhoun, ‘Wind Gust Detection and Impact Prediction for Wind Turbines’, Remote Sens., vol. 10, no. 4, p. 514, Mar. 2018, doi: 10.3390/rs10040514.

[25] C. Lindwall, ‘What Are the Effects of Climate Change?’, NRDC. Accessed: Jul. 23, 2024. [Online]. Available: https://www.nrdc.org/stories/what-are-effects-climate-change#weather

[26] Press Office, ‘Humidity – the second pillar of climate change’, Met Office. Accessed: Jul. 23, 2024. [Online]. Available: https://www.metoffice.gov.uk/about-us/news-and-media/media-centre/weather-and-climate-news/2020/scientists-investigate-humidity---the-second-pillar-of-climate-change

[27] A. Clifton, L. Kilcher, J. K. Lundquist, and P. Fleming, ‘Using machine learning to predict wind turbine power output’, Environ. Res. Lett., vol. 8, no. 2, p. 024009, Jun. 2013, doi: 10.1088/1748-9326/8/2/024009.

[28] W.C. Tsai, C.M. Hong, C.S. Tu, W.M. Lin, and C.H. Chen, ‘A Review of Modern Wind Power Generation Forecasting Technologies’, Sustainability, vol. 15, no. 14, p. 10757, Jul. 2023, doi: 10.3390/su151410757.

[29] S. Hanifi, X. Liu, Z. Lin, and S. Lotfian, ‘A Critical Review of Wind Power Forecasting Methods—Past, Present and Future’, Energies, vol. 13, no. 15, p. 3764, Jul. 2020, doi: 10.3390/en13153764.

[30] Weisberg et al., ‘Create a metric alert with dynamic thresholds’, Microsoft. Accessed: Aug. 06, 2024. [Online]. Available: https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-dynamic-thresholds