Completed Project  |  Azure Databricks  |  Batch Mode 2026

Globeleq Energy Intelligence
Platform

A full-stack Azure data engineering and machine learning pipeline built on 3 million rows of synthetic SCADA telemetry from 19 African power plants — powering predictive operations, revenue forecasting, and ESG reporting.

Azure Databricks Delta Lake Azure Data Factory MLflow XGBoost LightGBM Random Forest Isolation Forest PySpark Power BI Python SQL Server
3.02M
Total Rows Generated
19
African Power Plants
9
ML Models in Registry
R139M
Annual Portfolio Revenue
~92%
Operating Fleet Availability
Chapter 1

Why This Project Matters

Globeleq operates utility-scale power plants across six African nations. Each plant generates megawatts of electricity and terabytes of SCADA data. Without a unified data platform, these operational signals sit in siloed plant historians — invisible to the portfolio managers, revenue analysts, and maintenance planners who need them most.

"We are powering Africa's growth by developing and operating utility-scale power plants."

— Globeleq mission statement

The Business Problem

Operational Challenge
Fragmented SCADA Data
19 plants across 7 countries run different OEM control systems. Data lives in local historians with no cross-portfolio visibility.
Commercial Challenge
Revenue Leakage
Curtailment, settlement delays, and FX exposure erode contracted revenue. No real-time collection rate monitoring.
Maintenance Challenge
Reactive Maintenance
Forced outages cost on average 18 hours of lost generation per event. No predictive signal to dispatch maintenance proactively.
ESG Challenge
Emissions Reporting
Investors and DFIs require GHG scope 1/2 data. Manual aggregation from plant logs is error-prone and slow.

The Solution

Real-Time SCADA Intelligence

15-minute sensor telemetry ingested via ADF into Delta Lake Bronze — available for analysis within 30 minutes of generation.

📈

Predictive ML Suite

Nine models covering yield forecasting, availability classification, maintenance cost, anomaly detection, revenue projection, solar irradiance yield forecasting, and three production forecasting challengers.

🌎

ESG Automation

Automated CO₂ avoided, Scope 1 emissions, renewable generation share — updated daily, investor-ready monthly.

Chapter 2

The Portfolio

19 operating and in-construction plants spanning Northern, Western, Eastern and Southern Africa — Solar PV, Wind, Natural Gas, Heavy Fuel Oil and Geothermal technologies.

1,794 MW
Operating Capacity
485 MW
Under Construction
67%
Renewable Capacity
7 Countries
Across Africa
#Plant NameCountryTechnology CapacityOfftakerAgreementStatus
1ARC Solar PowerEgyptSolar PV66 MWpEETCPPA 25yrOperating
2Aries Solar PowerSouth AfricaSolar PV11 MWpEskomPPA 20yrOperating
3Azito PowerCôte d'IvoireNatural Gas713 MWGovt of CdIConcession 20yrOperating
4Boshof Solar PowerSouth AfricaSolar PV66 MWpEskomPPA 20yrOperating
5Cuamba Solar + BESSMozambiqueSolar PV+BESS19 MWp / 7 MWhOperating
6De Aar Solar PowerSouth AfricaSolar PV50 MWpEskomPPA 20yrOperating
7Dibamba PowerCameroonHeavy Fuel Oil88 MWENEOTolling 20yrOperating
8Droogfontein SolarSouth AfricaSolar PV50 MWpEskomPPA 20yrOperating
9Jeffreys Bay WindSouth AfricaWind138 MWOperating
10Klipheuwel Wind FarmSouth AfricaWind27 MWOperating
11Konkoonsies SolarSouth AfricaSolar PV11 MWpEskomPPA 20yrOperating
12Kribi PowerCameroonNatural Gas216 MWNational GridPPA 20yrOperating
13Malindi Solar PowerKenyaSolar PV52 MWpKenya PowerPPA 20yrOperating
14Mocuba SolarMozambiqueSolar PV41 MWpEDMPPA 25yrOperating
15SongasTanzaniaNatural Gas190 MWNational GridPPA 20yrOperating
16Soutpan SolarSouth AfricaSolar PV31 MWpEskomPPA 20yrOperating
17Winnergy SolarEgyptSolar PV25 MWpEETCPPA 25yrOperating
18Central Térmica TemaneMozambiqueNatural Gas450 MWEDMTolling 25yrIn Construction
19Menengai GeothermalKenyaGeothermal35 MWKPLCIn Construction
Chapter 3

Platform Architecture

A classic Azure Medallion Architecture: raw data lands in Bronze Delta tables via ADF, is cleansed and enriched in Silver, and aggregated into executive-ready Gold KPIs and an ML feature store.

MEDALLION ARCHITECTURE — AZURE DATABRICKS
Source
SCADA / Plant Historians
15-min telemetry · OEM protocols · ModBus/DNP3
Source
SQL Server DW
Dim/Fact tables · GlobeleqEnergyDW · 17 tables
Source
External Feeds
FX rates · Weather API · Regulator filings
↓ Azure Data Factory — Copy Activities + ForEach + WebActivity ↓
Bronze Layer
Raw Delta Tables
schema enforcement · audit columns · no transforms · 3M+ rows
Storage
ADLS Gen2
abfss://raw@
globeleqdatalake
↓ Databricks Notebook 02 — Silver Transform ↓
Silver Layer
scada_hourly
15-min → hourly agg · DQ scoring · joined to dim_plant
Silver Layer
fact_plant_ops_daily
rolling 7d/30d · lag features · cleaned types
Silver Layer
fact_outage / maint
IsForced · MTTR · overdue flags · enriched joins
↓ Databricks Notebook 03 — Gold KPIs ↓
Gold Layer
portfolio_daily_kpis
DoD delta · renewable share · emissions intensity
Gold Layer
plant_monthly_kpis
EBITDA proxy · MTTR · collection rate · ranking
Gold Layer
ml_feature_store_daily
lag features · rolling windows · outage target labels
↓ Notebook 04 — ML Training & Inference ↓
Consume
Power BI
30 DAX measures · Databricks SQL endpoint · scheduled refresh
Consume
MLflow Registry
9 models · versioned · staging → production promotion
Consume
Teams / Email
Pipeline success/fail webhook · anomaly alerts · daily digest
Chapter 4

The Data — 3.02 Million Rows

Synthetic data was generated using physically-motivated models: solar irradiance calculations based on plant latitude and day of year, Weibull wind speed distributions with AR(1) temporal correlation, and Poisson forced-outage arrival processes for thermal plants.

2,981,664
SCADA Telemetry Rows (15-min)
34,713
Daily Operations Rows
1,656
Outage Event Rows
3,878
Maintenance Work Orders

SCADA Telemetry Schema

-- 15-minute SCADA fact table (2.98M rows)
CREATE TABLE scada_telemetry_15min (
  Timestamp             TIMESTAMP,
  PlantKey              INT,
  PlantCode             VARCHAR(10),
  Technology            VARCHAR(30),
  ActivePowerMW         DOUBLE,     -- MW
  ReactivePowerMVAR     DOUBLE,
  GridFrequencyHz       DOUBLE,     -- ~50 Hz
  AmbientTemperatureC   DOUBLE,
  SolarIrradianceWm2    DOUBLE,     -- solar only
  WindSpeedMs           DOUBLE,     -- wind only
  InverterEfficiencyPct DOUBLE,
  TransformerTempC      DOUBLE,
  CurtailmentFactor     DOUBLE,
  CumulativeEnergyMWh   DOUBLE,
  AlarmCode             VARCHAR(20), -- NORMAL/OUTAGE/CURTAILMENT
  DataQualityFlag       VARCHAR(10)  -- GOOD/SUSPECT/BAD
)
PARTITIONED BY (PlantKey)

Data Quality Rules

Schema Enforcement

Delta Lake enforces column types at Bronze ingestion. Reject-on-schema-mismatch prevents silent null propagation.

SUSPECT Tagging

15-min readings during forced outages tagged DataQualityFlag='SUSPECT' — excluded from KPI denominators.

Deduplication

Silver layer drops exact duplicates on (Timestamp, PlantKey) using Delta dropDuplicates() before merge.

Simulated Generation Profile — Solar vs Wind vs Gas (hourly avg, 2024)
Chapter 5

ADF Pipeline — Daily Automation

The Azure Data Factory pipeline runs at 02:00 SAST every night, copying new data from SQL Server and ADLS Gen2, executing the four Databricks notebooks in sequence, conditionally retraining ML models on Saturdays, refreshing Power BI, and posting a Teams webhook.

1
Set Variables
RunId · timestamps
2
GetMetadata
Validate raw files in ADLS
3
Copy Dims
SQL → ADLS CSV
4
ForEach Facts
8 fact tables in parallel
5
Bronze NB
Databricks 01_bronze
6
Silver NB
Databricks 02_silver
7
Gold NB
Databricks 03_gold
8
ML Gate
Saturday = retrain else inference
9
PBI Refresh
Power BI REST API
10
Teams Alert
Success / failure webhook

Key Design Decisions

Parallelism
ForEach with batchCount=4
8 fact tables copy in parallel batches of 4, cutting ingestion time from ~40 min to ~12 min.
Saturday Gate
IfCondition for ML Retraining
Full model retraining runs weekly (Saturday only) using dayOfWeek(utcNow())==7 — daily runs execute inference only to save cluster cost.
Error Handling
Failure Branch to Teams
Any notebook failure triggers a red Teams card with run ID, timestamp, and direct ADF Monitor link.
Secrets
Azure Key Vault + Databricks Scope
Service principal credentials, PBI tokens, and Teams webhook URLs are resolved at runtime from Key Vault — never hard-coded.

Trigger Configuration

// tr_daily_0200_sast
{
  "type": "ScheduleTrigger",
  "recurrence": {
    "frequency": "Day",
    "interval": 1,
    "startTime": "2025-01-01T00:00:00Z",
    "schedule": {
      "hours": [0],   // 00:00 UTC = 02:00 SAST
      "minutes": [0]
    }
  }
}

Linked Services

NameTypeAuth
ls_adls_globeleqADLS Gen2Service Principal + KV
ls_databricks_globeleqDatabricksMSI Token
ls_sql_globeleqAzure SQLManaged Identity
ls_powerbiPower BI RESTOAuth2 Bearer
Chapter 6

Machine Learning Suite

Nine models address distinct business questions across the plant lifecycle — from next-day solar yield prediction to monthly revenue forecasting. Six operational models (XGBoost, LightGBM, Random Forest, Isolation Forest, LightGBM Regressor, Solar Yield LightGBM) are tracked in MLflow alongside three production-ready forecasting challengers, all versioned in the Model Registry and refreshed weekly.

# Model Algorithm Target Key Metrics Business Use
1 Energy Yield Forecaster XGBoost Regressor NetGenerationMWh (t+1 day) R²=0.998 · MAE=62 MWh · RMSE=151 MWh Grid scheduling & energy trading
2 Plant Availability Tier Classifier LightGBM Classifier ≥90% availability next month AUC=0.85 · AP=0.97 · prospective Predictive maintenance dispatch
3 Maintenance Cost Estimator Random Forest Monthly maintenance cost (ZAR) R²=0.999 · MAE=R103 · OOB=0.999 OPEX budgeting & planning
4 Curtailment Anomaly Detector Isolation Forest Anomalous curtailment months Anomaly rate=5% · Contamination=0.05 Grid congestion & inverter alerts
5 Portfolio Revenue Forecaster LightGBM Regressor Portfolio revenue ZAR (t+1 month) R²=0.93 · MAE=R2.8M · MAPE=3.1% Investor & cash-flow reporting

Model 1: Energy Yield Forecaster

XGBoost regressor trained on 80/20 split of 34,713 plant-day records. Features include 7-day and 30-day rolling averages, lag-1 and lag-7 generation, plant capacity, technology encoding, and seasonality (month).

xgb_params = {
  "n_estimators"    : 500,
  "max_depth"       : 6,
  "learning_rate"   : 0.05,
  "subsample"       : 0.8,
  "colsample_bytree": 0.8,
  "min_child_weight": 5,
  "reg_alpha"       : 0.1,
  "reg_lambda"      : 1.0,
}
# Top features: Gen_Lag1 > AvailabilityPct
# > GenMWh_7d > NameplateCapacity > Month

Model 2: Plant Availability Tier Classifier

LightGBM binary classifier with class balancing (scale_pos_weight). Threshold optimised to 0.35 to maximise recall — better to dispatch a maintenance crew unnecessarily than to miss a real fault.

# Class imbalance: only ~8% of days have
# a forced outage in the next 7 days
lgb_params = {
  "scale_pos_weight": scale_pos,  # ~11x
  "class_weight"    : "balanced",
  "num_leaves"      : 31,
  "min_child_samples": 30,
  "early_stopping"  : 50,
}
# Optimised threshold: 0.35 (not 0.50)
# Top features: AvailabilityPct, ForcedOut_Lag7
Chapter 7

Portfolio Results (2020–2024)

Key insights derived from 5 years of synthetic operational data across the Globeleq portfolio.

Solar Fleet — 89% of Plants

11 solar plants deliver 22–24% capacity factors across Africa's high-irradiance zones. Egypt plants (ARC, Winnergy) achieve the most consistent generation with minimal seasonal swing.

🌬

Wind — Highest Capacity Factor

Jeffreys Bay (JBAY, 138 MW) achieves ~38% capacity factor — nearly double the solar fleet average. South Africa's coastal wind corridor is among the best in the continent.

Gas Baseload — Revenue Anchor

Azito (713 MW, Côte d'Ivoire) alone generates more revenue than the entire South African solar fleet combined, operating at ~78% capacity factor 24/7.

📅

Availability: ~92% Operating Fleet

Operating fleet (17 plants) averaged ~92% availability across 2020–2024. Planned annual maintenance windows account for ~5% of downtime; forced outages for the remaining ~3%.

🌿

CO₂ Avoided: 8.4M Tonnes

The renewable fleet avoided 8.4 million tonnes of CO₂ over 5 years, equivalent to removing 1.8 million cars from Africa's roads annually.

💰

Collection Rate: 94.2%

Weighted settlement collection rate held above 90% across all regions. East Africa (Kenya, Tanzania) shows the most consistent collection; some West Africa months show 85–88%.

Annual Portfolio Net Generation (GWh) — 2020 to 2024
Technology Generation Mix — 5-Year Totals (GWh)
Chapter 8

How I Built It

Technical decisions, trade-offs, and the rationale behind each engineering choice.

Why Delta Lake over Parquet?
ACID + Time Travel
SCADA backfill corrections are common — Delta's MERGE allows late-arriving data to overwrite specific plant-timestamp rows without full table rewrites. Time travel enables point-in-time comparisons for regulatory audits.
Why 15-min SCADA not hourly?
Curtailment Detection Granularity
A 15-minute curtailment event is invisible in hourly aggregations. The Isolation Forest anomaly model was tuned to detect intra-hour curtailment spikes that correlate with grid congestion signals.
Why LightGBM for outage prediction?
Speed + Native Categorical Support
LightGBM trains 3× faster than XGBoost on the same data and handles the high class imbalance (8% positive class) with built-in scale_pos_weight and early stopping — critical for weekly retraining on a shared cluster.
Why partition by PlantKey?
Query Pruning at Scale
Power BI reports almost always filter by plant. Partitioning the 3M-row SCADA table by PlantKey means per-plant queries touch 1/17th of the data, cutting query time from ~8s to <1s on the Databricks SQL endpoint.

Skills Demonstrated

📊 Data Engineering

Medallion architecture · Delta Lake MERGE · PySpark window functions · ADF ForEach + IfCondition · ADLS Gen2 OAuth2 mounting · OPTIMIZE + ZORDER

🤖 Machine Learning

XGBoost / LightGBM / Random Forest / Isolation Forest · MLflow experiment tracking · Model Registry · SHAP explainability · Class imbalance handling · Threshold optimisation

🌍 Domain Knowledge

IPP commercial structure (PPA/Tolling/Concession) · SCADA telemetry schemas · Capacity factor / availability calculations · GHG Scope 1 accounting · DFI-grade ESG reporting

🔧 Tooling

Python · PySpark · SQL (Spark SQL + SQL Server DDL) · openpyxl · Azure Databricks · ADF · Power BI DAX · Git · JSON pipeline definitions

Chapter 9

Analytics & ML Diagnostics

13 publication-quality charts generated directly from 3,024,807 rows of synthetic operational data. Each chart is explained across four dimensions — What it shows, Why it matters commercially, Where it fits in the platform, and How to read it correctly.

📊 Operational Analytics  ·  Charts 01–04
Operational KPI Correlation Matrix
Chart 01 Operational KPI Correlation Matrix
🔍 What

Pearson correlation heatmap of 9 daily operational KPIs across 34,713 plant-days (2020–2024). Variables include Availability %, Capacity Factor %, Gross & Net Generation, Curtailment %, Planned and Forced Downtime, CO₂ Avoided, and Scope 1 Emissions.

💡 Why it matters

Before building ML models, understanding multicollinearity prevents feature redundancy. Identifying which KPIs move together reveals genuine physical relationships vs artificial duplicates, and informs which variables to include or drop from the feature store.

📍 Where it lives

ML feature engineering (Notebook 03 Gold layer). Also informs which DAX measures should be calculated independently vs derived from others in Power BI.

🔎 How to read it

Blue = strong positive correlation (e.g., GrossGen ↔ NetGen = 0.99 — expected). Red = negative (ForcedDowntime ↔ Availability = −0.68 — outages destroy availability). Near zero = no linear relationship (Curtailment ↔ Scope1 ≈ 0.02).

💡

Key Insight — Gross Generation and Net Generation are 99% correlated — only one needed as an ML target. CO₂ Avoided and Gross Generation are 97% correlated — the CO₂ model is a linear scaler, not an independent signal.

Plant Availability % — Heatmap by Year
Chart 02 Plant Availability % — Heatmap by Year
🔍 What

17 operating plants × 5 years grid showing mean daily Availability %, sorted by 2024 performance descending. Each cell is the annual average across all operational days for that plant.

💡 Why it matters

Reveals which plants show persistent underperformance (row-level trend) vs isolated bad years (single dark cell). Enables the asset management team to target plants for O&M contract renegotiation or equipment inspection.

📍 Where it lives

Portfolio Review (Chapter 7 Results), O&M planning, and Power BI page 3 — Plant Health Dashboard. Feeds directly into the Plant Availability Tier Classifier training data.

🔎 How to read it

Green cells (>95%) = high performers — no action needed. Yellow (85–95%) = watch list. Orange/red (<80%) = intervention candidates. Read each row left-to-right to see whether a plant is improving, stable, or declining over time.

💡

Key Insight — Azito Power (Natural Gas, 713 MW) consistently delivers 95%+ availability — the portfolio's most reliable revenue anchor. Droogfontein and Klipheuwel show slight 2024 dip, signalling potential ageing equipment.

Annual Gross Generation by Technology (TWh)
Chart 03 Annual Gross Generation by Technology (TWh)
🔍 What

Stacked bar chart of annual gross generation (TWh) broken down by primary technology: Natural Gas, Solar PV, Wind, Heavy Fuel Oil, Solar PV + BESS. Covers all 17 operating plants for 2020–2024.

💡 Why it matters

Shows portfolio energy composition and year-on-year stability. Helps identify whether the portfolio is growing, shrinking, or rebalancing — critical for PPA covenant compliance and offtake contract management.

📍 Where it lives

Executive summary, commercial reporting, and Power BI page 1 — Portfolio Overview. This is the headline chart a CFO or DFI would look at first.

🔎 How to read it

Each colour band = one technology. Natural Gas (dark green) consistently forms the largest block (~40+ TWh/yr) due to Azito's 713 MW baseload. Solar varies by season but is predictable year-on-year. Wind (cyan) is small but steady. Rising total height year-on-year = portfolio growth.

💡

Key Insight — Natural Gas contributes ~90% of all generation despite representing only 3 of 17 plants — the portfolio is highly concentrated. The 485 MW construction pipeline (CTT + Menengai) will shift the mix toward gas and geothermal by 2027.

Forced Outage Count & Average Duration (2020–2024)
Chart 04 Forced Outage Count & Average Duration (2020–2024)
🔍 What

Dual-axis chart: bars show total forced outage events per year (left axis, amber); the line shows average outage duration in hours per year (right axis, green). Data sourced from fact_outage_5yr with OutageType='Forced'.

💡 Why it matters

Reliability trend monitoring — a declining count with stable duration indicates improving O&M practices. A rising count or increasing duration is an early warning signal that should trigger deep-dive maintenance review before contractual availability guarantees are breached.

📍 Where it lives

HSE and Operations chapter, Power BI page 4 — Reliability Analytics, and the ADF IfCondition gate that decides whether to send a Teams alert after each pipeline run.

🔎 How to read it

Read bars (left axis) for frequency trend. Read the line (right axis) for severity trend. The ideal trajectory is both declining. A flat line with falling bars = fewer but more complex faults. A rising line = faults are getting harder to resolve, possibly indicating ageing assets or skill gaps.

💡

Key Insight — Count fell 2020→2023, then spiked in 2024 while duration also increased — a combined signal suggesting a maintenance backlog built up during a budget-constrained period. Action: review 2024 forced outage root causes for patterns across plant types.

🤖 ML Regression — Energy Yield Forecaster  ·  Charts 05–07
XGBoost Energy Yield Forecaster — Actual vs Predicted
Chart 05 XGBoost Energy Yield Forecaster — Actual vs Predicted
🔍 What

Scatter plot of 6,943 test-set observations (20% holdout, random_state=42). Each point represents one plant-day. Coloured by primary technology. The dashed diagonal is the perfect-fit line; the shaded band is ±10%.

💡 Why it matters

Primary model validation for the Energy Yield Forecaster. This is the single most important chart for a data scientist or technical reviewer — it proves the model generalises to unseen data, not just memorises training patterns.

📍 Where it lives

Notebook 04 ML section, MLflow experiment tracking, and README model scorecard. Would be displayed on Power BI page 7 — ML Model Performance.

🔎 How to read it

Points on the diagonal = perfect prediction. Points above = model underpredicts (plant outperformed forecast). Points below = model overpredicts. Tight clustering along the diagonal = low bias and low variance. Technology-specific clusters reveal whether any technology is systematically mis-forecast.

💡

Key Insight — R²=0.998 with MAE=62 MWh — the model captures 99.8% of variance in daily energy output. NameplateCapacity + AvailabilityPct together almost perfectly explain generation, confirming the synthetic data's physical consistency. Natural Gas (dark green, top right) shows the tightest cluster.

XGBoost Energy Yield — Residual Diagnostics
Chart 06 XGBoost Energy Yield — Residual Diagnostics
🔍 What

Two-panel residual analysis. Left: residuals (Actual − Predicted) plotted against predicted values with ±1σ bands. Right: histogram of the residual distribution with mean line and zero-bias reference.

💡 Why it matters

Residual analysis is mandatory for regression model sign-off. Random scatter in the left panel confirms homoskedasticity (equal variance across the prediction range). A normal, zero-centred histogram in the right panel confirms no systematic bias. Both are requirements before deploying a model to production.

📍 Where it lives

MLflow model validation step, Notebook 04 post-training diagnostics. The Databricks notebook would fail the IfCondition ADF gate if RMSE exceeded threshold — residual shape tells you why.

🔎 How to read it

Left panel: if residuals fan outward with increasing predicted values (funnel shape), the model is heteroskedastic — predictions are less reliable at high values. Dotted lines = ±1σ. Right panel: a normal bell centred on 0 = no systematic over/under-prediction. Skewed distribution = bias.

💡

Key Insight — Residuals are randomly scattered and normally distributed around zero (mean ≈ 0). No funnel shape. Model is unbiased and homoskedastic — safe to use for grid scheduling and PPA compliance reporting.

XGBoost Feature Importance — Energy Yield Forecaster
Chart 07 XGBoost Feature Importance — Energy Yield Forecaster
🔍 What

Horizontal bar chart of gain-based feature importance scores for the 7 input features: NameplateCapacity, AvailabilityPct, CurtailmentPct, Month, ForcedDowntimeHours, PlannedDowntimeHours, Year.

💡 Why it matters

Explainability is a regulatory requirement in energy markets. Lenders, offtakers, and regulators ask 'which variables drive your forecast?' — this chart answers directly. It also guides future feature engineering: low-importance features are dropped; high-importance ones are engineered further.

📍 Where it lives

Model governance documentation, MLflow experiment metadata, and the SHAP explainability module in Notebook 04. Referenced in the project evidence note as evidence of explainable ML practice.

🔎 How to read it

Longer bar = higher contribution to model decisions. Features with near-zero importance can be safely dropped to reduce model complexity. The top 2–3 features typically explain 80%+ of model behaviour. Compare across model versions to detect feature drift.

💡

Key Insight — NameplateCapacity is the dominant feature — physical capacity caps maximum possible generation. AvailabilityPct is second — when a plant is running, capacity factor flows through. Month captures seasonal irradiance patterns for solar. Year shows a mild upward trend. Forced and Planned Downtime provide marginal additional signal.

🎯 ML Classification & Anomaly Detection  ·  Charts 08–11
LightGBM Plant Availability Tier Classifier — ROC & Precision-Recall
Chart 08/09 LightGBM Plant Availability Tier Classifier — ROC & Precision-Recall
🔍 What

Two-panel classification diagnostics. Left: ROC curve (AUC=0.85) showing True Positive Rate vs False Positive Rate across all thresholds. Right: Precision-Recall curve (AP=0.97) showing the precision/recall trade-off with an operating point marked at threshold=0.50. Target: will this plant achieve ≥90% availability next month? Features use prior-month lagged values only — fully prospective, no data leakage.

💡 Why it matters

For a binary classifier, accuracy is misleading on imbalanced data. AUC and AP measure the model's ability to rank and retrieve the positive class correctly. The ROC curve tells operations how many false alarms they must accept to catch a given fraction of underperformance months. The PR curve optimises the precision/recall trade-off for actual dispatch decisions.

📍 Where it lives

Predictive maintenance scheduling, Power BI page 6 — Plant Risk Dashboard, and the ADF IfCondition that triggers a Teams notification if any plant's predicted availability drops below 85% in the next month.

🔎 How to read it

ROC: A curve hugging the top-left corner = excellent discrimination. The diagonal = random guessing. AUC=0.85 means the model ranks a randomly chosen low-availability month above a high-availability month 85% of the time. PR: High precision = few false alarms. High recall = few missed underperformance months. The amber dot shows the chosen operating threshold.

💡

Key Insight — AUC=0.85 on purely prospective features (prior month lagged values, no same-day data) confirms the model is genuinely predictive, not memorising. AP=0.97 is high because ≥90% availability months are the majority class — the model confidently predicts the dominant outcome while still identifying the risky minority.

Random Forest Maintenance Cost Estimator — Actual vs Predicted
Chart 10 Random Forest Maintenance Cost Estimator — Actual vs Predicted
🔍 What

Scatter plot of 776 test-set maintenance work orders (20% holdout). Each point is one work order coloured by technology. The diagonal is perfect fit; the shaded band is ±15% (typical OPEX budget tolerance). R²=0.999, MAE=R103, OOB=0.999.

💡 Why it matters

Maintenance cost estimation underpins OPEX budgeting, insurance valuations, and refinancing negotiations. An inaccurate estimator forces finance teams to hold excessive cash reserves. This chart proves the estimator is tight enough for budget-grade reporting.

📍 Where it lives

Finance chapter, Power BI page 5 — OPEX & Maintenance, and the Gold layer fact_maintenance_work_order aggregations. The model is called when a new work order is opened in the CMMS to estimate total cost before work begins.

🔎 How to read it

Points on the diagonal = budget estimate matches actual spend. Points above = actual exceeded estimate (budget overrun risk). Points below = actual was less than estimated (conservative budgeting). ±15% band = typical acceptable OPEX variance for a DFI-grade reporting standard. Technology colour shows whether any asset class is systematically misestimated.

💡

Key Insight — R²=0.999 with MAE=R103 reflects the synthetic data's deterministic cost structure: TotalCost ≈ f(ActualLabourHours × rate + CapacityProxyFee). In real CMMS data, MAE would be higher (~R8,000–R20,000) due to parts procurement variability — but the model architecture and feature set are production-ready.

Isolation Forest — Curtailment Anomaly Detection
Chart 11 Isolation Forest — Curtailment Anomaly Detection
🔍 What

Two-panel anomaly analysis. Left: scatter of Availability % vs Capacity Factor %, with 31,059 normal days (cyan) and 1,553 anomalous days (red, 5.0%) plotted. Right: histogram of anomaly scores — normal days cluster above 0; anomalies cluster below 0 (the decision boundary).

💡 Why it matters

Curtailment anomalies are often contractually compensable: if the grid operator curtails a plant without valid justification, the IPP is entitled to deemed energy payment. Detecting these events automatically and flagging them for commercial review can recover significant revenue — estimated R2–5M per event for large gas plants.

📍 Where it lives

Data quality pipeline (Silver layer Section 8), Power BI page 8 — Data Quality & Anomalies, and the ADF ForEach loop that checks each plant's anomaly count after every pipeline run.

🔎 How to read it

Left panel: anomalies (red) concentrate at the intersection of LOW availability AND LOW capacity factor — days where the plant was technically available but not generating, suggesting an external curtailment or sensor fault. Right panel: scores below the vertical amber line (0) are anomalies. The deeper negative, the more anomalous the observation.

💡

Key Insight — 5.0% anomaly rate (contamination=0.05) across 31,059 plant-days = 1,553 flagged events over 5 years. Solar PV plants show more anomalies than Gas due to intermittency patterns. Each flagged event triggers a FactDataQualityEvent record (NEVER DELETE, immutable audit) in the V2 SQL schema.

📈 Forecasting & Portfolio Overview  ·  Charts 12–14
Portfolio Revenue — LightGBM Forecaster vs Actual (2020–2024)
Chart 12 Portfolio Revenue — LightGBM Forecaster vs Actual (2020–2024)
🔍 What

60-month time series of total portfolio revenue (R millions/month). Amber line = actual; dashed green = LightGBM forecast; dotted cyan = linear trend; shaded band = 90% prediction interval. R²=0.93, MAPE=3.1%. Vertical grey lines mark year boundaries.

💡 Why it matters

Revenue forecasting is the commercial core of any IPP portfolio. PPA offtake agreements require quarterly generation reports; DFI loan covenants typically require annual revenue within ±5% of forecast. A MAPE of 3.1% keeps the portfolio within covenant tolerance in all 60 months.

📍 Where it lives

Commercial chapter, Power BI page 2 — Revenue & Settlements, executive dashboard, and the ADF WebActivity that pushes the latest monthly actuals to a Power BI push dataset for near-real-time reporting.

🔎 How to read it

Amber line vs dashed green: tight overlap = accurate forecast. When actual rises above forecast, the portfolio over-delivered (positive surprise for DFI reporting). The shaded band widens in 2024 — uncertainty grows further out. The cyan trend line confirms modest revenue growth across 5 years despite stable PPA tariffs, driven by improved availability.

💡

Key Insight — Revenue is stable and slightly growing (+2.1%/yr linear trend) despite fixed PPA tariffs because availability has improved year-on-year. 2022 shows a slight dip — aligns with the forced outage spike in Chart 04. The 90% prediction interval narrows in 2020–2022 (model has historical context) and widens in 2023–2024 (less history at training time).

Technology Performance Comparison — Availability, Capacity Factor & Generation
Chart 13 Technology Performance Comparison — Availability, Capacity Factor & Generation
🔍 What

Three-panel side-by-side bar chart comparing 5 technologies across: (1) Average Availability % — how reliably each technology runs; (2) Average Capacity Factor % — how hard it works when running; (3) Total Generation over 5 years (TWh) — absolute contribution to the portfolio.

💡 Why it matters

Technology benchmarking answers the strategic question: where should the next 485 MW of construction budget be deployed? It also informs O&M contract terms — technologies with high availability but low capacity factor (Solar PV) need different SLAs than high-CF baseload gas.

📍 Where it lives

Board-level strategy deck, Power BI page 1 — Portfolio Overview, and the investment committee report on the CTT (450 MW gas) and Menengai (35 MW geothermal) construction projects.

🔎 How to read it

Left: taller bar = more reliable. Right: taller bar = works harder per installed MW. Centre: any two technologies can trade off between left and right panels — a plant that is 98% available but 15% CF (Solar) vs 90% available and 78% CF (Gas) serve very different roles. Cross-reference all three panels to assess a technology's true portfolio value.

💡

Key Insight — Natural Gas runs 78% capacity factor (24/7 baseload) vs Solar PV at ~22% (daylight only) — but Gas availability (~91%) is actually lower than Solar (~95%) because gas turbines require more planned outages for inspection. Wind (JBAY + Klipheuwel) punches above its weight at ~34% CF with high availability.

Operational Driver Scatter Matrix (n=3,000 sample)
Chart 14 Operational Driver Scatter Matrix (n=3,000 sample)
🔍 What

5×5 pairplot of: Availability %, Capacity Factor %, Forced Downtime (hrs), Curtailment %, and Gross Generation (MWh). Sampled from Natural Gas, Solar PV, and Wind plants (1,000 per technology). Diagonal = KDE density curve. Off-diagonal = bivariate scatter.

💡 Why it matters

An analyst's first stop in any new dataset — the scatter matrix reveals distributional shapes, non-linear relationships, outlier clusters, and bimodal patterns in a single view. It directly informs which transformations and feature engineering steps are needed before modelling.

📍 Where it lives

Exploratory Data Analysis (EDA) phase, Silver layer validation, and the feature store design in Notebook 03 Gold. If a relationship appears non-linear here, it signals that a tree-based model (XGBoost, LightGBM) will outperform linear regression.

🔎 How to read it

Read the diagonal (top-left to bottom-right) for each variable's distribution shape. Read off-diagonal cells for pairwise relationships: a cigar-shaped cloud = linear; a fan = heteroskedastic; a blob = no relationship; two clusters = bimodal (often two technology types mixed). Each point's colour = technology type.

💡

Key Insight — Availability vs Capacity Factor shows a positive linear relationship — higher availability days also achieve higher capacity factors. Forced Downtime is heavily right-skewed (most days = zero, a few days = large values) — this is why tree models handle it better than linear regression. Curtailment is near-zero for gas, higher for Solar on grid-constrained days.