A full-stack Azure data engineering and machine learning pipeline built on 3 million rows of synthetic SCADA telemetry from 19 African power plants — powering predictive operations, revenue forecasting, and ESG reporting.
Globeleq operates utility-scale power plants across six African nations. Each plant generates megawatts of electricity and terabytes of SCADA data. Without a unified data platform, these operational signals sit in siloed plant historians — invisible to the portfolio managers, revenue analysts, and maintenance planners who need them most.
"We are powering Africa's growth by developing and operating utility-scale power plants."
— Globeleq mission statement15-minute sensor telemetry ingested via ADF into Delta Lake Bronze — available for analysis within 30 minutes of generation.
Nine models covering yield forecasting, availability classification, maintenance cost, anomaly detection, revenue projection, solar irradiance yield forecasting, and three production forecasting challengers.
Automated CO₂ avoided, Scope 1 emissions, renewable generation share — updated daily, investor-ready monthly.
19 operating and in-construction plants spanning Northern, Western, Eastern and Southern Africa — Solar PV, Wind, Natural Gas, Heavy Fuel Oil and Geothermal technologies.
| # | Plant Name | Country | Technology | Capacity | Offtaker | Agreement | Status |
|---|---|---|---|---|---|---|---|
| 1 | ARC Solar Power | Egypt | Solar PV | 66 MWp | EETC | PPA 25yr | Operating |
| 2 | Aries Solar Power | South Africa | Solar PV | 11 MWp | Eskom | PPA 20yr | Operating |
| 3 | Azito Power | Côte d'Ivoire | Natural Gas | 713 MW | Govt of CdI | Concession 20yr | Operating |
| 4 | Boshof Solar Power | South Africa | Solar PV | 66 MWp | Eskom | PPA 20yr | Operating |
| 5 | Cuamba Solar + BESS | Mozambique | Solar PV+BESS | 19 MWp / 7 MWh | — | — | Operating |
| 6 | De Aar Solar Power | South Africa | Solar PV | 50 MWp | Eskom | PPA 20yr | Operating |
| 7 | Dibamba Power | Cameroon | Heavy Fuel Oil | 88 MW | ENEO | Tolling 20yr | Operating |
| 8 | Droogfontein Solar | South Africa | Solar PV | 50 MWp | Eskom | PPA 20yr | Operating |
| 9 | Jeffreys Bay Wind | South Africa | Wind | 138 MW | — | — | Operating |
| 10 | Klipheuwel Wind Farm | South Africa | Wind | 27 MW | — | — | Operating |
| 11 | Konkoonsies Solar | South Africa | Solar PV | 11 MWp | Eskom | PPA 20yr | Operating |
| 12 | Kribi Power | Cameroon | Natural Gas | 216 MW | National Grid | PPA 20yr | Operating |
| 13 | Malindi Solar Power | Kenya | Solar PV | 52 MWp | Kenya Power | PPA 20yr | Operating |
| 14 | Mocuba Solar | Mozambique | Solar PV | 41 MWp | EDM | PPA 25yr | Operating |
| 15 | Songas | Tanzania | Natural Gas | 190 MW | National Grid | PPA 20yr | Operating |
| 16 | Soutpan Solar | South Africa | Solar PV | 31 MWp | Eskom | PPA 20yr | Operating |
| 17 | Winnergy Solar | Egypt | Solar PV | 25 MWp | EETC | PPA 25yr | Operating |
| 18 | Central Térmica Temane | Mozambique | Natural Gas | 450 MW | EDM | Tolling 25yr | In Construction |
| 19 | Menengai Geothermal | Kenya | Geothermal | 35 MW | KPLC | — | In Construction |
A classic Azure Medallion Architecture: raw data lands in Bronze Delta tables via ADF, is cleansed and enriched in Silver, and aggregated into executive-ready Gold KPIs and an ML feature store.
Synthetic data was generated using physically-motivated models: solar irradiance calculations based on plant latitude and day of year, Weibull wind speed distributions with AR(1) temporal correlation, and Poisson forced-outage arrival processes for thermal plants.
-- 15-minute SCADA fact table (2.98M rows) CREATE TABLE scada_telemetry_15min ( Timestamp TIMESTAMP, PlantKey INT, PlantCode VARCHAR(10), Technology VARCHAR(30), ActivePowerMW DOUBLE, -- MW ReactivePowerMVAR DOUBLE, GridFrequencyHz DOUBLE, -- ~50 Hz AmbientTemperatureC DOUBLE, SolarIrradianceWm2 DOUBLE, -- solar only WindSpeedMs DOUBLE, -- wind only InverterEfficiencyPct DOUBLE, TransformerTempC DOUBLE, CurtailmentFactor DOUBLE, CumulativeEnergyMWh DOUBLE, AlarmCode VARCHAR(20), -- NORMAL/OUTAGE/CURTAILMENT DataQualityFlag VARCHAR(10) -- GOOD/SUSPECT/BAD ) PARTITIONED BY (PlantKey)
Delta Lake enforces column types at Bronze ingestion. Reject-on-schema-mismatch prevents silent null propagation.
15-min readings during forced outages tagged DataQualityFlag='SUSPECT' — excluded from KPI denominators.
Silver layer drops exact duplicates on (Timestamp, PlantKey) using Delta dropDuplicates() before merge.
The Azure Data Factory pipeline runs at 02:00 SAST every night, copying new data from SQL Server and ADLS Gen2, executing the four Databricks notebooks in sequence, conditionally retraining ML models on Saturdays, refreshing Power BI, and posting a Teams webhook.
dayOfWeek(utcNow())==7 — daily runs execute inference only to save cluster cost.// tr_daily_0200_sast { "type": "ScheduleTrigger", "recurrence": { "frequency": "Day", "interval": 1, "startTime": "2025-01-01T00:00:00Z", "schedule": { "hours": [0], // 00:00 UTC = 02:00 SAST "minutes": [0] } } }
| Name | Type | Auth |
|---|---|---|
| ls_adls_globeleq | ADLS Gen2 | Service Principal + KV |
| ls_databricks_globeleq | Databricks | MSI Token |
| ls_sql_globeleq | Azure SQL | Managed Identity |
| ls_powerbi | Power BI REST | OAuth2 Bearer |
Nine models address distinct business questions across the plant lifecycle — from next-day solar yield prediction to monthly revenue forecasting. Six operational models (XGBoost, LightGBM, Random Forest, Isolation Forest, LightGBM Regressor, Solar Yield LightGBM) are tracked in MLflow alongside three production-ready forecasting challengers, all versioned in the Model Registry and refreshed weekly.
| # | Model | Algorithm | Target | Key Metrics | Business Use |
|---|---|---|---|---|---|
| 1 | Energy Yield Forecaster | XGBoost Regressor | NetGenerationMWh (t+1 day) | R²=0.998 · MAE=62 MWh · RMSE=151 MWh | Grid scheduling & energy trading |
| 2 | Plant Availability Tier Classifier | LightGBM Classifier | ≥90% availability next month | AUC=0.85 · AP=0.97 · prospective | Predictive maintenance dispatch |
| 3 | Maintenance Cost Estimator | Random Forest | Monthly maintenance cost (ZAR) | R²=0.999 · MAE=R103 · OOB=0.999 | OPEX budgeting & planning |
| 4 | Curtailment Anomaly Detector | Isolation Forest | Anomalous curtailment months | Anomaly rate=5% · Contamination=0.05 | Grid congestion & inverter alerts |
| 5 | Portfolio Revenue Forecaster | LightGBM Regressor | Portfolio revenue ZAR (t+1 month) | R²=0.93 · MAE=R2.8M · MAPE=3.1% | Investor & cash-flow reporting |
XGBoost regressor trained on 80/20 split of 34,713 plant-day records. Features include 7-day and 30-day rolling averages, lag-1 and lag-7 generation, plant capacity, technology encoding, and seasonality (month).
xgb_params = { "n_estimators" : 500, "max_depth" : 6, "learning_rate" : 0.05, "subsample" : 0.8, "colsample_bytree": 0.8, "min_child_weight": 5, "reg_alpha" : 0.1, "reg_lambda" : 1.0, } # Top features: Gen_Lag1 > AvailabilityPct # > GenMWh_7d > NameplateCapacity > Month
LightGBM binary classifier with class balancing (scale_pos_weight).
Threshold optimised to 0.35 to maximise recall — better to dispatch a maintenance
crew unnecessarily than to miss a real fault.
# Class imbalance: only ~8% of days have # a forced outage in the next 7 days lgb_params = { "scale_pos_weight": scale_pos, # ~11x "class_weight" : "balanced", "num_leaves" : 31, "min_child_samples": 30, "early_stopping" : 50, } # Optimised threshold: 0.35 (not 0.50) # Top features: AvailabilityPct, ForcedOut_Lag7
Key insights derived from 5 years of synthetic operational data across the Globeleq portfolio.
11 solar plants deliver 22–24% capacity factors across Africa's high-irradiance zones. Egypt plants (ARC, Winnergy) achieve the most consistent generation with minimal seasonal swing.
Jeffreys Bay (JBAY, 138 MW) achieves ~38% capacity factor — nearly double the solar fleet average. South Africa's coastal wind corridor is among the best in the continent.
Azito (713 MW, Côte d'Ivoire) alone generates more revenue than the entire South African solar fleet combined, operating at ~78% capacity factor 24/7.
Operating fleet (17 plants) averaged ~92% availability across 2020–2024. Planned annual maintenance windows account for ~5% of downtime; forced outages for the remaining ~3%.
The renewable fleet avoided 8.4 million tonnes of CO₂ over 5 years, equivalent to removing 1.8 million cars from Africa's roads annually.
Weighted settlement collection rate held above 90% across all regions. East Africa (Kenya, Tanzania) shows the most consistent collection; some West Africa months show 85–88%.
Technical decisions, trade-offs, and the rationale behind each engineering choice.
scale_pos_weight and early stopping — critical for weekly retraining on a shared cluster.Medallion architecture · Delta Lake MERGE · PySpark window functions · ADF ForEach + IfCondition · ADLS Gen2 OAuth2 mounting · OPTIMIZE + ZORDER
XGBoost / LightGBM / Random Forest / Isolation Forest · MLflow experiment tracking · Model Registry · SHAP explainability · Class imbalance handling · Threshold optimisation
IPP commercial structure (PPA/Tolling/Concession) · SCADA telemetry schemas · Capacity factor / availability calculations · GHG Scope 1 accounting · DFI-grade ESG reporting
Python · PySpark · SQL (Spark SQL + SQL Server DDL) · openpyxl · Azure Databricks · ADF · Power BI DAX · Git · JSON pipeline definitions
13 publication-quality charts generated directly from 3,024,807 rows of synthetic operational data. Each chart is explained across four dimensions — What it shows, Why it matters commercially, Where it fits in the platform, and How to read it correctly.
Pearson correlation heatmap of 9 daily operational KPIs across 34,713 plant-days (2020–2024). Variables include Availability %, Capacity Factor %, Gross & Net Generation, Curtailment %, Planned and Forced Downtime, CO₂ Avoided, and Scope 1 Emissions.
Before building ML models, understanding multicollinearity prevents feature redundancy. Identifying which KPIs move together reveals genuine physical relationships vs artificial duplicates, and informs which variables to include or drop from the feature store.
ML feature engineering (Notebook 03 Gold layer). Also informs which DAX measures should be calculated independently vs derived from others in Power BI.
Blue = strong positive correlation (e.g., GrossGen ↔ NetGen = 0.99 — expected). Red = negative (ForcedDowntime ↔ Availability = −0.68 — outages destroy availability). Near zero = no linear relationship (Curtailment ↔ Scope1 ≈ 0.02).
17 operating plants × 5 years grid showing mean daily Availability %, sorted by 2024 performance descending. Each cell is the annual average across all operational days for that plant.
Reveals which plants show persistent underperformance (row-level trend) vs isolated bad years (single dark cell). Enables the asset management team to target plants for O&M contract renegotiation or equipment inspection.
Portfolio Review (Chapter 7 Results), O&M planning, and Power BI page 3 — Plant Health Dashboard. Feeds directly into the Plant Availability Tier Classifier training data.
Green cells (>95%) = high performers — no action needed. Yellow (85–95%) = watch list. Orange/red (<80%) = intervention candidates. Read each row left-to-right to see whether a plant is improving, stable, or declining over time.
Stacked bar chart of annual gross generation (TWh) broken down by primary technology: Natural Gas, Solar PV, Wind, Heavy Fuel Oil, Solar PV + BESS. Covers all 17 operating plants for 2020–2024.
Shows portfolio energy composition and year-on-year stability. Helps identify whether the portfolio is growing, shrinking, or rebalancing — critical for PPA covenant compliance and offtake contract management.
Executive summary, commercial reporting, and Power BI page 1 — Portfolio Overview. This is the headline chart a CFO or DFI would look at first.
Each colour band = one technology. Natural Gas (dark green) consistently forms the largest block (~40+ TWh/yr) due to Azito's 713 MW baseload. Solar varies by season but is predictable year-on-year. Wind (cyan) is small but steady. Rising total height year-on-year = portfolio growth.
Dual-axis chart: bars show total forced outage events per year (left axis, amber); the line shows average outage duration in hours per year (right axis, green). Data sourced from fact_outage_5yr with OutageType='Forced'.
Reliability trend monitoring — a declining count with stable duration indicates improving O&M practices. A rising count or increasing duration is an early warning signal that should trigger deep-dive maintenance review before contractual availability guarantees are breached.
HSE and Operations chapter, Power BI page 4 — Reliability Analytics, and the ADF IfCondition gate that decides whether to send a Teams alert after each pipeline run.
Read bars (left axis) for frequency trend. Read the line (right axis) for severity trend. The ideal trajectory is both declining. A flat line with falling bars = fewer but more complex faults. A rising line = faults are getting harder to resolve, possibly indicating ageing assets or skill gaps.
Scatter plot of 6,943 test-set observations (20% holdout, random_state=42). Each point represents one plant-day. Coloured by primary technology. The dashed diagonal is the perfect-fit line; the shaded band is ±10%.
Primary model validation for the Energy Yield Forecaster. This is the single most important chart for a data scientist or technical reviewer — it proves the model generalises to unseen data, not just memorises training patterns.
Notebook 04 ML section, MLflow experiment tracking, and README model scorecard. Would be displayed on Power BI page 7 — ML Model Performance.
Points on the diagonal = perfect prediction. Points above = model underpredicts (plant outperformed forecast). Points below = model overpredicts. Tight clustering along the diagonal = low bias and low variance. Technology-specific clusters reveal whether any technology is systematically mis-forecast.
Two-panel residual analysis. Left: residuals (Actual − Predicted) plotted against predicted values with ±1σ bands. Right: histogram of the residual distribution with mean line and zero-bias reference.
Residual analysis is mandatory for regression model sign-off. Random scatter in the left panel confirms homoskedasticity (equal variance across the prediction range). A normal, zero-centred histogram in the right panel confirms no systematic bias. Both are requirements before deploying a model to production.
MLflow model validation step, Notebook 04 post-training diagnostics. The Databricks notebook would fail the IfCondition ADF gate if RMSE exceeded threshold — residual shape tells you why.
Left panel: if residuals fan outward with increasing predicted values (funnel shape), the model is heteroskedastic — predictions are less reliable at high values. Dotted lines = ±1σ. Right panel: a normal bell centred on 0 = no systematic over/under-prediction. Skewed distribution = bias.
Horizontal bar chart of gain-based feature importance scores for the 7 input features: NameplateCapacity, AvailabilityPct, CurtailmentPct, Month, ForcedDowntimeHours, PlannedDowntimeHours, Year.
Explainability is a regulatory requirement in energy markets. Lenders, offtakers, and regulators ask 'which variables drive your forecast?' — this chart answers directly. It also guides future feature engineering: low-importance features are dropped; high-importance ones are engineered further.
Model governance documentation, MLflow experiment metadata, and the SHAP explainability module in Notebook 04. Referenced in the project evidence note as evidence of explainable ML practice.
Longer bar = higher contribution to model decisions. Features with near-zero importance can be safely dropped to reduce model complexity. The top 2–3 features typically explain 80%+ of model behaviour. Compare across model versions to detect feature drift.
Two-panel classification diagnostics. Left: ROC curve (AUC=0.85) showing True Positive Rate vs False Positive Rate across all thresholds. Right: Precision-Recall curve (AP=0.97) showing the precision/recall trade-off with an operating point marked at threshold=0.50. Target: will this plant achieve ≥90% availability next month? Features use prior-month lagged values only — fully prospective, no data leakage.
For a binary classifier, accuracy is misleading on imbalanced data. AUC and AP measure the model's ability to rank and retrieve the positive class correctly. The ROC curve tells operations how many false alarms they must accept to catch a given fraction of underperformance months. The PR curve optimises the precision/recall trade-off for actual dispatch decisions.
Predictive maintenance scheduling, Power BI page 6 — Plant Risk Dashboard, and the ADF IfCondition that triggers a Teams notification if any plant's predicted availability drops below 85% in the next month.
ROC: A curve hugging the top-left corner = excellent discrimination. The diagonal = random guessing. AUC=0.85 means the model ranks a randomly chosen low-availability month above a high-availability month 85% of the time. PR: High precision = few false alarms. High recall = few missed underperformance months. The amber dot shows the chosen operating threshold.
Scatter plot of 776 test-set maintenance work orders (20% holdout). Each point is one work order coloured by technology. The diagonal is perfect fit; the shaded band is ±15% (typical OPEX budget tolerance). R²=0.999, MAE=R103, OOB=0.999.
Maintenance cost estimation underpins OPEX budgeting, insurance valuations, and refinancing negotiations. An inaccurate estimator forces finance teams to hold excessive cash reserves. This chart proves the estimator is tight enough for budget-grade reporting.
Finance chapter, Power BI page 5 — OPEX & Maintenance, and the Gold layer fact_maintenance_work_order aggregations. The model is called when a new work order is opened in the CMMS to estimate total cost before work begins.
Points on the diagonal = budget estimate matches actual spend. Points above = actual exceeded estimate (budget overrun risk). Points below = actual was less than estimated (conservative budgeting). ±15% band = typical acceptable OPEX variance for a DFI-grade reporting standard. Technology colour shows whether any asset class is systematically misestimated.
Two-panel anomaly analysis. Left: scatter of Availability % vs Capacity Factor %, with 31,059 normal days (cyan) and 1,553 anomalous days (red, 5.0%) plotted. Right: histogram of anomaly scores — normal days cluster above 0; anomalies cluster below 0 (the decision boundary).
Curtailment anomalies are often contractually compensable: if the grid operator curtails a plant without valid justification, the IPP is entitled to deemed energy payment. Detecting these events automatically and flagging them for commercial review can recover significant revenue — estimated R2–5M per event for large gas plants.
Data quality pipeline (Silver layer Section 8), Power BI page 8 — Data Quality & Anomalies, and the ADF ForEach loop that checks each plant's anomaly count after every pipeline run.
Left panel: anomalies (red) concentrate at the intersection of LOW availability AND LOW capacity factor — days where the plant was technically available but not generating, suggesting an external curtailment or sensor fault. Right panel: scores below the vertical amber line (0) are anomalies. The deeper negative, the more anomalous the observation.
60-month time series of total portfolio revenue (R millions/month). Amber line = actual; dashed green = LightGBM forecast; dotted cyan = linear trend; shaded band = 90% prediction interval. R²=0.93, MAPE=3.1%. Vertical grey lines mark year boundaries.
Revenue forecasting is the commercial core of any IPP portfolio. PPA offtake agreements require quarterly generation reports; DFI loan covenants typically require annual revenue within ±5% of forecast. A MAPE of 3.1% keeps the portfolio within covenant tolerance in all 60 months.
Commercial chapter, Power BI page 2 — Revenue & Settlements, executive dashboard, and the ADF WebActivity that pushes the latest monthly actuals to a Power BI push dataset for near-real-time reporting.
Amber line vs dashed green: tight overlap = accurate forecast. When actual rises above forecast, the portfolio over-delivered (positive surprise for DFI reporting). The shaded band widens in 2024 — uncertainty grows further out. The cyan trend line confirms modest revenue growth across 5 years despite stable PPA tariffs, driven by improved availability.
Three-panel side-by-side bar chart comparing 5 technologies across: (1) Average Availability % — how reliably each technology runs; (2) Average Capacity Factor % — how hard it works when running; (3) Total Generation over 5 years (TWh) — absolute contribution to the portfolio.
Technology benchmarking answers the strategic question: where should the next 485 MW of construction budget be deployed? It also informs O&M contract terms — technologies with high availability but low capacity factor (Solar PV) need different SLAs than high-CF baseload gas.
Board-level strategy deck, Power BI page 1 — Portfolio Overview, and the investment committee report on the CTT (450 MW gas) and Menengai (35 MW geothermal) construction projects.
Left: taller bar = more reliable. Right: taller bar = works harder per installed MW. Centre: any two technologies can trade off between left and right panels — a plant that is 98% available but 15% CF (Solar) vs 90% available and 78% CF (Gas) serve very different roles. Cross-reference all three panels to assess a technology's true portfolio value.
5×5 pairplot of: Availability %, Capacity Factor %, Forced Downtime (hrs), Curtailment %, and Gross Generation (MWh). Sampled from Natural Gas, Solar PV, and Wind plants (1,000 per technology). Diagonal = KDE density curve. Off-diagonal = bivariate scatter.
An analyst's first stop in any new dataset — the scatter matrix reveals distributional shapes, non-linear relationships, outlier clusters, and bimodal patterns in a single view. It directly informs which transformations and feature engineering steps are needed before modelling.
Exploratory Data Analysis (EDA) phase, Silver layer validation, and the feature store design in Notebook 03 Gold. If a relationship appears non-linear here, it signals that a tree-based model (XGBoost, LightGBM) will outperform linear regression.
Read the diagonal (top-left to bottom-right) for each variable's distribution shape. Read off-diagonal cells for pairwise relationships: a cigar-shaped cloud = linear; a fan = heteroskedastic; a blob = no relationship; two clusters = bimodal (often two technology types mixed). Each point's colour = technology type.