What a small team learned by building the whole Lumen Ridge platform on synthetic data first: what worked, what it cost, and what we would never do again.
Lumen Ridge runs an online casino and sportsbook, a wallet and payments stack, and a few hundred retail betting shops with cashiers, terminals and shifts. The real platform will pull from seven sources: five APIs, one PostgreSQL database and one file feed. Finance needs internal figures to agree with gateways and banks to within 0.1 percent, with every difference explained.
The constraint that shaped everything was money: a USD 200 monthly budget, email alerts at USD 100, 150, 176, 190 and 200, and an automated kill switch that stops all compute at USD 190. Every design choice in this book starts from that number.
Clean demo data proves nothing. Two generators produced a "dirty" month instead: late-arriving rows, exact re-sends of whole files, two different versions of a bet with the same timestamp, settlements that never arrive, and reference snapshots that change under the facts. The online generator is fingerprinted, so data from different code versions can never be mixed in one load.
| Dataset | Days | Rows landed | Built with |
|---|---|---|---|
| Online casino, sportsbook, wallet, payments | 30 | 106.7M | Notebooks (Bronze, Silver, Gold) |
| Retail slips, bets, cash, floats, shifts | about 30 | 114.9M | COPY INTO + dbt |
| Gold daily summaries | all | about 163k | dbt tables |
The platform is a medallion lakehouse in Azure Databricks with Unity Catalog. Bronze keeps every delivered version, never updated. Silver keeps the latest valid version per key and sends everything else to quarantine with a reason. Gold is a star schema with declared grains, and money lives on exactly one fact per process so joins cannot double count. All compute is serverless and stops when idle.
Every delivery arrives with a manifest: file checksum, size and row count. Nothing lands until all three match. Landing uses COPY INTO, which skips files it has already loaded, so a re-sent delivery cannot land twice. Every Bronze row carries its run id, source file, schema version and ingest time.
The rule that overrides everything else: Bronze may keep re-sent rows as audit history, but Silver and Gold hold exactly one row per declared grain. Each trusted Silver model declares its business key, compound keys included, and a unique_grain test fails the build before Gold is touched. A local run of 25 grain checks across Silver and Gold found zero duplicate keys, on data that deliberately contained re-sends.
Retail Silver is generated from a rules file rather than hand-written: types, allowed values, parent checks and version rules live as data, and a script writes 58 dbt models from them. The latest version per key wins by change time, then ingest time, then payload hash. If the same key arrives twice with the same change time and different content, both versions go to quarantine as CONFLICTING_DUPLICATE: a silent pick would hide a source problem.
Only tables that earn their storage are tables. A small check fails the build if a dbt model becomes a table without a written reason, which turned eleven unnecessary intermediate tables into views.
The first Power BI refresh against the row-level facts ran for more than twenty minutes and kept the warehouse awake the whole time. The fix was upstream: seven daily summary tables in Gold (casino, sportsbook, wallet and payments by day and operator; retail bets, cash and shifts by day and shop). They built in 53 seconds for about four US cents, and tests prove each one equals its fact exactly in rows and money.
Reconciliation is not a report at the end; it is a ladder of checks at every hop, each writing to an append-only results table with a cause code. When figures stop agreeing, the first failing rung tells you where to look.
| Rung | Compares | Typical cause codes |
|---|---|---|
| Source → file | manifest rows, money, checksum | SHORT_READ, HASH_MISMATCH |
| File → Bronze | rows and money landed | NOT_LANDED, DUPLICATE_LOAD |
| Bronze → Silver | accepted + quarantined + duplicates = Bronze keys | QUARANTINED, NOT_LOADED_TO_SILVER |
| Silver → Gold | key by key, money exact | MISSING_IN_GOLD, VALUE_MISMATCH |
| Gold → report | summary totals vs facts for the same window | REPORT_MISMATCH |
| Business rules | casino vs wallet, payment vs settlement, cash vs shift | MISSING_IN_WALLET, SETTLEMENT_NOT_DUE |
For the online month, Silver against Gold matched on all four checks. The cross-system breaks that remained were the ones the generator planted, each explained by a cause code.
Databricks reported USD 45.87 at list price for the first five days. Azure billed USD 5.89. The workspace was on a 14-day trial, so compute was free on the invoice, and the real charges were the network gateway (USD 4.74), a public IP and storage. The lesson: list-price usage tells you what drove cost; the cloud bill tells you what you pay. Track both, and decide what happens when the trial ends.
Four separate batches each paid their own start-up, reconciliation ran after every batch instead of once, and Power BI imported the full row-level facts. The rules since then: larger batches, light checks per batch with the full ladder once at the end, summaries first in Power BI, and the warehouse stopped straight after validation.
The rehearsal ran in one workspace. The real project will use two, dev and prod, each with its own catalog bound only to its own workspace, its own storage, Key Vault and service principal. Only code moves between them, through a reviewed pull request and an approval gate. Data never moves from dev to prod; prod rebuilds from its own Bronze. Dev works on synthetic or masked samples only.
Cost note: a second workspace with private networking adds a fixed network gateway charge of about USD 35 a month even when idle, so the decision belongs at project start, as a line item.
A single YAML file describes every table and column once: owner, domain, grain, source, refresh and personal-data class. A script turns it into Unity Catalog comments and tags and a live catalogue view that also shows each table's latest quality failure rate, reconciliation status and freshness. The first run described 171 objects and 2,120 columns and tagged 17 objects as holding personal data. A business glossary links each term, such as GGR, to the columns that implement it.
Both Power BI reports are generated as text projects, statically validated before anyone opens them, and read the Gold summaries by default. Detail tables load only a bounded date window, and a Reconciliation page shows the summaries matching the detail for the selected days.
A rehearsal should end on purpose. A scheduled runbook deletes the Databricks workspace one day before the trial ends, or immediately if spend ever reaches the full budget, whichever comes first. The network gateway goes with it. The data stays in storage, the code stays in git, and the whole platform can be rebuilt from Terraform in about an hour when the real project begins.