knaxuslearn · build · ship

Roadmaps / Data / Phase 4

Phase 4 of 4 · Study weeks 13-16

Governance, cost and platform

This phase makes the last checkpoint possible: a platform with owners, contracts, lineage, deletion and a cost you can explain.

Topics every week

  1. Week 138 h
    • Data contracts and ownership
    • Schema evolution rules
    • Breaking changes
    • Catalogs and discovery

    Hand inA contract test that fails on a seeded breaking change

  2. Week 148 h
    • Lineage with OpenLineage
    • Impact analysis
    • Documentation that stays true
    • Freshness targets and alerts

    Hand inA lineage graph from source to revenue table

  3. Week 158 h
    • Access control and masking
    • Personal data and deletion requests
    • Retention
    • Auditing

    Hand inA deletion request carried through every table and verified

  4. Week 168 h
    • Cost: bytes scanned, storage and compute
    • Finding the expensive queries
    • Platform choices
    • Review

    Hand inCheckpoint 4, the cost table and the memo

Practice projects

  • Practice 1, the breaking change (Week 13): rename a column in the source. Done when: the contract test fails before the change reaches the report table and the owner is told.
  • Practice 2, the impact question (Week 14): answer "what breaks if we drop this column". Done when: the lineage graph lists every downstream table and dashboard and you check two by hand.
  • Practice 3, the deletion (Week 15): delete a fake user. Done when: a query finds the user in no table, including snapshots after expiry, and the steps are written as a runbook.
  • Optional, a cost fix: rewrite the most expensive query. Done when: bytes scanned and time are measured before and after and the saving per month is estimated.

GitHub repos to study

Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.

RepoWhat to readWeek
OpenLineage/OpenLineageThe event model and the integrations14
datahub-project/datahubA metadata platform: ownership, lineage and search13
apache/icebergSnapshot expiry and row-level delete handling15

Milestone checkpoint

Run Checkpoint 4. Write the contracts, the freshness targets and the deletion runbook first. Add owners, run the breaking-change test, build the lineage graph, carry out the deletion and verify it, and produce the cost table. Write the memo with the numbers from the checkpoints. Hand in the lineage graph, the deletion verification, the cost table, ADR-7 (the contract and ownership policy) and the memo.

Check yourself

Answer each question in your head first, then open it to check.

  1. What is a data contract, and who enforces it?
    A written agreement on schema, meaning and freshness between a producer and its consumers. A test in the producer's pipeline enforces it, so a breaking change fails before release.
  2. Why is a deletion in a lakehouse more than a DELETE statement?
    Old snapshots and files still hold the data until you expire them and remove unreferenced files, and copies in other tables and backups also need handling.
  3. What does lineage let you do that a data catalog search does not?
    It shows the actual dependency path between tables and jobs, so you can answer impact and root-cause questions.
  4. How do you find the queries that cost the most?
    Aggregate bytes scanned and runtime per query or per user from the engine's history, and sort by total, not by single run.
  5. When would you not build a lakehouse?
    When the data is small enough for one database, when the team cannot run the extra moving parts or when the need is only a few dashboards. Put numbers on the cost first.