knaxuslearn · build · ship

Roadmaps / Data / Phase 2

Phase 2 of 4 · Study weeks 5-8

Batch you can trust

This phase makes the second checkpoint possible: daily tables that are correct, re-runnable and checked before anyone reads them.

Topics every week

  1. Week 58 h
    • Idempotent jobs: overwrite by partition and merge by key
    • Late-arriving data
    • Deterministic transforms
    • Run dates against event dates

    Hand inA daily job and the hash check that two runs match

  2. Week 68 h
    • Modelling: facts, dimensions and slowly changing dimensions
    • dbt models and tests
    • Incremental models

    Hand inRevenue and conversion models with tests

  3. Week 78 h
    • Data quality tests
    • Reconciliation against the source
    • Quarantine for bad rows
    • Alerts that mean something

    Hand inThe reconciliation report and the three seeded bad-data tests

  4. Week 88 h
    • Orchestration: dependencies, retries and backfills
    • Backfilling safely
    • Cost and runtime of a backfill
    • Predicted against measured

    Hand inCheckpoint 2: the 90-day backfill on a copy and its time

Practice projects

  • Practice 1, the double run (Week 5): run the same day twice. Done when: a hash of the output matches, and a test fails if a run appends instead of replacing.
  • Practice 2, the slowly changing dimension (Week 6): keep the history of a customer's country. Done when: a query returns the country as of 3 March and a test covers a change on that day.
  • Practice 3, the bad row (Week 7): inject a negative price, a duplicate order and a missing key. Done when: each is caught by a test, quarantined with a reason and never reaches the report table.
  • Optional, the same job in Spark: run the daily job on Spark in local mode. Done when: a table compares runtime and lines of code with your first engine.

GitHub repos to study

Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.

RepoWhat to readWeek
dbt-labs/dbt-coreHow models, tests and incremental strategies are compiled6
dagster-io/dagsterAssets, partitions and backfills8
great-expectations/great_expectationsHow checks are declared and run7
apache/sparkThe execution model and the adaptive query execution docs8
trinodb/trinoThe Iceberg connector and the query planner8

Milestone checkpoint

Build Checkpoint 2. Write the reconciliation query and the three bad-data tests first. Build the daily job and the models. Run each day twice and compare hashes. Run the 90-day backfill on a copy and time it. Hand in the hash comparison, the reconciliation report, the test results, ADR-3 (the write strategy: overwrite or merge) and ADR-4 (the quarantine and alert rules).

Check yourself

Answer each question in your head first, then open it to check.

  1. What makes a job idempotent?
    Running it again with the same inputs produces the same output, usually by overwriting a whole partition or merging on a key and never appending blindly.
  2. Why reconcile against the source, not only test the output?
    Tests catch rules you thought of. A reconciliation catches silent loss and duplication that no test described.
  3. What is the difference between event time and processing time?
    Event time is when the thing happened. Processing time is when your system saw it. Late data makes them differ, and reports usually need event time.
  4. How do you backfill without breaking readers?
    Build into a copy or a branch of the table, check it against the old one, then swap in one atomic commit, with a snapshot to roll back to.
  5. When is an alert worse than none?
    When it fires often enough that people ignore it, or fires with no clear action, so a real problem gets lost.