Roadmaps / Distributed Systems / Phase 4
Phase 4 of 4 · Study weeks 13-16
Testing, models and operations
This phase makes the last checkpoint possible: a system tested the way its failures will test it, with a runbook for the failures you found.
Topics every week
- Week 138 h
- Jepsen-style testing: generators, nemeses and checkers
- Reading a published analysis
- Writing a nemesis that matters
Hand inA 30-minute run with kills, partitions and clock changes
- Week 148 h
- Formal models with TLA+ or a model checker
- Safety and liveness
- Checking one rule
- Seeding a wrong rule
Hand inA model of the replication rule with a counterexample for the wrong rule
- Week 158 h
- Membership changes
- Rolling upgrades
- Backups of consensus state
- Observability for a cluster
Hand inA rolling upgrade drill with no lost write
- Week 168 h
- Failure drills and runbooks
- Choosing a consistency level
- Cost of consensus
- Review
Hand inCheckpoint 4, the runbook and the consistency decision guide
Practice projects
- Practice 1, the nemesis (Week 13): write a partition pattern that isolates the leader. Done when: the run finds a seeded bug that random partitions missed, and you say why.
- Practice 2, the model (Week 14): model the commit rule in TLA+ or a Python model checker. Done when: the model checks one safety property and a seeded wrong rule fails with a counterexample.
- Practice 3, the runbook (Week 16): write steps for a lost node and a lost majority. Done when: a peer follows them on your lab cluster without asking you anything.
- Optional, a read of a published analysis: reproduce one result from a published Jepsen analysis. Done when: a note says what the vendor claimed, what the test found and the fix.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.
| Repo | What to read | Week |
|---|---|---|
| jepsen-io/jepsen | A test directory for one database, end to end | 13 |
| tlaplus/Examples | The Raft and Paxos specifications | 14 |
| donnemartin/system-design-primer | Worked design questions, for the interview practice | 16 |
Milestone checkpoint
Run Checkpoint 4. Write the 30-minute run, the model and the runbook. Run the full test, fix what it finds, run the rolling upgrade drill and the model check with the seeded wrong rule. Write a one-page guide on when to choose consensus, a single leader with async replicas or an eventually consistent store, with a number or an example for each. Hand in the run summary, the model output, the drill report, ADR-7 (the testing policy) and the runbook.
Check yourself
Answer each question in your head first, then open it to check.
What does a nemesis do, and why use more than one?
It injects faults such as partitions, kills and clock jumps. Different faults expose different bugs, so one pattern leaves gaps.What can a model checker prove that a test cannot, and what can it not?
It explores every state of a small model, so it can show a rule is safe within that model. It does not show your code matches the model, and it does not scale to the full system.Why are membership changes dangerous in a consensus system?
Two overlapping configurations can each form a majority and elect two leaders. Change one node at a time or use a joint configuration.How do you run a rolling upgrade without losing a write?
Upgrade one node at a time, wait for it to rejoin and catch up before the next, keep a majority up throughout and upgrade the leader last.When is consensus the wrong tool?
When you do not need a single order of all operations: counters, caches and data where merging conflicts is acceptable. Consensus costs latency and availability.