Roadmaps / Distributed Systems / The milestone project
Read first
The milestone project
Scenario. A team needs a small configuration and lock store for its services. It must stay correct when a node dies or the network splits, even if that means refusing writes for a while. quorumkv is a key-value store with read, write and compare-and-set operations, run as 3 nodes (5 for the senior scope). Targets: operations are linearizable, which a checker confirms over a 10-minute run with random partitions and node kills; after a leader is killed, a new leader serves writes within 5 s; no acknowledged write is lost; 1,000 writes a second at p99 of 100 ms or less with 5 ms of simulated delay between nodes; the log on disk stays under 100 MB over a 1-hour run because of snapshots.
| Phase | Checkpoint | Done when |
|---|---|---|
| 1 | Failure-aware | Nodes exchange heartbeats over the simulated network. A failure detector marks a node suspected within 3 s of a kill and unsuspects it after it returns. A table compares three timeout settings with predicted and measured false suspicions and detection time. Messages carry ids, and a duplicate-delivery test shows every handler is idempotent. A seeded bug (a handler that applies a duplicated message twice) is caught by the test. |
| 2 | Replicated | A leader is elected and a log is replicated to a majority before a write is acknowledged. A kill of the leader produces a new leader within 5 s, in 20 runs. The checker confirms linearizability over a 10-minute run with partitions for a single key. A seeded bug (acknowledging a write before a majority stores it) is caught by the checker, with the failing history saved. |
| 3 | Stored and scaled | State survives a restart through a write-ahead log and snapshots, and the log stays under 100 MB over 1 hour. Reads are linearizable by a stated method (log reads or lease reads) with its assumption written down. Throughput and p99 are measured at 1,000 writes a second with spread over 3 runs. A seeded clock-skew test breaks a lease read, and the checker catches it. |
| 4 | Tested and operated | A 30-minute run with random kills, partitions and clock changes passes the linearizability checker. A short formal model (TLA+ or a Python model checker) of the replication rule is checked for one safety property, and a seeded wrong rule makes it fail. A runbook covers a lost node, a lost majority and a rolling upgrade, and the upgrade drill is run with no lost write. |
| W | Work version (Staff and Principal) | On a real system with a staging copy: write a failure test (one kill and one partition) with a checker for its main guarantee, run it, fix what it finds and record the result. Pass: a guarantee agreed in writing beforehand holds under the test, or the number of violations found falls at least 5 times over the same scenarios. The ADR is signed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later. Timebox: six weeks alongside Phases 3 and 4. |
Evidence to hand in. A repository whose README opens with the guarantee and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option; raw test output with the checker's verdict and any failing history; a one-page design doc; a diagram of a write from client to majority. Skeletons: Checkpoint 1 is the timeout table below with your numbers. Checkpoint 2 is the election table (kill, new leader time, split votes) and the checker report. Checkpoint 4 is the 30-minute run summary, the model-checker output and the runbook.
| Level | What you build and hand in |
|---|---|
| Fresher | Core scope: 3 nodes, an existing consensus library for the log, your own server, client and tests, the four checkpoints, ADRs 1 to 4. |
| Senior | Adds your own leader election and log replication, 5 nodes, snapshots, a benchmark in CI and ADRs 5 to 7. |
| Staff | Adds written decisions with numbers: a failure-testing policy, a guide for choosing a consistency level and a cost model. |
| Principal | Adds a decision on replication and consistency for several teams, with a measured outcome from the work version and a checklist other teams reuse. |
Do this at work. A solo lab is not promotion evidence. Take a real system that stores state in more than one place and run Checkpoint W on it. Start with a baseline over a month: incidents caused by partial failure and how long each took to diagnose. Make one change. Measure again. A promotion committee accepts the real decision and who made it, the named stakeholder who relied on it, before and after numbers, and a result checked again three months later.
Solo version. With no real system, take a published Jepsen analysis of a database, reproduce one of its failure scenarios in the workbench or on a local cluster, and write what the guarantee really is. Ask a peer or mentor to check your reasoning.
Laptop scale. Nodes are processes on one machine and the network is simulated, so the 5 ms delay is injected, not real. Say so in the memo. Nothing needs cloud spend.
Honest sizing. A Raft implementation takes an experienced engineer several weeks to get right, not days. That is why Phase 2 is four full weeks and why the fresher scope uses a library for the log. The checker and the fault tests are where most of the hours go. Phase 2 is the heaviest for a fresher, so use the extra fresher week there.
Worked reference: what a "3 Proven" hand-in for Checkpoint 2 looks like
The numbers are illustrative. The shape (a short election timeout detects a dead leader sooner and causes more split votes and false elections) is what you should expect. Hand in your own measurements.
| 3 nodes, 5 ms simulated delay, 20 leader kills each | Timeout 150-300 ms | Timeout 500-1000 ms | Timeout 1500-3000 ms |
|---|---|---|---|
| Time to new leader, median | 0.4 s | 1.1 s | 2.6 s |
| Time to new leader, worst | 1.9 s | 2.3 s | 4.4 s |
| Elections with no winner (split vote) | 6 of 20 | 1 of 20 | 0 of 20 |
| Unneeded elections in a quiet 10-minute run | 5 | 0 | 0 |
| Linearizability checker verdict | valid | valid | valid |
ADR-3 excerpt: Use an election timeout of 500 to 1000 ms with randomisation. The short range recovered fastest but started five elections in a quiet run, because a slow heartbeat looked like a dead leader, and each unneeded election pauses writes. The long range never split a vote but took up to 4.4 s to recover, close to the 5 s target. The middle range met the target with margin and caused no unneeded elections. Rejected: the short range, because pausing writes without a failure is worse than a slower recovery here. Rejected: fixed timeouts, because identical timeouts make nodes start elections together. Revisit if the real network delay is above 50 ms, which would move the whole range up.