Roadmaps / Go / The milestone project
Read first
The milestone project
Scenario. A SaaS product tells its customers about events (an invoice paid, a user added) by calling a URL each customer registers. hookd accepts events over HTTP, stores them, and delivers each one to its endpoint with a signed payload, retries and a limit per endpoint. Load is 2,000 events a second across 400 endpoints. Some endpoints are slow and some are down. Targets: enqueue p99 under 50 ms; delivery starts within 2 s of enqueue at p99; at most 5 deliveries in flight per endpoint; one hanging endpoint does not delay the others; RSS at most 256 MB; the goroutine count is flat over a 1-hour run; no event is lost when the process is killed with SIGKILL.
| Phase | Checkpoint | Done when |
|---|---|---|
| 1 | Bounded delivery | A worker pool delivers 100,000 events to 400 fake endpoints, 10% of which hang. A counter in the test shows in-flight deliveries never pass 5 per endpoint. Cancelling the context stops all work within 1 s and the goroutineleak profile reports none. The race detector passes 100 runs. A seeded leak (a goroutine blocked on a channel nobody reads) is caught by the leak profile. |
| 2 | Durable service | Events are stored in Postgres and claimed with leases. With the process killed with SIGKILL 20 times during 10,000 events, none is lost, and a duplicate delivery carries the same idempotency key. net/http has read, write and idle timeouts. On SIGTERM the service drains in-flight work within 10 s. Payloads are signed with an HMAC and a test rejects a tampered body. |
| 3 | Operated service | Traces and metrics show an event from enqueue to delivery. A dashboard has an alert for a delivery-delay objective (99% within 2 s). A load test at 2,000 events a second meets every target, and a ramp finds the saturation point. A drill with one endpoint hanging 30 s leaves the other 399 unaffected. Time-dependent tests use testing/synctest and take under 1 s each. |
| 4 | Shipped | CI runs go vet, golangci-lint, the race detector and govulncheck. A seeded vulnerable dependency fails the build. A release build is reproducible, signed and ships an SBOM. The runbook covers a hanging endpoint, a leak and a database outage. An upgrade memo says which Go version you run and when you move. |
| W | Work version (Staff and Principal) | On a real service: measure a baseline (goroutine count, p99 and RSS over a week), make one change that removes a leak, stall or lost-work case, and measure again. Pass: a target agreed in writing beforehand is met, or the worst stall falls at least 5 times on the same load. The ADR is signed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later. Timebox: six weeks alongside Phases 3 and 4. |
Evidence to hand in. A repository whose README opens with the problem and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option; raw measurement output and a test report; a one-page design doc; a diagram of the event path. Skeletons: Checkpoint 1 is the worker-pool comparison table below with your numbers and the failing seeded-leak run. Checkpoint 2 is the 20-kill table (kill, events sent, events stored, deliveries, duplicates with the same key). Checkpoint 3 is the dashboard screenshot, the load report and the hanging-endpoint drill.
| Level | What you build and hand in |
|---|---|
| Fresher | Core scope: one process, constant limits, the four checkpoints, ADRs 1 to 4. |
| Senior | Adds the load test in CI, a fault suite, a gRPC intake and ADRs 5 to 7. |
| Staff | Adds written decisions with numbers: delivery objective, alert thresholds, runbook, cost of the instances and a Go upgrade policy. |
| Principal | Adds a service template for many teams, a review checklist other teams reuse, and a measured outcome from the work version. |
Do this at work. A solo lab is not promotion evidence. Take a real Go service that has leaked goroutines, stalled or dropped work and run Checkpoint W on it. Start with a baseline over a week. Make one change. Measure again. A promotion committee accepts the real decision and who made it, the named stakeholder who relied on it, before and after numbers, and a result checked again three months later.
Solo version. With no real service, use a named open-source Go service, read how it bounds goroutines and shuts down, and write a one-page review. Ask a peer or mentor to check your measurements. Replay a recorded event file instead of live traffic.
Laptop scale. Everything runs on a laptop against fake endpoints and a Postgres container. 2,000 events a second is a scaled number: report what your laptop reaches and extrapolate in the memo. Nothing needs cloud spend.
Honest sizing. A first worker pool takes an experienced engineer a day. The checkpoint adds per-endpoint limits, cancellation tests, a leak profile and a seeded fault. Phase 2 has the heaviest load for a fresher (leases, signing and shutdown together), so use the extra fresher week there.
Worked reference: what a "3 Proven" hand-in for Checkpoint 1 looks like
The numbers are illustrative. The shape (unbounded goroutines grow with the slow endpoints; a global pool lets one endpoint take every slot) is what you should expect. Hand in your own measurements.
| 100,000 events, 400 endpoints, 10% hung | go per event | Global pool of 64 | Pool of 64 and 5 per endpoint |
|---|---|---|---|
| Peak goroutines | 31,000 | 70 | 70 |
| Peak in flight to one endpoint | 780 | 64 (a hung endpoint fills the pool) | 5 |
| Healthy events delivered in 60 s | 72,000 | 24,000 | 90,000 |
| Peak RSS | 740 MB | 90 MB | 90 MB |
ADR-2 excerpt: Use a pool of 64 workers and a limit of 5 per endpoint, with a bounded queue of 100 per endpoint. Little's law (requests in flight equal arrival rate times time in system) gives, for a healthy endpoint at 5 events a second and 0.2 s a call, 1 delivery in flight. A hung endpoint with a 10 s timeout at the same arrival rate would hold 50, so a limit of 5 protects the pool. Normal load needs about 20 workers across 400 endpoints, so 64 leaves 3 times headroom. Rejected: one goroutine per event, because memory grows with the slowest endpoints. Rejected: a global limit alone, because one endpoint can take every slot. Revisit if the timeout or the event rate changes.