knaxuslearn · build · ship

Roadmaps / Go / Phase 3

Phase 3 of 4 · Study weeks 9-12

Operating the service

This phase makes the third checkpoint possible: a service you can see into, load test and break on purpose.

Topics every week

  1. Week 98 h
    • OpenTelemetry traces and context propagation
    • Prometheus metrics and labels
    • Cardinality

    Hand inA trace from enqueue to delivery and a dashboard

  2. Week 108 h
    • Load testing
    • The saturation point
    • Coordinated omission
    • Service level objectives and burn-rate alerts

    Hand inA load report with the saturation point and the delivery-delay alert

  3. Week 118 h
    • Failure drills: a hanging endpoint, a slow database, a full disk
    • Backpressure and load shedding
    • Runbooks

    Hand inDrill results and a first runbook

  4. Week 128 h
    • Deterministic tests with testing/synctest
    • Property tests
    • Benchmarks with testing.B
    • Profiling in production safely

    Hand inCheckpoint 3, with predicted and measured numbers side by side

Practice projects

  • Practice 1, the dashboard (Week 9): show enqueue rate, delivery delay and in-flight per endpoint. Done when: an injected slow endpoint is visible in under 1 minute, and the label set stays under 1,000 series.
  • Practice 2, the burn-rate alert (Week 10): inject 5 s of delivery delay. Done when: the alert fires within its stated window and stays silent on a normal run.
  • Practice 3, the hanging endpoint (Week 11): hang one of 400 endpoints for 30 s. Done when: the other 399 keep their delay and the hung endpoint's queue sheds at its bound.
  • Optional, a production profile: capture a CPU profile from a running service. Done when: the capture adds under 5% overhead and exposes no payload data.

GitHub repos to study

Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.

RepoWhat to readWeek
open-telemetry/opentelemetry-goThe SDK, spans and context propagation9
prometheus/prometheusThe client model: counters, histograms and labels9
golang/gotesting/synctest and how it fakes the clock12

Milestone checkpoint

Run Checkpoint 3. Write your prediction for the saturation point first. Add the traces and metrics, run the load test and the hanging-endpoint drill, and rewrite the time-based tests with synctest. Hand in the dashboard, the alert test, the load report, ADR-5 (what the service sheds and when) and the first draft of the runbook.

Check yourself

Answer each question in your head first, then open it to check.

  1. Why is a high-cardinality label a problem in Prometheus?
    Each label value creates a series, so an unbounded label such as an event id grows memory and query cost without limit.
  2. What does coordinated omission do to a load test?
    A generator that waits for a slow reply stops sending, so the slow period is under-counted. Use a fixed arrival rate.
  3. Why use a burn-rate alert for the delivery objective?
    It tells you how fast you are spending the error budget, so a brief spike and a slow burn page at the right urgency.
  4. What does synctest change about a test that uses timers?
    It runs the test in a bubble with a fake clock, so timers fire when all goroutines are blocked and the test takes no real time.
  5. When is shedding load better than queueing?
    When a queue only adds latency for everyone and still fills. A bounded queue with a clear rejection keeps accepted work inside the objective.