Phase 1 of 4 · Study weeks 1-4
Models, serving and cost
This phase makes the first checkpoint possible: a model client you can measure, cap and make cheaper.
Topics every week
- Week 18 h
- Tokens and context windows
- Sampling and temperature
- What a model can and cannot do
- Structured output
Hand inA client that returns a validated JSON answer, with a test for bad output
- Week 28 h
- Latency anatomy: time to first token and tokens per second
- Streaming
- Timeouts, retries and backoff
- Rate limits
Hand inThe latency table with predictions and the seeded no-timeout hang fixed
- Week 38 h
- Cost: input and output tokens, caching and batching
- Smaller models and routing
- Per-user budgets
- Spend limits that really cap spend
Hand inA cost per answer table and a per-run token cap in code
- Week 48 h
- Serving open-weight models: quantisation, batching and the KV cache
- Local serving with an inference server
- Predicted against measured
Hand inCheckpoint 1: the latency and cost table and the cache saving
Practice projects
- Practice 1, the structured client (Week 1): make the model return a fixed JSON shape. Done when: a test sends 50 prompts and the client rejects and retries every malformed reply, and you report the failure rate.
- Practice 2, the timeout (Week 2): run the client against a slow stub that hangs. Done when: the request fails within the timeout, retries with backoff and a test proves it.
- Practice 3, the cost cut (Week 3): add a cache for repeated questions and route easy ones to a smaller model. Done when: a table shows cost and quality (on 30 labelled questions) before and after.
- Optional, local serving: serve a small open-weight model locally. Done when: a table compares its latency and cost per answer with the hosted model.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months. Look at the README for archive notices before you rely on one.
| Repo | What to read | Week |
|---|---|---|
| vllm-project/vllm | The scheduler and the paged KV cache design | 4 |
| ggml-org/llama.cpp | Quantisation formats and the local server | 4 |
| sgl-project/sglang | Prefix caching and the serving runtime | 4 |
| BerriAI/litellm | A gateway that unifies providers, with budgets and fallbacks | 3 |
| Dao-AILab/flash-attention | Why attention is memory-bound and how the kernel fixes it | 4 |
Milestone checkpoint
Build Checkpoint 1. Write your latency and cost predictions first. Build the client, run 20 calls for each of two models or settings and fill the table. Seed the missing-timeout hang, find it and fix it. Add the cache and the router and measure the cost change. Hand in the table with your own numbers, the tests, ADR-1 (the model choice, with name, version and date) and ADR-2 (the retry, timeout and token-cap rules).
Check yourself
Answer each question in your head first, then open it to check.
What makes up the latency of one answer, and which part do you control?
Network time, time to first token (mostly processing the input) and then generation time per output token. You control the length of the input and output, the model size and streaming.Why do output tokens usually cost more than input tokens?
Each output token needs a full pass through the model one after another, while input tokens are processed in parallel.What does a cache save, and when is it wrong to cache?
It saves the cost and time of repeated identical or near-identical requests. It is wrong when the answer depends on fresh data, on the user or on anything that changes.Why can a spending limit that only alerts still let you overspend?
An alert tells you after the money is spent. A hard cap stops requests, so check which kind your provider offers and add your own cap in code.What does the KV cache store, and why does it limit batch size?
It stores the attention keys and values of every token so far, so each new token is cheaper. It uses memory per sequence, so long contexts and many concurrent requests exhaust it.