Roadmaps / AI / The milestone project
Read first
The milestone project
Scenario. A software company wants an assistant that answers support questions from 20,000 help articles and release notes, and can open a ticket or look up an order status through tools. Load is 2,000 questions a day with a peak of 5 a second. Wrong answers cost money, so the assistant must say "I do not know" when the documents do not cover the question. Targets, measured on an evaluation set you write (200 questions for the senior scope, 100 for the core scope, including 20 percent that the documents cannot answer): at least 85 percent of answerable questions answered correctly; at least 90 percent of unanswerable questions refused; every claim in an answer cites a document passage, with at least 95 percent citation accuracy; p95 latency of 6 s or less; a cost per answer under a budget you set from the provider's current price list; a suite of 50 prompt-injection attacks causes no tool call outside the allow-list and no leak of the system prompt or another user's data.
| Phase | Checkpoint | Done when |
|---|---|---|
| 1 | Measured model | A client calls the model with a retry rule, a timeout and a token cap. A table reports time to first token, tokens per second, total latency and cost per answer for two models or two settings, with your predictions beside the measured values and the spread over 20 runs. A seeded failure (a model call with no timeout that hangs a request) is found and fixed. A cache or a smaller model cuts cost by a stated amount on a repeated question set. |
| 2 | Evaluated | An evaluation set of questions with expected answers and source passages is in the repository. A scorer (exact checks plus a model judge you have checked against 30 human labels) gives a score per question. The report shows the mean and spread over 5 runs. A seeded regression (a prompt change that makes the assistant over-refuse) is caught by the evaluation in CI. |
| 3 | Grounded | Documents are chunked, embedded and searched with a measured recall at 5 on your evaluation set. A table compares vector search, keyword search and a hybrid with reranking. Answers cite passages and a citation checker confirms each cited passage supports the claim. Unanswerable questions are refused at the stated rate. A seeded stale-index bug (an updated document that is still served in its old form) is caught by a test. |
| 4 | Safe and shipped | The assistant can call two tools through a server, with an allow-list, argument validation and a confirmation step for the one action that writes. The 50-attack injection suite runs in CI with the stated pass rule. Per-user rate and cost caps work. Traces show each step, its tokens and its cost. A memo says what you would change for a team of ten and how you would handle a model upgrade. |
| W | Work version (Staff and Principal) | On a real AI feature: build an evaluation set of at least 50 real or realistic cases, score the feature, fix the worst failure class and record the result. Pass: a target agreed in writing beforehand is met, or the failure rate on the chosen class falls at least by half on the same set. The ADR is signed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later. Timebox: six weeks alongside Phases 3 and 4. |
Evidence to hand in. A repository whose README opens with the problem and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option and recording the model name, version and date; the evaluation set, its scorer and raw score files; a one-page design doc; a diagram of a question flowing through retrieval, model and tools. Skeletons: Checkpoint 1 is the latency and cost table with your numbers. Checkpoint 2 is the score report with spread and the failing regression run. Checkpoint 3 is the retrieval comparison table. Checkpoint 4 is the injection suite report and a trace of one answer.
| Level | What you build and hand in |
|---|---|
| Fresher | Core scope: 2,000 documents, one model, a 100-question set, the four checkpoints, ADRs 1 to 4. |
| Senior | Adds 20,000 documents, reranking, a tool server, a 200-question set, a benchmark in CI and ADRs 5 to 7. |
| Staff | Adds written decisions with numbers: an evaluation policy, a cost and safety policy and a plan for upgrading the model. |
| Principal | Adds a decision on an AI platform for several teams, with a measured outcome from the work version and a checklist other teams reuse. |
Do this at work. A solo lab is not promotion evidence. Take a real AI feature and run Checkpoint W on it. Start with a baseline over a month: the failure classes you can name from user reports, the cost per request and the latency at p95. Make one change. Measure again on the same set. A promotion committee accepts the real decision and who made it, the named stakeholder who relied on it, before and after numbers, and a result checked again three months later.
Solo version. With no real feature, use a public documentation set (for example the documentation of an open-source project) and write your own questions from it. Ask a peer or mentor to check 30 of your labels. Generate attacks from a public list of injection patterns plus your own.
Laptop scale. 2,000 documents and a small local model run on a laptop. 20,000 documents work with an embedding model run locally if you accept a slower index build. If you use a hosted model, the full evaluation of 200 questions over 5 runs costs a few dollars at typical prices; check the current price list and cap it.
Honest sizing. A basic question-answering demo takes an afternoon. The checkpoints add an evaluation set, a judge you have validated, retrieval comparisons, an injection suite and cost control, which is where the hours go. Writing and labelling the evaluation set alone takes about 8 hours, so it sits in Week 5. Phase 2 is the heaviest for a fresher, so use the extra fresher week there.
Worked reference: what a "3 Proven" hand-in for Checkpoint 3 looks like
The numbers are illustrative. The shape (hybrid search with reranking beats either alone, and the gain shows up in refusals as well as answers) is what you should expect. Hand in your own measurements.
| 200 questions, 5 runs, mean | Vector only | Keyword only | Hybrid, no rerank | Hybrid with rerank |
|---|---|---|---|---|
| Recall at 5 | 0.71 | 0.64 | 0.82 | 0.89 |
| Correct on answerable | 0.74 | 0.68 | 0.81 | 0.87 |
| Refused on unanswerable | 0.77 | 0.70 | 0.84 | 0.92 |
| p95 latency | 3.1 s | 2.8 s | 3.4 s | 4.6 s |
| Spread (standard deviation) | 0.03 | 0.03 | 0.02 | 0.02 |
ADR-6 excerpt: Use hybrid search with a reranker on the top 30 results. Vector search alone missed questions that used exact product names and error codes, and keyword search alone missed paraphrases. The hybrid raised recall at 5 from 0.71 to 0.82, and the reranker lifted correct answers to 0.87 and refusals to 0.92, at a cost of 1.2 s more p95 latency, which stays under the 6 s target. Rejected: vector only, because it fails the exact-code questions that support asks most. Rejected: a larger model with no reranker, because it cost 4 times more per answer for a smaller gain in the same test. Revisit when the document set doubles or the reranker model changes.