knaxuslearn · build · ship

Roadmaps / AI

Data, AI and web · 16 study weeks · 4 phases

AI Roadmap Beta

A demo that works once is easy. The job is a system that keeps working when the model is wrong, slow or under attack. These 16 study weeks teach how to serve and budget a model, how to evaluate a system that gives a different answer each time, how to ground answers in your own documents and how to build agents and tool servers that survive failure and abuse. You build one system, helpdesk, a support assistant that answers from a document set and can take a few safe actions, and every phase adds a checkpoint to it. At 8 hours a week the core path is 128 hours and takes 16 weeks. At 12 hours a week it takes about 11 weeks, and at 16 hours a week about 8 weeks. At the end you can show, with an evaluation set and numbers, how often your system is right, what it costs and what it refuses to do.

This roadmap is in beta and keeps improving over time. If you spot an error, please email your suggestions and corrections to [email protected].

Read first

  1. Phase 1 · Study weeks 1-4

    Models, serving and cost

    This phase makes the first checkpoint possible: a model client you can measure, cap and make cheaper.

    • 5 repos
    • Milestone checkpoint
    1. Study week 1Tokens and context windows8 h
    2. Study week 2Latency anatomy: time to first token and tokens per second8 h
    3. Study week 3Cost: input and output tokens, caching and batching8 h
    4. Study week 4Serving open-weight models: quantisation, batching and the KV cache8 h
  2. Phase 2 · Study weeks 5-8

    Evaluation

    This phase makes the second checkpoint possible: a way to say whether a change made the system better, backed by an evaluation set and a judge you have checked.

    • 3 repos
    • Milestone checkpoint
    1. Study week 5Evaluation sets: sources, coverage and labels8 h
    2. Study week 6Scorers: exact checks, rubrics and model judges8 h
    3. Study week 7Variance: temperature, seeds and sampling8 h
    4. Study week 8Regression tests in CI8 h
  3. Phase 3 · Study weeks 9-12

    Retrieval and grounding

    This phase makes the third checkpoint possible: answers that cite their sources, and refusals when the sources are silent.

    • 3 repos
    • Milestone checkpoint
    1. Study week 9Chunking: size, overlap and structure8 h
    2. Study week 10Keyword search and hybrid search8 h
    3. Study week 11Grounded answers and citations8 h
    4. Study week 12Freshness: updates, deletions and re-indexing8 h
  4. Phase 4 · Study weeks 13-16

    Agents, tools and safety

    This phase makes the last checkpoint possible: an assistant that uses tools safely, resists injection and can be run and changed by a team.

    • 5 repos
    • Milestone checkpoint
    1. Study week 13Tool calling8 h
    2. Study week 14Tool servers with the Model Context Protocol8 h
    3. Study week 15Prompt injection: direct and indirect8 h
    4. Study week 16Tracing and observability8 h

Also in this roadmap