Roadmaps / Java / The milestone project
Read first
The milestone project
Scenario. A shop's checkout calls orders-service to place an order. The service writes the order to Postgres and publishes an OrderPlaced event to Kafka for billing and shipping. Average load is 300 requests a second, peak 900. The container has 2 CPUs and 1 GB. Targets: p99 under 300 ms at 300 requests a second; heap at most 512 MB; GC pause p99 at or under 30 ms; ready to serve within 20 s of start; no lost and no duplicated event after a broker restart; zero failed requests during a rolling deploy.
| Phase | Checkpoint | Done when |
|---|---|---|
| 1 | Measured service | A JMH benchmark, a JFR recording and a GC log exist for the service at 300 requests a second, with your predictions beside the results. A seeded leak (a static map that grows per request) is found from two heap dumps and fixed. A seeded pinned virtual thread (a blocking native call) is found from a thread dump and removed. p99 and GC pause meet the targets. |
| 2 | Data and events | The modules are separated and a Spring Modulith test fails on a cross-module call. A schema change ships in two steps (expand, then contract) with no downtime in a test. Events go through an outbox table. With the broker killed 20 times during 10,000 orders, no order is lost and a duplicate delivery is ignored by an idempotent consumer. A slow database (500 ms added) trips a timeout, not a pile-up. |
| 3 | Operated service | The service runs in kind with probes and resource limits, and a rolling deploy at 300 requests a second drops zero requests. Metrics and traces show every request. A service level objective (99.5% of orders under 300 ms) has an alert. A k6 run at 900 requests a second finds the saturation point, and the service sheds load past it with a 429 (too many requests) instead of failing. |
| 4 | Changed and shipped | The service moves from the long-term support JDK to the latest feature JDK with a parallel run and a written compare of latency and memory. Architecture rules run as tests. The build produces a software bill of materials (SBOM), and a seeded vulnerable dependency is caught in CI. The runbook covers a stall, a leak and a broker outage. An upgrade policy memo says which JDK you run and when you move. |
| W | Work version (Staff and Principal) | On a real service: measure a baseline (p99, GC pause and heap over a week), make one change that removes a stall, leak or failure mode, and measure again. Pass: a target agreed in writing beforehand is met, or the worst stall falls at least 5 times on the same load. The ADR is signed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later. Timebox: six weeks alongside Phases 3 and 4. |
Evidence to hand in. A repository whose README opens with the problem and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option; raw measurement output and a test report; a one-page design doc; a diagram of the order path. Skeletons: Checkpoint 1 is one table (run, load, p99, GC pause p99, heap) plus the two seeded-fault runs. Checkpoint 2 is the 20-kill table (kill, orders sent, orders stored, events delivered, duplicates ignored). Checkpoint 4 is a before and after table for the JDK move.
| Level | What you build and hand in |
|---|---|
| Fresher | Core scope: one service, constant limits, the four checkpoints, ADRs 1 to 4. |
| Senior | Adds the load test in CI, a seeded-fault suite and ADRs 5 to 7. |
| Staff | Adds written decisions with numbers: service level objective, alert thresholds, cost of the instances and an upgrade policy. |
| Principal | Adds a platform standard for many services, a review checklist other teams reuse, and a measured outcome from the work version. |
Do this at work. A solo lab is not promotion evidence. Take a real Java service that has stalled, leaked or failed under load and run Checkpoint W on it. Start with a baseline over a week. Make one change. Measure again. A promotion committee accepts the real decision and who made it, the named stakeholder who relied on it, before and after numbers, and a result checked again three months later.
Solo version. With no real service, use a named open-source Spring application, read its code and write a one-page review of how it bounds threads, memory and retries. Ask a peer or mentor to check your measurements. Replay a recorded request file instead of live traffic.
Laptop scale. Everything runs on a laptop in containers. Peak 900 requests a second on a laptop is a scaled number: report what your laptop reaches and extrapolate in the memo. A cloud run is optional and its cost goes in the memo.
Honest sizing. A service that works takes an experienced engineer a few days. The checkpoints add seeded faults, measurement and a rolling-deploy test, which is where the hours go. Phase 2 is the heaviest for a fresher (modules, schema change and events together), so use the extra fresher week there.
Worked reference: what a "3 Proven" hand-in for Checkpoint 1 looks like
The numbers are illustrative. The shape (a blocking call that pins a carrier thread caps throughput far below the offered load) is what you should expect. Hand in your own measurements.
| 300 requests a second, 40 ms database call | 200 platform threads | Virtual threads, blocking call pins | Virtual threads, pin removed |
|---|---|---|---|
| Threads or carriers in use at peak | 200 of 200 | 2 of 2 carriers stuck | 2 carriers, 14 mounted |
| p99 | 260 ms | 4,100 ms | 110 ms |
| Heap after 10 minutes | 410 MB | 405 MB | 395 MB |
| GC pause p99 | 18 ms | 17 ms | 17 ms |
ADR-2 excerpt: Use virtual threads with a limit of 64 database connections. Little's law (requests in flight equal arrival rate times time in system) gives 300 x 0.04 = 12 database calls in flight at average load and 36 at the 900 peak. A pool of 64 leaves headroom of 1.8 times at peak. Platform threads would also work at this load, but the plan expects the service to grow and virtual threads keep the code simple. Rejected: an unbounded virtual-thread fan-out, because the database, not the JVM, is the limit. Revisit if the database call grows past 100 ms.