knaxuslearn · build · ship

Roadmaps / Linux / Phase 4

Phase 4 of 4 · Study weeks 13-16

Diagnose, operate and hand over

This phase makes you fast and systematic when orders-api breaks, and able to change the host without losing orders. Read Brendan Gregg's Systems Performance (2nd edition, 2020; not checked for a newer edition) chapters on methodology, CPUs and file systems, and the perf and bpftrace manual pages.

Topics every week

  1. Week 138 h
    • Profiling: perf stat, perf record and flame graphs, strace -c -T (VM note: hardware counters may be missing, so fall back to the cpu-clock event)

    Hand inA flame graph of the FAULT=spin build with the hot function named (for Python 3.12 or later run the service by hand with python3 -X perf, outside the unit, because PrivateTmp hides the /tmp/perf-PID.map file that perf needs to name frames), and a strace summary of a slow request

  2. Week 148 h
    • eBPF tools: bcc and bpftrace one-liners (on Debian 13 the bcc tools are *-bpfcc and bpftrace is 0.23.x), BTF and kernel requirements, overhead

    Hand inThree measurements (run-queue latency, block I/O latency, TCP connect latency) with the overhead you measured

  3. Week 158 h
    • Operating: apt upgrades and pins, snapshot.debian.org for older versions, kernel updates and reboots, backups and restores, sysctl discipline, journal retention

    Hand inAn upgrade with rollback, and a restore from backup with the recovery time measured

  4. Week 168 h
    • The incident game, the runbook and the postmortem

    Hand inCheckpoint 4, built in four steps: the fault script; six diagnoses; the runbook; the restore and rollback captures

Practice projects

  1. Diagnose each of two faults using only perf, strace and /proc (no eBPF). Done when each write-up states the hot function or syscall and the evidence line.
  2. Write a USE check script that prints utilisation, saturation and errors for CPU, memory, disk and network on one page. Done when it flags each of your seeded faults and writes a one-minute time series of NRestarts, the oom_kill count and CPU and memory pressure (PSI) to a file.
  3. (Optional for a fresher) Write one bpftrace program of your own for a question about orders-api. Done when its output answers the question and its overhead is stated.

GitHub repos to study

RepoWhat to readWeek
brendangregg/FlameGraphflamegraph.pl and stackcollapse-perf.pl (stable scripts, little recent activity)13
iovisor/bcctools/*_example.txt files for output you can compare with yours14
brendangregg/perf-toolsRead for ideas only; unmaintained since 202013
bpftrace/bpftraceThe one-liner tutorial under docs/14

Milestone checkpoint

Run the incident game, write the runbook, do the restore and one rollback. Done when the Checkpoint 4 row passes. Evidence: six write-ups (cause, evidence, fix, prevention), the runbook, the restore log with the recovery time, the rollback log, ADR-4 (backup method and what is not covered, for example "a snapshot taken during a write is crash-consistent, not application-consistent") and ADRs 5 and 6 for Senior and above.

Check yourself

Answer each question in your head first, then open it to check.

  1. What is the first question in the USE method for each resource?
    Utilisation, then saturation, then errors, for each resource.
  2. A host shows 90% CPU. How do you tell a hot loop from a stalled lock?
    Sample on-CPU stacks (perf record -g): a hot loop shows one function dominating. perf samples only time on the CPU, so a lock stall shows little CPU at all; confirm it with off-CPU evidence (strace -f -T showing long futex waits, /proc/PID/task/*/wchan, or a bcc offcputime run).
  3. Why can perf show no hardware events inside a VM?
    The hypervisor may not expose the performance counters, so only software events such as cpu-clock work.
  4. What makes a backup you have never restored unsafe to trust?
    It may be incomplete, corrupt or missing a file; only a restore test proves the recovery and gives a recovery time.
  5. After a kernel update, what must you check before the reboot?
    That the bootloader entry and initramfs for the new kernel exist and the old kernel is still selectable, so you have a way back.