Roadmaps / BeagleBone / The milestone project
Read first
The milestone project
Scenario. A single-axis positioning stage in a small production cell: it moves a part under a camera, one move per job. Linux runs the job queue, a command-line tool, logging and updates. One PRU generates step pulses at up to 20 kHz (a 50 µs period; 375 rpm for a 200-step motor at 1/16 microstepping) with trapezoidal speed ramps. The other PRU watches a limit switch and a heartbeat from Linux. The cell runs 8 hours a day, about 480 jobs a day (each job a short move; the checkpoint runs one long 10-minute move as a stress test), and loses power about once a week. A lost or extra step scraps the part, so the target is zero lost steps and a step period within 1 µs of the plan, even when Linux is busy.
| Phase | Checkpoint | Done when |
|---|---|---|
| 1 | Baseline harness | A pin toggles every 500 µs (a 1 kHz square wave) and the interval is the time between consecutive edges. A report gives the sample count, p50, p99 and max interval in three conditions (idle, loaded stock kernel, loaded RT kernel). Each condition has three runs of 10 minutes, nine runs in all (about 1.2 million edges each), on the v6.12 kernel line with the CPU governor set to performance ("idle" means the same kernel as "loaded stock"). The toggler runs SCHED_FIFO with locked memory and absolute-time sleeps on both kernels, with one extra SCHED_OTHER row, and the report records uname -a. The script flags any interval over 1 ms. A seeded 5 ms stall is flagged by the script. |
| 2 | Hardware description | An overlay plus a driver (or a libgpiod service) exposes enable, direction and limit_hit. dtc builds with no warnings. For a kernel driver: 1,000 bind and unbind cycles give no kernel warnings, and unbinding while the device is open does not crash the kernel (seeded fault). For a libgpiod service: 1,000 restarts, and kill -9 while a client is connected, leave the pins in a known state. A second program drives a test pin, wired to the limit_hit input, 10,000 times at 100 Hz (about 100 s) with the load running. The event reaches user space in under 5 ms every time, measured against the test program's own CLOCK_MONOTONIC write time (use the RT kernel or a SCHED_FIFO thread if the stock kernel misses); report p99 too. The service acts on the first edge and then ignores bounce for a set time. |
| 3 | PRU step engine | PRU0 runs a 10-minute move at 20 kHz (12 million steps). The step period stays within 1 µs of plan (checked by PRU1 and by the logic analyzer), and the count captured by PRU1 equals the count commanded. PRU1 keeps a count, the minimum and maximum period and a histogram, not every timestamp. Both hold while the Checkpoint 1 load runs, with the supervisor at SCHED_FIFO on the RT kernel and zero false safe-state trips in the run. Stopping the supervisor with kill -STOP makes stepping enter the safe state within 10 ms. A test pulse on the limit input during a move stops PRU0 within 1 ms. The logic analyzer is used in windowed captures (for example 10 s at the start, middle and end), while PRU1 covers the full run. A seeded out-of-order write to PRU shared RAM is caught by the sequence-number check. The fresher runs at constant rate; senior and above add the ramp. PRU1's 32-bit cycle counter wraps after about 21.5 s, so handle the wrap, and state PRU1's own timestamp error with its RPMsg path included. |
| 4 | Shippable image | A 24-hour soak (a job file that repeats a 10-second move every minute) shows zero lost steps (fresher: 8 hours). 20 power cuts in the middle of an update always end on a bootable slot (fresher: 5). A seeded bad update (the health check fails) rolls back by itself. The build is reproducible: two clean builds from the same checkout and pinned inputs produce root filesystems with identical file lists and checksums, apart from timestamps you list. |
| W | Work version (Staff and Principal) | On a real product: a Checkpoint 1 style baseline, one change that moves the deadline-critical part, and a second measurement. Pass: a miss budget agreed in writing before the change is met, or missed deadlines fall at least 10 times on the same load. The ADR is signed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later (it falls outside the 16 weeks). Level 4 also needs evidence that another team used your checklist and recorded a finding or decision with it. Timebox: six weeks alongside Phases 3 and 4. |
Terms. The supervisor is the Linux program that sends moves and a heartbeat message every 1 ms over RPMsg. Checkpoints 3 and 4 run it at SCHED_FIFO with locked memory on the RT kernel: the stock kernel's loaded worst case (6.4 ms in the reference below) would trip the timeout. PRU1 handles RPMsg and passes moves to PRU0 through shared RAM; PRU0 never touches RPMsg or DDR inside the step loop. A lost step is a commanded step with no edge on the step pin. Loopback proves that pulses were emitted, not that a motor moved: position needs an encoder or a limit-switch homing run, so the stage homes after every power-up. Safe state: if PRU1 sees no heartbeat for the timeout, PRU0 stops stepping at once, raises a "position uncertain" flag, keeps enable asserted (holding torque) and waits for a homing command. The stop is abrupt because a loaded NEMA 17 cannot follow a ramp-down inside a few milliseconds at 20 kHz; a stage that must keep its position needs a longer timeout and a ramp-down, and ADR-3 records its deceleration limit. The timeout starts at 5 ms and is at least twice the largest heartbeat delay you measured in Week 10. The pass criterion is the time from the last heartbeat to the last step edge, at most the timeout plus 1 ms and never over 10 ms, with PRU1's step count at standstill equal to PRU0's own. To seed the faults: stop the supervisor with kill -STOP, have a test PRU firmware write a record with an old sequence number into the shared RAM, and stall a thread for 5 ms.
Who owns each signal. In Phase 2 Linux owns enable, direction and limit_hit through your overlay and driver. In Phase 3 a second overlay hands step, enable and direction to PRU0, PRU1 reads limit_hit, and Linux reads status over RPMsg. Each signal has one owner at a time.
Evidence to hand in. A repository whose README opens with the problem and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option; raw captures and a test report; a one-page design doc; a block diagram and a timing diagram. Skeletons: Checkpoint 1 is one table (condition, run, count, p50, p99, max, intervals over 1 ms) plus the raw files and the flagged seeded stall. Checkpoint 2 is the specification page, dtc output, the 10,000-event table and the bind or restart log. Checkpoint 4 is a soak report (steps commanded against counted, per hour), a 20-row power-cut table (cut point, slot booted, recovered) and the rollback log.
| Level | What you build and hand in |
|---|---|
| Fresher | Core scope: loopback wiring, a constant step rate (the ramp is optional), a 1 µs budget, four checkpoints, ADRs 1 to 5. |
| Senior | Adds trapezoid ramps in fixed-point maths, jitter histograms, a fault-injection suite run from CI, a budget table of PRU cycles, and ADRs 6 and 7. |
| Staff | Adds written decisions with numbers: PRU against RT thread against microcontroller, update strategy, operability (logs, metrics, field diagnostics), and a cost and supply memo for the AM335x. |
| Principal | Adds a platform decision memo for a product line, a review checklist for "real time on Linux" designs that other teams reuse, and a measured outcome from the work version (Checkpoint W). |
Do this at work. A solo lab is not promotion evidence. Take a real product that bit-bangs a signal, polls in a hot loop, or misses deadlines under load, and run Checkpoint W on it. Start with Checkpoint 1 style measurement: measure a baseline with cyclictest and a loopback capture. Then move the deadline-critical part to the right place (an RT thread, a PRU, or a microcontroller) and measure again. A promotion committee accepts: the real decision and who made it, the named stakeholder who relied on it (for example the product owner or hardware lead), before and after numbers on missed deadlines, and a result checked again three months later. Write the decision as an ADR the stakeholder signed, and put the three-month recheck in the calendar.
Solo version. If you have no real product or stakeholder, use a named open-source codebase with a real-time path instead: pick one, write a one-page review of how it meets its deadlines, and ask a peer or mentor to review your measurements. Replay a recorded job file in place of live production traffic.
Laptop scale. Everything builds on a laptop or the board, loopback needs a jumper wire, and nothing needs cloud spend.
Honest sizing. Phase 3 gets four weeks (32 hours) for one checkpoint, and an experienced embedded engineer may finish a first PRU step generator in a few days. A fresher should plan six weeks for each of Phases 3 and 4 (see the time model), take the reduced soak and power-cut counts in the Checkpoint 4 row, and skip the optional items.
Worked reference: what a "3 Proven" hand-in for Checkpoint 3 looks like
The numbers below are illustrative. The shape (Linux's worst case grows under load, the PRU's does not) is what you should expect, but you must hand in your own captures. Do not copy these figures.
| Step period 50 µs, 10 minutes per column | Stock kernel, user-space toggle | RT kernel, SCHED_FIFO thread | PRU0 firmware |
|---|---|---|---|
| Idle: max period error | 180 µs | 60 µs | 0.015 µs (3 cycles) |
Loaded (stress-ng plus eMMC writes): max period error | 6,400 µs | 210 µs | 0.015 µs |
| Missed steps in 10 minutes, loaded | 4,812 | 3 | 0 |
ADR-3 excerpt: Step generation runs on PRU0, not on an RT thread. The budget is 1 µs: 2% of the 50 µs period, the speed ripple this stage is assumed to tolerate. If the stage tolerated 100 µs, the RT thread would pass and this ADR would choose it. An RT thread missed it by about 200 times under load, so it fails the budget. A stepper motion controller IC with its own ramp generator (a TMC5160 class part) removes the timing problem for one axis, but adds a part, an SPI link and a driver to certify. An external microcontroller adds a second firmware to ship and update. The eHRPWM can make a pulse train, but counting exactly N pulses under a changing rate needs software in the loop, which brings the latency problem back. Consequences: 8 KB of PRU program memory, a minimal C runtime, a compiler chain whose TI release dates from 2019 (so keep gnupru as the second option), and a link to the AM335x (exit plan in ADR-7). Revisit if the step rate passes the budget you measured, or the AM335x supply changes.