NVIDIA-NeMo/RL #5176,
stacked on #4508.
Permalinks pin the PR head 981376a; βtodayβ means #4508's head 788133a.
Both paths run the same cut code: _cut_batched
is now a generator that hands each batch out as a PrefixWrite and waits for its keys
(recovery.py:925).
With the pipeline off, checkpoint_cut
answers each one right away on the same thread. With it on,
run_cut
gives the generator to PrefixWritePipeline, whose prepare thread steps it and whose single write thread does the writes.
The figure follows one real inventory from the PR's test fixture: calls c0 (3 new tokens) and c1 (4).
π΄ review comment C5 β prefix_pipeline.py:363 (row 5)
They meet twice. Before the write, both lanes run the same generator (steps 1β2). After encoding, both produce the same bytes (step 3); the PR's parity test checks this. They also end at the same place: the same staging rows in TQ, sealed only after the write is confirmed (step 6). What differs is only who does the work while the write runs:
| Today | This PR (all on) | |
|---|---|---|
| Encoding | per-row tensors, then stacked | whole columns in NumPy |
| Who waits on the write | the cut's own thread | one write thread; the cut thread keeps preparing |
| Who talks to TQ | the vLLM worker process | a separate 1-CPU process on the same node |
| Rows per write | one cut's batch | batches merged across cuts, within row, token and byte limits |
| Calls frozen at once | β€ one batch | up to every admitted cut's batch, until the queue drains |
The PR's benchmark (4 nodes, router replay, 630 outstanding calls per engine; prefix cut + TQ save time, single runs):
| Write path | Snapshot 1 | Snapshot 2 |
|---|---|---|
| Today (#4508) | 52.44 s | 107.60 s |
| + column encoder + pipeline, worker's own client | 31.65 s | 81.20 s |
| + writer process + NumPy packing | 25.24 s | 72.27 s |
Today's path stays the default: it holds at most one batch of calls frozen and uses no extra process. The pipeline is worth its cost: the encoder plus pipeline cut the save time by about a third, far more than run-to-run noise. The writer process adds one more step, from a single run bundled with a packing change the PR itself says did not help on its own.
So the review asks to land the encoder and pipeline first, with one config block next to the
existing rollout_recovery.generation_prefix_batch_*
limits instead of three switches and two copied limits under vllm_cfg. It also asks to cap how far preparation runs
ahead of the writes, so fewer calls stay frozen (row 5), and to move the writer process to its own PR
with repeated runs. Two CI items must be fixed first: the lint check is red, and the new tests are not marked to run in the Gym test lane.
C1 β lint, recovery.py:872C2 β tests in no CI laneC3 β one config blockC5 β frozen callsC8 β split the writer actor
Review comment links work once the review is published.