How a generation-prefix cut gets written β€” today, and with this PR's pipeline and writer actor

NVIDIA-NeMo/RL #5176, stacked on #4508. Permalinks pin the PR head 981376a; β€œtoday” means #4508's head 788133a.

Before
One thread does a whole cut: it freezes a batch of calls, encodes their rows one by one, writes them with the worker's own store client, waits, and only then starts the next batch.
This PR
Adds three opt-in pieces: a faster column encoder, a pipeline that prepares the next batch while one write runs (and merges batches from different cuts), and a CPU writer process that does the write.
Status
Proposed β€” off by default, not ready to merge. The review found CI lint red and the 78 new tests scheduled in no CI lane, and asks to reshape the config before merge.
Background. When a rollout checkpoint is saved, Gym asks each vLLM engine for a cut of every unfinished model call it lists in an inventory. Each call's new tokens become one row in TQ. A call is frozen until its row's write is confirmed, and only then does Gym get a receipt. Dotted words have a short explanation: hover or tap them.

Both paths run the same cut code: _cut_batched is now a generator that hands each batch out as a PrefixWrite and waits for its keys (recovery.py:925). With the pipeline off, checkpoint_cut answers each one right away on the same thread. With it on, run_cut gives the generator to PrefixWritePipeline, whose prepare thread steps it and whose single write thread does the writes. The figure follows one real inventory from the PR's test fixture: calls c0 (3 new tokens) and c1 (4).

Same cut, two ways to write it

Today (#4508) This PR, all three pieces on same in both differs by design raised in review 1 Β· Gym asks inventory "checkpoint-1": calls [c0, c1] β†’ run_cut() 2 Β· freeze, read new tokens c0: 3 tokens [10, 20, 21] c1: 4 tokens [10, 20, 21, 22] _cut_batched :800 β€” one generator drives both lanes 3 Β· encode one TensorDict per row, then stack _encode_record Γ—2 :851 columns built in NumPy, no row tensors encode_prefix_groups :73 same 24 columns, 624 B Β· token_ids_delta int64, row offsets [0, 3, 7] parity test :53 4 Β· who writes the cut's own thread writes with the worker's TQ client, then waits store.put_many :875 write thread merges rows from several cuts β†’ 24 byte buffers (624 B) β†’ CPU writer process _collect :271 pack_columns :95 actor put :225 the writer is pinned to the engine's node 5 Β· while the write runs nothing else is prepared; only this batch's calls are frozen at most one batch frozen at a time prepare thread freezes the next cuts' calls they stay frozen until every write ahead drains can_prepare :363 β€” capped by bytes only 6 Β· after TQ confirms keys come back β†’ seal both calls β†’ receipt to Gym No receipt before the write is confirmed, in either lane. A failed merged write retries each cut's own batch. Values in rows 1–3 come from the PR's test fixture (_batch_worker_fixture, two calls). Row 4–5 thread roles are read from the code at 981376a.

πŸ”΄ review comment C5 β€” prefix_pipeline.py:363 (row 5)

Where the two paths meet, and what still differs

They meet twice. Before the write, both lanes run the same generator (steps 1–2). After encoding, both produce the same bytes (step 3); the PR's parity test checks this. They also end at the same place: the same staging rows in TQ, sealed only after the write is confirmed (step 6). What differs is only who does the work while the write runs:

TodayThis PR (all on)
Encodingper-row tensors, then stackedwhole columns in NumPy
Who waits on the writethe cut's own threadone write thread; the cut thread keeps preparing
Who talks to TQthe vLLM worker processa separate 1-CPU process on the same node
Rows per writeone cut's batchbatches merged across cuts, within row, token and byte limits
Calls frozen at once≀ one batchup to every admitted cut's batch, until the queue drains

The PR's benchmark (4 nodes, router replay, 630 outstanding calls per engine; prefix cut + TQ save time, single runs):

Write pathSnapshot 1Snapshot 2
Today (#4508)52.44 s107.60 s
+ column encoder + pipeline, worker's own client31.65 s81.20 s
+ writer process + NumPy packing25.24 s72.27 s

Why keep both, and what the review asks

Today's path stays the default: it holds at most one batch of calls frozen and uses no extra process. The pipeline is worth its cost: the encoder plus pipeline cut the save time by about a third, far more than run-to-run noise. The writer process adds one more step, from a single run bundled with a packing change the PR itself says did not help on its own.

So the review asks to land the encoder and pipeline first, with one config block next to the existing rollout_recovery.generation_prefix_batch_* limits instead of three switches and two copied limits under vllm_cfg. It also asks to cap how far preparation runs ahead of the writes, so fewer calls stay frozen (row 5), and to move the writer process to its own PR with repeated runs. Two CI items must be fixed first: the lint check is red, and the new tests are not marked to run in the Gym test lane.

C1 β€” lint, recovery.py:872C2 β€” tests in no CI laneC3 β€” one config blockC5 β€” frozen callsC8 β€” split the writer actor

Review comment links work once the review is published.