Research-1366: Frame read-ahead in the vmaf CLI¶

• Status: Active
• Workstream: ADR-1366 (RC3 performance)
• Last updated: 2026-09-29

Question¶

At 3840x2160 every backend showed a per-frame floor of about 7 ms that did not depend on the feature: on an i9-12900K with an Arc B580, psnr, float_psnr, motion, adm and vif on the B580 all measured 6.5 to 8.5 ms per frame, and the CPU twins on 16 threads sat on the same floor whenever their extraction was cheaper. Where does the floor come from, can it overlap scoring, and what must stay unchanged when it does?

Sources¶

• core/tools/vmaf.cpp on master 2d9d5b069: run_frame_loop(), fetch_picture(), copy_picture_data(), classify_frame_fetch(), release_unpaired_pictures(), skip_initial_frames(), preallocate_cli_pictures(), open_cli_inputs().
• core/tools/vidinput.c, y4m_input.c, yuv_input.c: the per-stream reader state (video_input owns its FILE * and its context; no shared statics).
• core/src/picture_pool.cpp: vmaf_picture_pool_fetch() blocks on a condition variable until a picture is released, and pooled_picture_release() signals it; vmaf_picture_pool_close() waits for every picture.
• core/src/libvmaf.c: vmaf_read_pictures(), threaded_read_pictures_batch() (a worker job holds the reference, the distorted picture and a counted snapshot of the previous reference), read_pictures_update_prev_ref() (the context keeps one previous reference between calls), dispatch_gpu_double_buffer().
• ADR-0104, ADR-1262, ADR-0520.

Findings¶

The floor is the read, on the scoring thread, one stream after the other. run_frame_loop() called fetch_picture() for the reference, then for the distorted stream, then vmaf_read_pictures(). Each fetch takes a pool picture, freads the frame into the reader's buffer and copies it plane by plane into the picture: 12.4 MB twice per 4K 8-bit 4:2:0 frame. With --feature psnr on 16 threads master spends about 7 ms per frame, which is the cost of those two reads: the psnr work itself runs on the workers.

The two streams can be read concurrently. Each video_input owns its FILE * and parse state; the Y4M and raw readers keep no mutable statics. The picture pool is thread-safe. Only vmaf_read_pictures() has an ordering contract (pairs in stream order, consecutive indices, one caller), so the reads can move to other threads as long as that call stays where it is.

Pool sizing decides whether a reader can deadlock. Once its workers are idle, libvmaf holds only the previous reference (plus, for a GPU extractor with submit/collect, the reference it keeps for its pending frame). The master pool is 2 * (threads + 1) + 1. A reader that reserves a ring slot before it takes a picture holds at most kReadaheadDepth pictures, so adding 2 * kReadaheadDepth to the pool keeps every picture libvmaf could hold before and leaves at least one picture for a reader whenever the scorer waits on it. A reader blocked on the pool while the other reader's ring is full is woken by the scorer consuming that ring, or, at shutdown, by request_stop() draining it; joining one reader before draining the other could wait forever.

Two handles on one object must not be read in parallel. Separately opened regular files have separate offsets, but one pipe opened twice (for example /dev/stdin on both sides) hands each read whichever bytes come next. The serial loop gave the reference the even frames and the distorted stream the odd ones; two threads would make that assignment depend on timing. fstat() device and inode identify this on POSIX. --no-reference opens the distorted input twice (ADR-0520), which the same test sends to the inline path. _fstat64 on Windows reports no inode, so there only two regular files are read in parallel.

What the readers change on stderr. Scoring, classify_frame_fetch() and the progress line stay on the main thread, so their output is unchanged. The input readers print their own diagnostics (Error reading YUV frame data.), and a reader running ahead can hit a damaged frame past the point where the other stream ended; the message then appears on a run that exits 0. On a 4K --threads 16 default-model run the est_params: covariance matrix was singular on N of M solves line differs between runs of master itself (12 or 16), because it sums per-thread SpEED contexts; it is not caused by read-ahead.

Bit-exactness¶

JSON reports compared after removing the wall-clock fps field, master 2d9d5b069 against the branch, same host and binaries' libvmaf.

Stderr was byte-identical in every CPU case except the SpEED line above.

Timing¶

Host: i9-12900K, Arc B580, Docker Desktop WSL2 VM with 22 logical CPUs, shared with other agents' builds (load average 6 to 14 during the runs). Input: BBB 3840x2160 8-bit 4:2:0, page-cached. Master and branch ran back to back in each repetition; the tables give the median and the min..max spread in ms per frame.

In-loop, 61 frames, 5 repetitions, two sessions. The CLI runs under a pseudo-terminal and the last FPS on its progress line is read: that value is frames divided by the wall time since the frame loop started, so device init, model load and pool allocation are excluded. GPU runs take flock /f/gpu.lock for the whole run. Session A ran at load average 6 to 14. Session B ran the CPU rows at load 3 to 9 and the GPU rows after a Docker Desktop restart had stopped the other containers (load 0.5 rising to 7.6).

In session B the branch was faster than master in every repetition of every row except one repetition of the SYCL default model. The --feature names on --backend sycl run the SYCL twins (ADR-1359). With the default model the JSON feature_backends lists cambi_sycl, speed_chroma_sycl and motion_sycl on the device and adm on the CPU (the model's ADM options keep the CPU extractor), so that row is limited by serial CPU ADM, not by reading.

Session C (load average 0.8 to 1.6), same method; the UHD 770 is the i9-12900K's integrated GPU (level_zero:1):

The integrated GPU shares memory bandwidth with the CPU and computes more slowly, so the read cost is a smaller share of each frame; the branch still saves 3.4 to 4.2 ms per frame there, faster in every repetition.

Startup-difference method, 3 repetitions: wall time of --frame_cnt 22 minus --frame_cnt 2, divided by 20. It agrees on the CPU; on the GPU the device start-up varies by hundreds of milliseconds between runs, which puts several ms of noise on each estimate (some repetitions came out negative), so the in-loop numbers above are the GPU result.

The CPU default model on 16 threads is limited by extraction, not by reading. Its 22-frame estimate is dominated by start-up variance: five more repetitions gave master 24.85 [-13.96..27.86] and branch 27.95 [23.11..32.56], and a 100-frame window (--frame_cnt 102 minus 2) gave master 22.41 [14.30..24.35] and branch 21.28 [20.29..21.94]. With the in-loop method the two are equal within the spread.

With read-ahead the single-feature GPU runs settle at 3.8 to 4.6 ms per frame in session B (vif at 6.8). The two readers are no longer the limit there (each reads one 12.4 MB frame per step in parallel); what remains is on the scoring thread: the host-to-device upload of the pair inside vmaf_read_pictures() and the wait for the device result. This split was not profiled.

Queue depth¶

The same branch built with kReadaheadDepth 1 and 3, each interleaved with depth 2 (in-loop method, 61 frames, 5 repetitions, load average below 2.5):

(Depth 2 was measured once against each variant, hence two entries.) Depth 1 was slower than depth 2 in four of five serial psnr repetitions; depth 3 gained nothing and costs two more pool pictures. Depth 2 it is.

Alternatives explored¶

• One producer thread reading the reference and then the distorted frame. It would keep the serial byte order even for a shared pipe, but the two reads stay serial. Master already spends about 7 ms per 4K frame on them with --feature psnr --threads 16, where the extraction is on the workers, so a single producer could not go below that; two readers reach 3.5 to 4 ms.
• USE_DIRECT_READ (video_input_fetch_into_vmaf_picture()): removes the copy into the picture but keeps the read on the scoring thread, and the Y4M path rejects the formats that need conversion. It is independent of this change and could be combined with it.
• Detaching the readers on an error instead of joining them: a detached reader would still use the picture pool and its video_input after vmaf_close(). Joining costs at most one frame read.
• An opt-out flag: outputs do not depend on the path, so it would add a CLI surface without a determinism reason.

Open questions¶

• The upload inside vmaf_read_pictures() is now the GPU floor. Overlapping it with the previous frame's kernels needs double-buffered device pictures in the backend, not a CLI change.
• USE_DIRECT_READ on top of read-ahead would remove one copy per frame on the reader threads; not measured.

Related¶

• ADR-1366, this change.
• ADR-0104, picture pool.
• ADR-1262, exit codes the readers must preserve.
• core/tools/test/test_vmaf_frame_readahead.sh.