Research-0744 — Cross-Backend Parity Baseline (Pre-ncu-opt)¶

Date: 2026-05-28 Branch: research/cuda-cross-backend-baseline-20260528 Purpose: Establish a numeric and throughput baseline for CPU and CUDA (SYCL skipped — no Intel device exposed to one-off container) before perf/cuda-vif-filter1d-ncu-driven and future optimization PRs land. All future performance PRs should cite this digest as the pre-optimization reference point.

Environment¶

Workloads¶

Wall-Time Baseline (3 runs each; all values in ms)¶

Interpretation: CPU is faster than CUDA on all three workloads at these frame counts. CUDA overhead (init + H2D/D2H transfers) dominates at 3–48 frames. Throughput advantage only appears with larger batch sizes; the perf_benchmark_results.json snapshot (BBB 1080p, 48 frames) shows CUDA at ~249 fps vs CPU at ~44 fps — a 5.7× advantage at that batch size. The crossover point is somewhere between 10 and 48 frames for 1080p content.

Score Correctness vs CPU (ADR-0119 tolerance)¶

The ULP values reflect the IEEE-754 bit-level distance between the two float64 values. The delta at WL2 is ~1.1×10⁻⁷ relative error, well within the established "close but never bit-exact" GPU tolerance (memory note: golden gate is CPU-only; GPUs NOT bit-exact). No correctness finding.

Netflix golden gate (WL1): CPU score 76.667830 — the Python assertAlmostEqual(..., places=4) gate expects 76.6678 at 4 decimal places. CUDA 76.667829 also passes at places=4. No regression.

Per-Feature Parity (WL2, CPU vs CUDA)¶

Feature gaps noted: integer_adm3 and integer_aim are absent from CUDA pooled_metrics. These are model-layer post-processing scalars; the CUDA feature extractor computes integer_adm2 plus the four scale components but does not emit the derived adm3 aggregate or the aim (Attention-weighted Inverse Metric) score. The vmaf model JSON references these; the CPU path computes them as a post-processing pass. Whether the CUDA path omits the collection step or the JSON-writer filters them requires a code-path check — outside scope of this baseline.

Existing Snapshot Comparison¶

testdata/perf_benchmark_results.json records a previous BBB 1080p 48-frame run:

The new baseline uses Netflix golden YUV pairs rather than BBB, so scores are not comparable. However the SYCL 93 fps at 1080p (from the snapshot) and CUDA 249 fps at 1080p provide reference throughput numbers that future PRs should beat.

Reproducer¶

# One-off container run (isolates from shared state)
docker run --rm --gpus all \
  --entrypoint /bin/bash \
  -v /home/kilian/dev/vmaf/python/test/resource/yuv:/yuv:ro \
  vmaf-dev-mcp:cuda13.3 \
  -c '/build/vmaf/core/build/tools/vmaf \
      -r /yuv/src01_hrc00_576x324.yuv \
      -d /yuv/src01_hrc01_576x324.yuv \
      -w 576 -h 324 -p 420 -b 8 \
      -m path=/build/vmaf/model/vmaf_v0.6.1.json \
      -o /tmp/out.json --json -q \
      --backend cpu'
# Replace --backend cpu with --backend cuda for CUDA run

Observations and Open Questions¶

1. CUDA slower than CPU for < ~10 frames at 1080p. The crossover depends on frame count, not resolution alone. The perf_benchmark_results.json BBB data shows CUDA wins at 48 frames. Budget 100–150ms for CUDA init per invocation.

2. integer_adm3 / integer_aim missing from CUDA output. These features are emitted by the CPU path; the CUDA float_adm_cuda extractor apparently does not expose them in the pooled metrics JSON. Requires investigation: ADR-0574 added VMAF_feature_aim_score and VMAF_feature_adm3_score to float_adm_cuda. The missing output may be an integer vs float ADM path distinction — the vmaf_v0.6.1 model uses integer ADM by default, and the CUDA path may route through integer_adm_cuda which predates ADR-0574.

3. SYCL baseline unavailable. The Intel Arc A380 device is not exposed to one-off containers via --device /dev/dri/renderD129. The prior snapshot shows SYCL at ~93 fps on 1080p. To get SYCL baseline, run via the shared vmaf-dev-mcp container: docker exec vmaf-dev-mcp vmaf ... --backend sycl.

4. Score determinism is good. All three CPU runs and all three CUDA runs produce identical scores across runs (no non-determinism from threading or GPU scheduling observed at this clip length).

Future Comparison Baseline¶

When perf/cuda-vif-filter1d-ncu-driven lands, compare against:

• WL1 CPU median: 90ms, CUDA median: 149ms
• WL2 CPU median: 73ms, CUDA median: 154ms
• WL1 VMAF score (CPU): 76.667830, (CUDA): 76.667829, delta: −1×10⁻⁶
• WL2 VMAF score (CPU): 35.068672, (CUDA): 35.068668, delta: −4×10⁻⁶
• Existing BBB 1080p CUDA throughput snapshot: 249 fps (from perf_benchmark_results.json)