ADR-0203: Tiny-AI training prep — implementation decisions¶

• Status: Accepted
• Date: 2026-04-28
• Deciders: Lusoris, Claude (Anthropic)
• Tags: ai, training, fork-local, onnx, docs

Context¶

ADR-0242 specified the scope for training tiny-AI models on the original Netflix VMAF training corpus. This ADR records the implementation decisions made by the follow-up PR (feat/tiny-ai-netflix-training-prep) for the loader, ground-truth pipeline, dataset adapter, evaluation harness, and training entry point under ai/data/ + ai/train/ — and the first concrete training run executed against the full corpus (results section below). ADR-0242's architecture-search table is converted into the three concrete archs registered with the new entry point.

Decision¶

1. Distillation from vmaf_v0.6.1 is the default ground-truth source. Per-frame vmaf predictions are cached as the training target. The published Netflix MOS table is not used in the default path; a future PR can add a --targets-source mos switch.
2. Held-out validation is one source out, not k-fold. Default --val-source Tennis removes 6 pairs (Tennis_*) from training and uses them as a content-disjoint validation set (~9 % of the corpus by clip count, ~12 % by frame count given Tennis is 24 fps).
3. Three architectures are registered with the entry point: linear, mlp_small, mlp_medium. ADR-0242's "transformer / 1-D CNN" rows remain deferred.
4. Per-clip JSON cache under $VMAF_TINY_AI_CACHE (default ~/.cache/vmaf-tiny-ai) keyed by <source>/<dis_basename>.json. One write-then-rename per pair so crashed runs leave the cache consistent. Distillation scores and feature vectors share the same JSON envelope (one libvmaf invocation per pair) so the corpus is touched at most once per cache-warming pass.
5. Smoke command works without the real corpus. --epochs 0 --assume-dims WxH substitutes a zero-filled payload provider so CI can verify the harness end-to-end (subprocess test in ai/tests/test_train_smoke.py).

Alternatives considered¶

A. Ground-truth source — distillation vs Netflix MOS¶

B. Validation split — 1-source-out vs k-fold¶

C. Model architectures registered with the entry point¶

D. Caching strategy¶

Training results — first run on the full Netflix corpus¶

Hyperparameters used for the canonical first run:

Final validation metrics (720 frames from the held-out Tennis source):

Three-arch sweep — full Netflix corpus, val=Tennis, 30 epochs each¶

What this tells us:

• Linear is a useful sanity floor, not a viable model. PLCC 0.43 / RMSE 67 confirms the 6 features do carry VMAF-predictive signal (otherwise even the MLPs couldn't do better) but the relationship is strongly non-linear. The SVR's kernel and the MLPs' hidden ReLU non-linearities are doing real work.
• mlp_small wins on ranking (PLCC 0.975 / SROCC 0.979). The tiny model ranks clips identically to vmaf_v0.6.1 to within 2.5 %.
• mlp_medium wins on absolute fit (RMSE 6.35 vs 10.62, −40 %) but loses ranking (PLCC −0.023, SROCC −0.032). With ~720 train frames on a 2,561-param net, the model memorises per-clip absolute scale (helping RMSE) at the expense of cross-clip ranking — classic small-corpus overfitting.
• Latency: all three sub-25 µs / clip-row on onnxruntime CPU. Even mlp_medium is essentially free vs the SVR.

Recommendation: mlp_small is the default tiny-AI model, shipped as vmaf_tiny_v1.onnx. mlp_medium ships as vmaf_tiny_v1_medium.onnx for users who want absolute-VMAF agreement on the Netflix-corpus distribution and tolerate the ranking loss. The linear baseline is not shipped.

Next experiments (not yet run): leave-one-source-out cross-validation (9 folds × 3 archs) to confirm the RMSE-vs-correlation trade-off generalises across content; longer training with regularisation on mlp_medium to see if overfitting can be controlled.

Wall-clock time: feature-extraction prewarm 3.5 min on CPU (cache 4/9 warm from a prior aborted run; cold-start would be ~7–8 min); 30 epochs over 720 train samples completed in <30 s per arch.

ONNX files in-tree (both ≤ 50 MB):

• model/tiny/vmaf_tiny_v1.onnx (= run 1 / mlp_small_final.onnx)
• model/tiny/vmaf_tiny_v1_medium.onnx (= run 2 / mlp_medium_final.onnx)

Eval reports at model/tiny/training_runs/run{1,2_mlp_medium,3_linear}/eval_report.json.

Consequences¶

• Positive: The training pipeline is fully runnable on the local corpus. CI verifies the harness end-to-end without needing the 37 GB of YUV. Cache invariants are documented; rebuilding is one rm -rf ~/.cache/vmaf-tiny-ai away. Distillation removes the paywalled-MOS dependency.
• Negative: The teacher's systematic errors are inherited; if vmaf_v0.6.1 mis-predicts on a class of distortions, the tiny model will too. The 1-source-out split has higher variance than 9-fold; reporting a single PLCC is statistically thin (the user is expected to re-run with multiple --val-source choices for any serious quality claim).
• Neutral / follow-ups:
• Add a --targets-source mos path once the published Netflix MOS subset is wired up.
• Add a Lightning module variant (current loop is plain torch optim — Lightning is overkill for a 257-param net but the user may want it for callbacks / logging on longer training runs).
• Promote the leading checkpoint to model/tiny/ via vmaf-train register and update docs/ai/models/ per ADR-0042.

References¶

• ADR-0042 — tiny-AI doc-substance rule.
• ADR-0108 — six deep-dive deliverables.
• ADR-0242 — scope of the Netflix-corpus tiny-AI workstream (parent of this ADR).
• Research digest 0019.
• Source: req (direct user instruction in 2026-04-28 daily prep routine asking for the runnable loader / eval / Lightning harness on top of the ADR-0242 scaffold).