Research-0025 — FoxBird outlier resolved via Netflix + KoNViD-1k combined training¶

Updated: 2026-04-28.

Question¶

Research-0023 §5 identified FoxBird as the per-fold outlier on every architecture (mlp_small / mlp_medium / linear) of the LOSO sweep on the Netflix Public 9-source corpus — PLCC ≈ 0.93 vs ≥ 0.99 on the other 8 sources. The hypothesis was a content-distribution mismatch within the existing 9-source corpus; the proposed unblocker was a different / larger training corpus (KoNViD-1k, BVI-DVC, AOM-CTC).

PRs #178 (KoNViD-1k acquisition + loader bridge) and #180 (combined trainer driver) shipped the infrastructure. The 1 200-clip KoNViD-1k parquet was acquired on 2026-04-28 (270 051 frames, ~26 min wall on the ryzen-4090 profile). This digest reports the empirical result of the canonical combined run.

Setup¶

python ai/train/train_combined.py \
    --netflix-root .workingdir2/netflix \
    --konvid-parquet ai/data/konvid_vmaf_pairs.parquet \
    --model-arch mlp_small \
    --epochs 30 \
    --batch-size 256 \
    --lr 1e-3 \
    --val-mode netflix-source-and-konvid-holdout \
    --val-source Tennis \
    --konvid-val-fraction 0.1 \
    --out-dir runs/tiny_combined_canonical

Data composition:

• Netflix Public 9 sources × 70 distortion pairs ≈ 9 690 frames (Tennis held out for validation).
• KoNViD-1k 1 200 clips × variable length = 270 051 frames (10 % of clip keys held out for validation by --seed 0).

KoNViD frame count dominates by ~30×. Training time was <1 minute on 8-core CPU thanks to the per-clip JSON cache being warm from the acquisition pass.

Per-clip result¶

Combined model runs/tiny_combined_canonical/mlp_small_combined_final.onnx scored against each Netflix source independently:

Mean across 9 clips: PLCC = 0.9983, SROCC = 0.9991, RMSE = 1.367.

Comparison to Netflix-only baselines¶

The canonical baselines from Research-0022 / Research-0023 trained on Netflix-only:

FoxBird specifically¶

Improvement¶

• PLCC delta on FoxBird: +0.0304 (Netflix-only → combined). That's a 3.04-percentage-point absolute gain on the canonical outlier — moves FoxBird from a 0.93-class outlier to a 0.99+-class clip indistinguishable from the rest of the corpus.
• RMSE on FoxBird: 17.296 → 3.216 = 5.4× lower.
• SROCC on FoxBird: +0.0233.
• The combined model also beats both Netflix-only baselines on the held-out Tennis clip (PLCC 0.9966 vs 0.9745 / 0.9448 baseline reading on Tennis from Research-0023 §3.1).

LOSO sweep on combined corpus (proper held-out generalisation)¶

The §"Per-clip result" numbers above are training-fit (FoxBird is in the training set). To address the caveat in §"Caveats" #1, this section reports the proper LOSO sweep on the combined corpus — 9 fold ONNXes, each trained on (8 Netflix sources + 90 % of KoNViD clip keys) and evaluated on the held-out 9th Netflix source.

Reproducer:

bash /tmp/loso_combined_sweep.sh   # ~3.5 min wall on cached corpus
python3 /tmp/eval_loso_combined.py

Per-fold result (each clip is the held-out source for its fold):

Comparison to Research-0023 Netflix-only LOSO¶

The PLCC standard deviation across folds drops 5.6× (0.0214 → 0.0038). That's the most significant finding: not just FoxBird, but every fold tightens to within 0.01 of the mean. The combined corpus eliminates content-distribution variance across the 9 Netflix sources — exactly what Research-0023 §5 hypothesised.

Held-out FoxBird beats Netflix-only baselines¶

fold_FoxBird/mlp_small_combined_final.onnx was trained without ever seeing FoxBird frames. Its FoxBird PLCC (0.9932 held-out) is materially better than:

• Research-0023 mlp_small Netflix-only LOSO on FoxBird ≈ 0.93
• vmaf_tiny_v1.onnx (Netflix-only mlp_small training-fit on FoxBird as a non-train sample): 0.9632

This is the proper validation: the model generalises to FoxBird without ever training on it, because the KoNViD UGC distribution covers the same high-motion / heavy-grain regime FoxBird inhabits.

Interpretation¶

The 30× frame-count expansion (9 690 Netflix → 280 K combined) and the UGC content distribution KoNViD-1k provides specifically address the FoxBird failure mode:

1. High-motion + heavy-grain regime is now in-distribution. FoxBird's low-light handheld content shares more structure with typical KoNViD-1k UGC clips (phone-shot, varied lighting, camera shake) than with the controlled Netflix Public sources. Adding 1 200 UGC clips broadens the feature distribution at the high-motion / low-bitrate end.
2. No regression on the Netflix-native sources. PLCC stays ≥ 0.998 on 7/9 Netflix clips after KoNViD addition; Tennis (the formal val clip) holds 0.9966 — within noise of the Netflix-only baselines. Adding KoNViD did not "wash out" the Netflix-tuned features.
3. Content-distribution variance, not architecture variance. Research-0023 §5 was correct that FoxBird wasn't an mlp_small-vs-mlp_medium problem. Same architecture, more diverse data, FoxBird converges with the rest.

What this unlocks¶

• Production model swap candidate. The combined-trained mlp_small_combined_final.onnx is a strict superset improvement over the shipped vmaf_tiny_v1.onnx on the 9-source LOSO evaluation. A future PR can register it as vmaf_tiny_v1_combined.onnx (or replace vmaf_tiny_v1.onnx outright after a sidecar-pinned A/B test on a held-out KoNViD fold).
• Closes Research-0023 §5 open question. No need to acquire BVI-DVC or AOM-CTC for FoxBird specifically — KoNViD-1k is sufficient. Larger-corpus work can target other goals (e.g. C2 NR metric per ai/configs/nr_mobilenet_v1.yaml).
• Validates PR #178 + #180 infrastructure end-to-end. Acquisition pipeline, loader bridge, combined trainer, eval harness all work as designed. Numbers reproducible from the parquet + the canonical CLI command above.

Caveats¶

• Validation set is mostly Tennis, not held-out FoxBird. --val-mode netflix-source-and-konvid-holdout holds out Tennis (Netflix) + 10 % of KoNViD clip keys. FoxBird is in the training set. The 0.9936 PLCC reported above is a training-fit metric on FoxBird, not a true held-out generalisation number. A LOSO sweep on the combined corpus with FoxBird specifically held out is the proper validation — that's the natural follow-up.
• Per-clip numbers are not directly comparable to Research-0023's per-fold LOSO numbers. Research-0023's FoxBird result was fold-level — model trained on the other 8 sources, evaluated on FoxBird. This digest's FoxBird result is training-fit — model trained on all 9 + KoNViD, evaluated on FoxBird.
• KoNViD-1k synthetic-distortion targets are libx264 CRF=35 round-trip. Same recipe as the Netflix dis-pairs, so the feature distribution is consistent — but real-world distortion diversity is wider than CRF-35 H.264. Adding AV1 / VP9 / HEVC distortions in a future corpus extension would broaden coverage further.

Next experiments¶

1. LOSO sweep on combined corpus (priority 1) — train 9 fold ONNXes with each Netflix source held out (plus 10 % KoNViD held out per fold). Report per-fold PLCC table; expect FoxBird's fold-level PLCC to rise from 0.93-class to 0.98+-class.
2. Compare against mlp_medium combined — does the larger architecture exploit the bigger corpus?
3. Cross-corpus transfer — train Netflix-only, eval on KoNViD held-out subset (and vice versa). Quantifies whether KoNViD addition is "more data" or "different data".
4. vmaf_tiny_v1_combined.onnx registration — sidecar JSON per ADR-0049 / ADR-0050 with dataset = "nflx+konvid-1k".

References¶

• req (popup, 2026-04-28): user direction "yes start the trainers and then to the recommendation".
• Research-0019 — Netflix corpus training methodology.
• Research-0022 — LOSO baseline for mlp_small.
• Research-0023 — 3-arch LOSO; §5 flagged the FoxBird outlier.
• ADR-0203 — tiny-AI Netflix-corpus training prep.
• PR #178 — KoNViD-1k acquisition + loader bridge.
• PR #180 — combined trainer driver.
• Reproducer: see "Setup" §; output checkpoint at runs/tiny_combined_canonical/mlp_small_combined_final.onnx.
• Per-clip eval helper: /tmp/eval_combined.py (reuses _load_session + _load_clip + CLIPS from ai/scripts/eval_loso_mlp_small.py).