gRPC streaming (ScoreStream)¶

Status: Accepted / implemented (ADR-0933). Phase 1 shipped the schema + server stub; Phase 2 (2026-06-13) wired the handler to the libvmaf engine via the in-memory pkg/libvmaf.StreamScorer. The RPC now returns real per-frame scores. Both vmafx-server and vmafx-node (ADR-1109) serve it.

Why a streaming RPC¶

The original vmafx-server gRPC service exposes a single unary scoring RPC, Score(ScoreRequest) -> ScoreResponse, where ScoreRequest is path-based: the client passes absolute filesystem paths to a reference and a distorted clip on shared storage, and the server opens both, runs libvmaf to EOF, and returns one pooled score.

That shape does not cover two real workloads:

1. Live encode quality monitoring. A transcoder produces frames as it goes. The client wants per-frame VMAF while the encode is still running, not one pooled number after the file is final.
2. In-memory raw pixels without disk inflation. When the client already has the pixels in RAM (an ffmpeg sidecar reading frames from stdout, an MCP tool, a Rust test harness), shoving them through a single unary ScoreRequest means a multi-GB request payload — gRPC's default max-recv-msg-size is 4 MiB and even with that lifted, a single jumbo request destroys flow control and blocks the server's HEAD-of-line queue.

The new RPC is bidirectional streaming:

rpc ScoreStream(stream ScoreStreamRequest) returns (stream ScoreStreamResponse);

Both directions are streams so the client can begin consuming per-frame scores while it is still pushing later frames; gRPC handles backpressure in both directions automatically.

Message shape¶

The proto lives in proto/vmafx.proto. Summarised:

Supported pixel formats (PixelFormat): YUV 4:2:0, 4:2:2, 4:4:4 in 8-bit and 10-bit-little-endian — a subset of FFmpeg's AVPixelFormat that libvmaf already accepts on its picture-import path.

Backwards compatibility¶

The unary Score and Health RPCs are unchanged. The proto package stays vmafx.v1. All existing clients keep working without recompiling.

Deprecation of unary Score is staged for the next major version, after Phase 3 lands. The follow-up ADR will document the timeline.

Phase rollout¶

How per-frame scoring works¶

Several VMAF features (notably motion) are temporal — they only finalise once the whole sequence has been read and flushed. The handler therefore ingests every FramePair first, and once the client half-closes it flushes the engine and harvests the per-frame scores via vmaf_score_at_index, streaming back the FrameScore messages followed by the terminal AggregateScore. gRPC flow control still bounds the request side while frames are pushed, so a multi-GB sequence does not inflate a single message. The streaming pooled VMAF is bit-identical to the file-reading ScoreDirect path (verified on the 48-frame golden pair).

Client usage¶

The pkg/score package wraps the generated client so callers do not have to hand-craft the oneof framing:

package main

import (
    "context"
    "errors"
    "io"
    "log"

    vmafxv1 "github.com/VMAFx/vmafx/gen/go"
    "github.com/VMAFx/vmafx/pkg/score"
)

func main() {
    cli, err := score.Dial("vmafx-server:50051")
    if err != nil {
        log.Fatal(err)
    }
    defer cli.Close()

    ctx := context.Background()
    stream, err := cli.OpenScoreStream(
        ctx,
        1920, 1080,
        vmafxv1.PixelFormat_PIXEL_FORMAT_YUV420P,
        "vmaf_v0.6.1",
        0, // frame_count_hint: unknown
    )
    if err != nil {
        log.Fatal(err)
    }

    // Push frames from your decoder loop ...
    for i := uint32(0); i < numFrames; i++ {
        ref, dist := nextFramePair()
        if err := stream.PushFrame(i, ref, dist); err != nil {
            log.Fatal(err)
        }
    }
    if err := stream.CloseSend(); err != nil {
        log.Fatal(err)
    }

    // Drain per-frame scores + terminal aggregate.
    for {
        fs, agg, err := stream.Recv()
        if errors.Is(err, io.EOF) {
            break
        }
        if err != nil {
            log.Fatal(err)
        }
        if agg != nil {
            log.Printf("pooled VMAF: %.4f over %d frames in %dms",
                agg.Score, agg.FramesProcessed, agg.ElapsedMs)
            continue
        }
        log.Printf("frame %d: %.4f", fs.Index, fs.Score)
    }
}

Reproducer / smoke test¶

# From a fresh clone, with a CPU libvmaf build at core/build-cpu:
meson setup core/build-cpu core -Denable_cuda=false -Denable_sycl=false
ninja -C core/build-cpu src/libvmaf.so.3.0.0

# End-to-end streaming scoring against the 48-frame golden pair:
CGO_ENABLED=1 go test ./pkg/libvmaf/ -run StreamScorer            # engine
CGO_ENABLED=1 go test ./cmd/vmafx-server/ -run ScoreStream        # server RPC
CGO_ENABLED=1 go test ./cmd/vmafx-node/server/ -run ScoreStream   # node RPC

The end-to-end tests push the real 576×324 / 48-frame YUV fixtures over an in-process gRPC stream and assert the server returns one FrameScore per frame plus a terminal AggregateScore whose pooled VMAF matches ScoreDirect. Validation-only cases (zero dimensions, non-config first message, frame-size mismatch, unloadable model) assert the corresponding codes.InvalidArgument / codes.NotFound without needing the fixtures.

References¶

• ADR-0933 — decision record for this rollout.
• ADR-1109 — vmafx-node serves the same VmafxScoring surface.
• ADR-0703 — original vmafx-server unary surface.
• ADR-0711 — controller that will consume ScoreStream directly.