WebGPU Player

WebGPU Player is a media playback engine for the browser. It plays sources a browser cannot play natively, and presents HDR and Dolby Vision through its own color pipeline.

• Demux: Mediabunny, from the fork in vendor/mediabunny/, in a worker.
• Decode: WebCodecs, or the decoder packages a host loads at runtime: WGSL video decoders that run on the GPU, and WebAssembly audio decoders.
• Present: WebGPU, with tone mapping for HDR10, HDR10+, HLG, and Dolby Vision.
• Audio: an AudioWorklet output that owns the playback clock.

Supported formats lists the containers, codecs, and HDR formats it plays, and those it does not.

The engine gives a host page two layers:

1. A custom pipeline for direct-play sources the browser has qualified. Demux, decode, presentation, and audio all run in the engine.
2. WebGPU presentation of frames the host’s own <video> element decodes, for color control of SDR video.

Nothing native is offered until the running browser has proven it. A native decode route is qualified by decoding a known stream and checking the exact output. A decoder package carries its own vectors instead, which run only when the host verifies it. An HDR or Dolby Vision presentation is authorized by rendering a known input through the production shader on the current GPU device.

The engine knows nothing about Jellyfin’s player, server API, or UI; it reads Jellyfin’s media metadata shapes as its input. Its host is the Jellyfin plugin jellyfin-plugin-webgpu-player, whose web client add-on embeds the engine as a git submodule at jellyfin-webgpu-client/vendor/webgpu-player/. The add-on supplies the Jellyfin player, the device profile, the settings, and the HTML fallback, and the plugin documents them in its own book, in docs/ of the plugin repository.

Invariants

• A host advertises only what the selected route implements and has exact-output evidence for. A route is never gated on resolution, level, frame rate, bitrate, or a throughput benchmark.
• capability/CustomContainerCodecSupport.ts is the only table of which container carries which codec. There are no decoder pair blacklists.
• A failure in the custom pipeline falls back to the host’s HTML player in the same session, or asks for one renegotiation. A player is never selected recursively.
• WebGPU needs a secure context. Validate over HTTPS against a local server. A successful negotiation is not a successful playback.

Conventions

• Engine paths are relative to the engine root. Paths inside src/ may drop the prefix; demux/, decode/, presentation/, audio/, capability/, color/, validation/, and pipeline/ are its folders.
• Symbols are written file:symbol or Class.method.

Where to look

Set up a checkout

You need

• Node.js 24 and npm 11.
• For the engine’s WebAssembly modules: GNU Make, Git, a POSIX shell (Git Bash on Windows), rustup, and Emscripten 4.0.13, which the build can install itself with Python 3.
• To build the decoder packages from the wgpud/ submodule instead of installing their pinned release: also CMake, Perl, and GNU patch (see WebAssembly build).
• For the codec vectors: Python 3.10 or later, and the FFmpeg build named in Codec vectors.
• To convert mpv shaders: Python 3.10 or later, and naga-cli 30.0.1; see Migrate an mpv shader to a wgpup.
• For this book: mdBook 0.5.
• To run the engine: Chrome, Edge, or Firefox with WebGPU and WebCodecs, on a secure context (HTTPS or localhost). Firefox on Windows has no WebCodecs HEVC, so HEVC plays there only through a GPU decoder package; see Decisions.
• On Windows, to build the decoder packages: long path support. The decoder projects’ FFmpeg and cargo trees pass 260 characters in a nested checkout. Enable LongPathsEnabled and Git’s core.longpaths; GNU Make and the MSVC linker that cargo uses must accept long paths too.

Steps

1. Install the dependencies. In a standalone checkout, run npm ci in the engine root. When a host lists the engine as an npm workspace, run npm ci in the host root instead; the engine then has no node_modules of its own. Inside a host, an npm install or npm ci run in the engine root resolves to the host’s workspace and rewrites its lockfile; pass --workspaces=false to install the engine standalone there.

2. Fetch the Mediabunny fork, which the engine demuxes with:

   git submodule update --init --depth 1 vendor/mediabunny
   

   Nothing installs it from npm: tsconfig.json maps mediabunny to the folder, and node scripts/mediabunny.mjs builds its dist/ with the fork’s own pinned toolchain. The typecheck, test, lint, and build scripts run that step first and skip it while dist/webgpu-player-build.json names the checked-out commit; --force rebuilds. See Decisions for why it is a fork, and which pull requests it merges.

3. Fetch the decoder projects and the pass projects, which are repositories of their own checked out at wgpud/ and wgpup/, and install their pinned releases:

   git submodule update --init wgpud wgpup
   npm run pinned-releases
   

   The checks need both submodules: npm test runs the decoder projects’ tests and the pass tests that read wgpup/, and the typecheck and lint cover wgpud/*/test/. npm run pinned-releases downloads the decoder packages and pass files that tools/pinned-releases.json pins into bin/wgpud/ and bin/wgpup/, checking each file’s SHA-256; it needs the network only for files it installs (see WebAssembly build). To build them from the submodules instead, run make -C wgpud sources and npm run decoders, which need the decoder toolchain, and npm run passes; the installer keeps such local builds unless you pass --force.

4. Build the engine’s WebAssembly modules, once per checkout and again after their sources change:

   make -C wasm all
   

   Emscripten comes from the emsdk that EMSDK names, which emsdk_env sets. Without EMSDK, run make -C wasm emsdk first: it installs the pinned Emscripten into the engine’s own emsdk in bin/emsdk/, about 2 GB, and the Makefiles use that emsdk from then on. The build writes bin/wasm/, which is ignored. See WebAssembly build.

   ./build.sh runs every build step: after checking its tools (node, npm, git, make, cargo, and the pinned Emscripten, which it checks through wasm/Makefile), it installs the pinned releases with scripts/pinned-releases.mjs, then builds the Mediabunny fork, runs make -C wasm all, and runs npm run build; --production minifies the workers. It checks every tool before it removes or builds anything, and without EMSDK it runs make -C wasm emsdk itself when the engine’s emsdk is missing or of another version. It builds no decoder package; build those from the submodule as step 3 says. ./build_clean.sh, which runs ./build.sh --clean, first removes bin/wasm/, bin/libraries/, bin/build-info.json, bin/wgpud/, bin/wgpup/, bin/installed-releases.json, the sandbox’s Vite build, the WebAssembly build tree, and the libdovi crate target, and rebuilds the Mediabunny fork too; it keeps the submodules, and the decoder projects’ build trees belong to make -C wgpud clean. Run both in Git Bash on Windows.

5. Run the checks from the engine root:

   npm run typecheck
   npm run lint
   npm test
   npm run build
   python scripts/generate_all_codec_vector_assets.py --check
   

   With the decoder toolchain, npm run decoders -- --check also checks that the wgpud/ submodule at the pinned commit rebuilds the pinned release byte for byte. Inside a host workspace, run the npm scripts from the host root with -w webgpu-player, for example npm test -w webgpu-player.

What the checks cover

A host may lint the engine with its own ESLint config as well; the Jellyfin add-on does. Engine changes must then pass both configurations.

Repository layout

bin/ holds:

• bin/wasm/: the engine modules’ builds from make -C wasm.
• bin/wgpud/: the decoder packages of the pinned release from npm run pinned-releases, or local builds from npm run decoders.
• bin/wgpup/: the pass files of the pinned release from npm run pinned-releases, or local copies from npm run passes.
• bin/installed-releases.json: the files npm run pinned-releases installed, with their SHA-256.
• bin/libraries/ and bin/build-info.json: the served assets from npm run build.
• bin/playback_smoke_media/: local playback media from the smoke media generators.
• bin/plantuml/: the pinned PlantUML jar that tools/render-diagrams.mjs downloads.
• bin/emsdk/: the engine’s own emsdk at the pinned Emscripten, which make -C wasm emsdk installs and the Makefiles use when EMSDK is unset.
• bin/codec_vector_assets/: the generated codec vectors. This is the one committed folder, so every change to a vector shows in review.

Folder names live in one file

tools/constants.json names every engine folder. Read a path from it instead of spelling it out, so moving a folder is a one-line change.

Some files cannot read it. Recipes lists them.

Supported formats

This chapter lists what the custom pipeline plays, and what it does not. A source plays when its container, its video track, and its selected audio track all appear below, and the running browser and GPU pass the route’s evidence. A codec whose decoder is a package plays only when the host loaded a decoder package for it. Anything else is ineligible, and the host falls back. HEVC and Dolby Vision support holds the full HEVC and Dolby Vision matrix, and Eligibility and routes the checks, routes, and evidence behind every row.

The pipeline plays a direct-play file over http(s), or a local File the host holds. It does not play live streams, HLS or DASH, or a server transcode.

Containers

PCM in Matroska is 8-bit unsigned, 16, 24, or 32-bit integer in either byte order, or little-endian 32 or 64-bit float. PCM in MP4, M4V, and MOV is 16, 24, or 32-bit integer or 32 or 64-bit float, in either byte order; MOV also carries 8-bit, mu-law, and A-law PCM. PCM in AVI is 8-bit unsigned, little-endian 16, 24, or 32-bit integer or 32 or 64-bit float, mu-law, or A-law, under its own format tag or as the SubFormat of WAVE_FORMAT_EXTENSIBLE, which FFmpeg writes for a layout other than mono or stereo, a rate above 48 kHz, or integer samples wider than 16 bits. FLAC and Vorbis in AVI play when their strf extradata holds the STREAMINFO block or the three Xiph-laced Vorbis headers, as FFmpeg’s muxer writes them. WMA plays from a Matroska A_MS/ACM track and from an AVI or ASF stream, whose WAVEFORMATEX names it by its format tag: 0x0160 WMA v1, 0x0161 WMA v2, 0x0162 WMA Pro, 0x0163 WMA Lossless, and 0x000A WMA Voice, as FFmpeg’s RIFF tags do, under its own tag or as the SubFormat of WAVE_FORMAT_EXTENSIBLE. Any other tag in these containers is refused by its tag. An AVI audio stream whose format tag the fork names no codec for opens codec-less with its tag, in decimal, as its codec ID, as a video stream of an unknown FourCC does, and keeps its packets timed as FFmpeg’s demuxer times them. The rows come from capability/CustomContainerCodecSupport.ts.

AVI plays every packet its indexes list, read as FFmpeg reads them: a stream’s OpenDML index when it has one (its indx super index and the ix## standard index that index locates in each RIFF list), else idx1, which covers only the first RIFF list. An OpenDML file past about 1 GiB, whose later AVIX RIFF lists only the OpenDML indexes cover, therefore plays to its end and seeks there. A video stream that still outruns its index, as in a truncated file or one whose OpenDML index cannot be read in full, is refused rather than stopped where the index ends. MPEG-4 Part 2 in the packed bitstream of DivX 5 and Xvid, which DivX user data ending in p marks, is unpacked to one VOP per packet as FFmpeg’s mpeg4_unpack_bframes filter does. AVI keeps no presentation times, only each chunk’s slot in decode order, so the fork’s AVI demuxer gives H.264 and HEVC packets their pictures’ presentation times from the picture order counts in their slice headers, as FFmpeg does, and B-frames play in order on every browser. The worker restamps MPEG-4 Part 2’s B-VOPs into presentation order itself, as it does for Matroska Video for Windows tracks (see Eligibility and routes). A video stream whose FourCC the fork names no codec for, such as DIV3, MP42, or H263, opens codec-less with its FourCC as its codec ID, so the MPEG-4 decoder package’s route takes it by that FourCC and any other route refuses it by its FourCC. HEVC in AVI plays under the HEVC, H265, hvc1, and hev1 tags, and under the zero tag FFmpeg’s AVI muxer writes unless told one, which the fork recognizes by the parameter sets of the first chunk; FFmpeg, and so Jellyfin, reads such a stream as uncompressed video. MPEG audio, AC-3, and E-AC-3 in AVI play one frame per packet however the muxer cut its chunks: the fork’s AVI demuxer reads a stream’s chunks as one byte stream and splits it into frames, as FFmpeg’s parser does, timing a byte-timed stream’s frames by their byte offsets and a chunk-timed stream’s by their chunk’s time, and drops zero-filled chunks, a leading ID3 tag, and a frame cut short. ASF is read by the fork’s ASF demuxer, as FFmpeg’s asfdec_f.c reads it: the Header Object’s streams, and each media object reassembled from its payloads (compressed payloads and audio spread included), timestamped by its presentation time less the preroll. Those timestamps are in decode order, as FFmpeg reads them and as its muxer writes them, so the worker restamps VC-1, WMV3, and MPEG-4 Part 2 B-frames into presentation order, as for Matroska Video for Windows tracks; H.264 and HEVC in ASF, which the fork would hand on in decode order, are refused. A stream opens by its FourCC or format tag as an AVI stream does: WVC1 and WMVA as VC-1, and WMV3, WMA, and the MPEG-4 Part 2 and MS-MPEG4 FourCCs codec-less, with the FourCC or the tag, in decimal, as its codec ID. A DRM-protected ASF file (a Content Encryption, Extended Content Encryption, or Advanced Content Encryption Object, or a stream flagged as encrypted) is refused, and so is a live broadcast, whose data packets vary in size; WMV1 and WMV2 have no route. MPEG program streams, MPEG-1 system streams and MPEG-2 program streams such as DVD-Video’s VOB files, are read by the fork’s program stream demuxer: MPEG-1 and MPEG-2 video and MPEG audio by their stream IDs, and the AC-3, DTS, and LPCM substreams of private stream 1 as FFmpeg’s mpeg.c numbers them. Its tracks come in the order of their first PES packets, as FFmpeg’s demuxer creates them, and their packets are FFmpeg’s, split as in MPEG-TS and stamped with the PTS of the PES packets they begin in. A picture whose PES packet carries no PTS, as on DVD-Video, which often stamps only the I pictures, is timed by MPEG video’s display order. MPEG audio Layer II (mp2) has no route, and neither does DVD-Video’s LPCM, which Jellyfin names pcm_dvd; subpictures and DVD-Video navigation packets are skipped. A .wma file is audio only, so it falls back to the host as every audio-only file does. An AIFF file opens, but the pipeline plays only sources with video, so AIFF, like every audio-only file, falls back to the host. A track whose edit list Mediabunny cannot honor, as getUnsupportedFeatures() reports, is refused rather than played out of sync.

Video

Every other video codec is not supported, among them WMV1, WMV2, Theora, VVC, ProRes, DNxHD, and Motion JPEG. Eligibility refuses Theora as a known codec with no route (video-codec-without-route), and the others as unsupported (codec-unsupported). WebCodecs is the browser’s own decoder. A GPU decoder package is WGSL a host loads, which decodes on the presenter’s device; it plays only what no native route plays, and the decoder projects in wgpud/ build one for HEVC, MPEG-1 with MPEG-2, VC-1 with WMV3, MPEG-4 Part 2 with H.263 and MS-MPEG4, and JPEG 2000. A package’s frames take the format that holds the stream’s pixel format, so 4:2:2 and 4:4:4 streams keep their chroma. The JPEG 2000 package converts X’Y’Z’ (xyz12le) to R’G’B’ as FFmpeg’s converter does, gamma 2.6, the XYZ to BT.709 matrix, then gamma 1/2.2, and writes it as limited-range BT.709 Y’CbCr at 12-bit 4:4:4, which presents as an untagged SDR stream’s BT.709.

HDR

Audio

Every other audio codec is not supported, among them MP2, ALAC, AMR, and Speex. A source whose selected audio track is not supported does not play through the pipeline. Every route resamples to 48 kHz. The host’s speaker layout setting picks the output layout: auto takes 7.1.4 on a device with 12 channels, 7.1.2 with 10, 7.1 with 8, 5.1 with 6, and stereo below; 5.1.2 and 5.1.4 play only when chosen. Dolby Atmos a package renders as objects plays to the whole layout, or to binaural stereo for headphones when the host asks for it on a stereo output. Every other source plays within the layout’s floor: a three-channel or 5.1 source outputs 5.1, a 7.1 source 7.1, or 5.1 on a 5.1 floor, and everything else downmixes to stereo.

Subtitles

The engine reads no subtitle track and renders no subtitles, and a source’s subtitle tracks never affect its eligibility. The host draws subtitles over the video: style.css keeps the host’s subtitle layers (.videoSubtitles, libass canvases, and other canvases) above the WebGPU canvas.

Architecture

Layers

A session takes one of two paths:

• (a) HTML decode with WebGPU presentation, for known-SDR input only. The <video> element or hls.js decodes, and the media element is the clock. Frames go rVFC, importExternalTexture, the identity route’s one-pass plan, canvas. For HDR or unknown input the presenter is off and the <video> shows.
• (b) The custom pipeline, for eligible direct-play VOD. The worker posts 'frame' messages (a VideoFrame, or raw planes in a pooled buffer, with Dolby Vision and HDR10+ metadata), and fMP4 segments for native-media audio. On the frame-pool route it decodes no video and posts compressed 'packet' messages instead, which a GPU decoder package’s WGSL decodes on the presenter’s device into a frame pool that the presenter reads in place. For a package with a parser, such as the HEVC decoder project’s, each 'packet' carries one picture’s record instead, which the package’s parser writes on the worker’s coordinator worker and its parse workers; a packet that holds several pictures, such as an AV1 temporal unit with hidden frames, posts one 'packet' for each. Decoded PCM goes from the worker’s audio decode worker straight to the AudioWorklet, over a channel the page hands over, and the page learns only each chunk’s place and length (audio-progress). The frame loop, CustomPlaybackFrameLoop, takes the controller’s current frame each animation frame and presents it. Decoded audio plays through an AudioWorklet in a pooled 48 kHz context, and AC-3 and E-AC-3 can play through a hidden <audio> element and MSE.

The host player and its HTML backend with any hls.js runtime are host code. A host drives the custom pipeline through PresentationSetup, which gates the eligibility’s presentation routes and configures the presenter for the chosen one, and runs one CustomPlaybackFrameLoop per playback. Everything else in the diagram is engine code. The plugin’s book follows the Jellyfin host’s half of a session.

The engine’s sources follow the stages a custom session runs: src/demux/ reads the container and shapes each track’s packets and the metadata they carry, and src/decode/ turns the packets into frames and PCM. src/presentation/ and src/audio/ present the frames and play the PCM, and src/pipeline/ runs the stages for each session (see Module map).

Clock

• Decoded PCM: the worklet’s render time, corrected to physical output time by BrowserCustomAudioOutput through getOutputTimestamp, re-anchors a performance.now MediaClock. After each flush the worklet renders silence from the flush position up to the first chunk’s timestamp, so an audio track that starts later than the video still clocks from the start position. A first chunk more than 30 s on is refused with the release and telemetry reason leading-gap, which the producer and the page’s bridge both report as the leading gap failure, failing the attempt as audio-output-failed, since a gap that long is a timeline fault that silence would hide. When the output starts, the clock holds its start for the output’s estimated latency, so video moves when the first sample is heard.
• Native-media audio: <audio>.currentTime. A first fragment more than 40 ms after the start position parks the element at the fragment, and play() waits out the gap while the MediaClock runs alone.
• No audio: MediaClock alone, held while video starves. An audio track that ends before the video hands the clock to the MediaClock once its tail has played out.

The clock runs at the requested playback rate, from 1/16x to 16x (pipeline/CustomPlaybackRate.ts), 1 by default. From 0.25x to 4x audio plays at that rate and stays the master: the worklet time-stretches the decoded PCM with its pitch kept (AudioWorkletTimeStretchSource.ts), and native-media audio plays at the rate on its <audio> element. Outside that range a generation plays without audio, and the MediaClock alone runs it, as for a source without audio. The worklet’s ring holds media-rate PCM, and the stretcher reads it at the rate, so its telemetry advances media time by the rate per output frame and reports the rate each frame rendered at; BrowserCustomAudioOutput scales the physical output correction, which is context time, by that rate into media time. At 1x with nothing left in the stretcher the worklet copies the ring sample for sample. A change to or from 1x continues the input, or crossfades one hop into it, so media time moves at most the 15 ms search half-width. When the ring cannot feed the stretcher, the frames it holds play out, the rest of its input at 1x, before the quantum counts as an underflow, so a track’s tail is never held back.

Video is pulled: each rAF draws the newest frame at or before the clock, and hands the presenter its timestamp and the clock’s rate, from which it measures the display’s refresh rate. Queued frames hold off video starvation, except before a generation or a resync takes its first frame: then frames whose earliest is more than the 2 s decode lag bound past the clock, while they hold every video credit of the route, count as starvation, so the stall bound falls back instead of waiting for the clock to reach them. After the first frame, frames that far ahead are a real gap in the video, such as a still or a slideshow, which the last frame holds as before. While a plan deinterlaces, each rAF takes frames one frame ahead of the clock, the presenter shows the frame before the newest, and a tick that takes no frame draws a held frame’s next field from history (see Presentation). While a plan holds a display-rate pass, each rAF takes frames one refresh ahead of the clock, and a tick that takes no frame redraws from history at its clock time.

Startup of a custom session

1. The host runs prewarmBrowserAudioContext(48000) synchronously inside its play call (the user-activation window) and calls presenter.startSession.
2. The host runs eligibility, described in Eligibility and routes, once the authorization keys the source needs have settled. Its own setup bound should end where the controller starts, so the controller’s startup bound applies after that.
3. The host gives the presenter a source-less <video>. The presenter enters push mode, and the host installs the shaders and authorizes the selected route through the presenter.
4. controller.play creates a generation, resets the clock, emits waiting, and starts the startup bound. It samples the source bytes the worker reads before its readiness, which source-progress reports each 256 KiB, and the decode counters every second, and fails after 20 s without progress, or at 60 s regardless. The fallback message names the counters it reached, so a timeout shows where startup stalled. CustomDecodeSession.start posts the generation’s start to the session’s decode worker, the prebuilt libraries/webgpu-player/CustomDecode.worker.js keyed per build with ?v=, which the session’s first start creates. The worker’s first start with decoded PCM audio spawns its audio decode worker, the prebuilt CustomAudioDecode.worker.js beside it, which it keeps for its life. Its first start for a package with a parser spawns its coordinator worker, the prebuilt GPUDecoderCoordinator.worker.js, which it keeps for its life too; that worker spawns the parse workers, the prebuilt GPUDecoderParse.worker.js, and keeps them while the package plays. Video gets 4 frame credits, 2 for raw planes, or 8 packet credits on the frame-pool route.
5. The worker prepares its tracks (canDecode), scans the static HDR metadata of the first 16 access units or 8 MiB, and posts ready. The frame-pool route skips canDecode, requires the codec its GPU decoder takes, and adds the track’s decoder configuration record, such as an hvcC, as codecPrivate. With a parser it opens the attempt’s parser before ready, which then carries the largest picture the parser declared in the codec private data. The session then creates the GPU decode session, or reuses the previous generation’s when its decoder, format, coded size, bit depth, codec private data, and declared maximum match. It scans HEVC SEI on the native PQ route, and AV1 MDCV and CLL metadata OBUs when the first sequence header signals PQ and no RPU route is selected. Video and audio then stream concurrently.
6. The session is ready when the first frame is queued and at least 100 ms of PCM has been submitted, or, on the native-media audio route, when the first segment is appended. The controller also waits for the presentation prepared for the track’s chroma format and size, which the session’s configured event carries (see Presentation).
7. completeStartupIfReady starts audio, resumes the clock, and emits ready and playing. The host then starts the frame loop.

Steady state

Video credits:

• takeFrame closes and re-credits older frames.
• A presented VideoFrame closes after submit(), but its credit returns only after queue.onSubmittedWorkDone(). This keeps the decoder’s surfaces from starving.
• Every wait on queue work, this one, the frame-pool route’s status readbacks, and the authorization probes’, goes through waitForGPUCompletion (GPUCompletion.ts), which polls the device with an empty submit every 2 ms while the wait is pending. Firefox completes onSubmittedWorkDone() and mapAsync() only when its GPU process polls the device, at a later submit or every 100 ms, so without the polls a credit or a readback waited up to 100 ms there (see Decisions).
• In raw mode the posted buffer is the credit, and it returns through recycle-frame.
• On the frame-pool route a credit is a packet queued on the page, and it returns once the packet’s decode work is submitted. With slicing on, which the host chooses through createGPUVideoDecodeSessionFactory and leaves off by default, a packet’s decode work submits in slices of at most 96 dispatches, of about equal length, so a large picture’s wavefront splits into several submits; off, every packet submits whole. GPUVideoDecodeFeed keeps up to two slices in flight, a whole packet counting as one, and submits the next once one completes or a status readback applies, so the GPU decodes on while the page learns of a completion, and presentation queues behind at most two slices of decode work. A reset, as a seek or a new video epoch makes, or a resync submits no more of the packet in progress. With a parser, a credit is one picture: the worker reads packets until their pictures run 8 ahead of the credits, a packet still parsing counting as one, so its parse workers stay busy, and a packet without a picture, such as a skipped leading picture, spends none. A presented frame returns its pool slot right after its submit, because the queue orders later decode work behind it.
• The owned HEVC, AV1, and VP9 paths read a packet only while they hold a credit, then wait up to 10 ms for the decoder to return a frame before reading the next. The credit is spent only when a frame is posted, so without that wait a hardware decoder, which answers a few milliseconds after taking a packet, received a whole group of pictures at once (see Decisions). Steady playback therefore decodes one packet per presented frame, while seek preroll still runs at decoder speed, because each returned frame ends the wait.
• A hardware decoder returns a frame once its decode is queued on the GPU, not once the GPU ran it, so a preroll that reads at decoder speed queues decode work faster than the GPU finishes it. On the VideoFrame route a preroll therefore decodes in batches of 4 frames (OWNED_VIDEO_PREROLL_BATCH_FRAME_COUNT): the batch’s last frame goes to the worker’s decode fence (VideoDecodeFence.ts), and the packet after the next batch waits until the fence completes, so the decoder runs at most two batches ahead of the GPU and never idles between batches. The fence is a small WebGPU device of the decode worker’s own that reads one texel of the frame and waits for that work, which waits for the frame’s decode; the attempt whose first packet precedes its start creates it, and until it is ready, or where the worker has no WebGPU, frames pass unfenced. The frame that ends the preroll posts once its own fence completes, so its presentation never waits behind the preroll’s decode work, and each wait ends after 250 ms at most. The trace records each wait as video-decode-fence.

Video presentation interval, above 1x:

• The display shows at most one frame per refresh, so above 1x the controller hands the session a presentation interval: the rate times the measured refresh interval (WebGPUPresenter.getDisplayRefreshIntervalMicroseconds, which the frame loop passes on when it changes), 60 Hz until measured, rounded to a whole microsecond. At or below 1x there is none; the session keeps the interval for later generations and sends set-video-presentation-interval to the running one.
• After an attempt’s first presented frame, a frame within which no multiple of the interval falls (VideoPresentationInterval.ts, in container time) is never posted. The owned paths close it unposted without spending a credit and record video-frame-decimated; decimated frames count toward the preroll’s fenced batches of 4, so the decoder stays at most two batches ahead of the GPU. The sample cursor route (H.264, VP8) closes it and keeps the credit it took, and raw planes never copy it. Nothing is decimated while a Dolby Vision enhancement layer decodes, and the frame-pool route never decimates in the worker, because a package’s parser must see every packet; there the page’s takeFrame drops the frames it never presents.
• The owned HEVC path does not decode a picture no later picture references (HEVCTemporalSubLayers.ts: every base-layer VCL NAL unit a sub-layer non-reference type at the highest temporal sub-layer every SPS declares) when it can never present: it ends at or before the attempt’s start, or it follows the first presented frame off the interval’s grid. Its HDR10+ and RPU metadata is still read in decode order, the trace records video-packet-skipped, and a seek’s preroll skips such pictures too. An SPS that does not parse leaves the highest sub-layer unknown for the attempt, and nothing is skipped; nothing is skipped while an enhancement layer decodes either.

Raw planes:

• A decoded VideoFrame is copied once, into a buffer of 256-byte-aligned rows, which the frame message transfers to the page.
• A run keeps spare buffers: those the page recycled and those of frames closed unposted. A drained frame or a copy takes a spare of its byte length, or a new buffer when none fits, and at most 2 spares of each byte length are kept, as many as the page can hold; a 4K 10-bit frame is about 25 MB.

Audio credits:

• Decoded PCM: the audio decode worker’s WorkletPCMProducer owns the worklet’s credit window. Chunks are 40 ms to 12000 frames, with 8 credits, so the credits can never hold more than the 2 s worklet ring. One credit returns per chunk the worklet consumes, with the chunk’s buffers, which later chunks reuse. The audio decode worker reconciles decoder timestamps within 2 s before the resampler (see Decisions), and fails the attempt as decode-failed beyond that. A gap, an overlap, or an overflow that reaches the producer is therefore an engine fault, and raises audio-output-failed.
• The worker sends each decoded PCM attempt’s input in batches that close at 40 ms of media time or 64 inputs: compressed packets for a decoder package, or Mediabunny’s decoded samples cut to the start. 4 batches are in flight at once, and each returns its credit once the audio decode worker has rendered it, so demux runs at most a few batches ahead of the worklet’s window.
• Native-media audio uses 2 MiB or 2 s segments, 2 credits, and appends at most 6 s ahead.

Lag guard, in CustomPlaybackController:

• A frame more than 2 s behind the clock is discarded and re-credited, and a video wait starts.
• An empty queue for at least 100 ms while playing also starts a wait.
• A wait of 10 s or more fails with playback-stalled.
• Above 1x the decoder may never catch up with the clock, so video jumps ahead instead: a taken frame more than 500 ms of wall time behind, times the rate in media time and at most the 2 s bound, is discarded (decode-lag), and video restarts at the next keyframe at or after 250 ms of wall time ahead of the clock (session.resyncVideo(target, 'next')), recorded as video-resync. A missing frame jumps the same way when the decoder is that far behind and no jump is pending. The keyframe becomes the attempt’s start and presents first, even when it arrives later than the lag allows, so every jump shows a picture and ends the wait; a frame after it that still lags jumps again. A stream without a keyframe there ends its video. While a jump is pending the stranded-queue check and the catch-up discard are off, since its queue starts ahead of the clock by design. At or below 1x, and when the session declines the jump, the waits above apply.

Hidden page, through setPageVisibility and drainBackgroundVideo:

• No rAF runs while the page is hidden, so the host takes and discards due frames against the clock on a timer (25 ms in the Jellyfin host). Credits keep flowing and audio stays the master.
• After 10 s hidden, audio-clocked native decode sends suspend-video. The worker unwinds only its video attempt and releases the decoder. Once the audio track has ended, only video can reach the end of the stream, so video is no longer suspended and audio-ended resyncs a suspended decoder at once.
• On return, a suspended video, or one more than 2 s behind, gets resync-video at the clock. The worker restarts video alone at the preceding keyframe and skips frames before the target. Waits restart, so hidden time never counts toward playback-stalled.
• Every resync or suspension advances the video epoch. Frames carry videoEpoch, and the session closes frames of a replaced epoch and returns their credits.
• A reclaimed WebCodecs decoder (QuotaExceededError) posts video-interrupted. Video resyncs at once when visible, and on return when hidden.
• When audio outlasts the video track, the worker posts video-ended before the run’s ended, and the controller holds the last frame without waiting, stalling, suspending, or resyncing.

Geometry: WebGPUPresenter.bindLayoutHandling invalidates layout on resize observers, class and style mutations of the video and its ancestors, CSS motion events, window resize, seek, and refresh. The backing size is the CSS size times the device pixel ratio, capped by maxTextureDimension2D.

Presentation

• Every route presents through the pipeline in src/presentation/passes/.
• A route’s built-in chain is its fixed signal, which reads the source and reconstructs Dolby Vision, and its default color functions: transfer decode, tone mapping, output encode, display controls, and dither.
• planPresentation arranges the chain and any user passes into GPU passes. Without user passes every route is one pass, which renders exactly as the legacy fused shaders did.
• PresentationExecutor owns a plan’s pipelines, textures, history rings, and bind groups. A frame costs one uniform write, the source’s import or upload, and one encode per pass, and the presenter submits once.
• A new plan compiles while the previous executor keeps presenting, and replaces it when ready.
• The device keeps the most recently used 64 compiled passes (PresentationPipelineCache.ts), keyed by their WGSL, bind group layout, and target format, so an executor built again for passes it compiled before, in a replan or in a later session on the kept device, compiles nothing.
• A starting generation builds the plan its track declares before the first frame is due: the session’s configured event carries the chroma format and coded size of the worker’s ready response, the controller hands them to the preparer the frame loop registered (setVideoPresentationPreparer), the presenter plans with them as with a frame’s facts (prepareDecodedVideo), and the generation reports ready only once that executor is installed. The first frame of those facts then presents without a replan, and a frame that differs replans as before.
• User passes come from WebGPUPresenter.setPresentationPreset. A pass that cannot run is left out of the plan, and getPresentationPlanReport names it with a reason code and a terse detail.
• The spine starts with the plane stages, LUMA, CHROMA, and NATIVE, which hold the signal’s planes as normalized codes (see Planes). Each signal offers its planes (SignalDefinition.planes): its own reads of the luma and chroma codes, chroma’s siting, and its conversion to ‘source’ RGB. A plan reads a plane apart into a texture only for a pass that reads it, and when a pass replaced a plane, a merge pass converts the planes into the spine’s first image at luma’s size, reading each plane no pass replaced from the source, so it converts exactly as the signal does. The raw and frame-pool routes hold the decoded planes, and the Dolby Vision routes offer none. The external routes derive them from the browser’s RGB through the stream’s own matrix and range (SignalPlanes.transformCode), the external HDR route from the codes its signal recovers, with chroma at the stream’s subsampling. The frame-pool route reads a pool slot in place, whose planes are padded to 64 rows and columns, and clamps every tap to the frame’s visible picture, which the executor writes into the slot’s layout uniform as it binds the frame, so a picture’s edge repeats as on the raw route.
• Every signal reads 4:2:0 chroma at the location the stream declares, and 4:2:2 chroma on its left luma column; the external routes derive chroma at those sites. The decode worker reads the subsampling and the location from the track’s decoder configuration (VideoChromaFormat), from its first key packet’s in-band SPS where an H.264 or HEVC track’s configuration holds none, and MPEG-1 as center-sited 4:2:0; its ready response names them, and every frame of every output mode carries them, so the presenter plans with them without a host fact. A deinterlacing plan derives them from its deinterlaced image through the same transform (SignalPlanes.imageCode), so plane passes run on interlaced sources without reading the source again.
• The planes hold the frame’s visible samples (PlanFacts.sampleWidth and sampleHeight, the luma.width and luma.height facts), and chroma’s extent is luma’s over the subsampling divisor, as libplacebo sizes plane rects; the presenter plans again when the frame’s sample size changes while preset passes plan, and when the stream’s chroma changes.
• Every image the plan holds has a plan-time offset, as libplacebo’s texture rects place mpv’s textures: chroma starts at its siting, a pass’s output takes the hooked image’s offset scaled to its size plus its declared offset, and offset: align returns it to the reference grid. No read is shifted; the merge resamples chroma onto luma’s grid through both offsets, and the scaler corrects its image’s offset within its resample.
• A pass file’s texture blocks reach the planner inside the manifests of the passes that bind them, and the plan names each by its content identity (PresentationPlan.passTextures). The presenter’s PassTextureCache uploads a block when a plan first binds it, hands the uploads to PresentationExecutor.create, and after installing an executor releases the uploads neither its plan nor the preset holds.
• A pass’s when reads its parameters, and a const parameter compiles into the plan’s modules, so a change to either builds a new plan (PlanRequest.parameterValues), which the executor of an unchanged plan adopts without rebuilding; any other parameter is a uniform write.
• The prelude gives a body that calls them the source’s transfer function in mpv’s scale of linear light (BuiltinChain.linearLight, from the built-in transfer decode’s code) and its planes transform.
• Temporal history advances only on new frames; redraws from history, of a deinterlacing or display-rate plan, do not advance it. It resets on a seek, a new generation, a video epoch change, and a frame whose timestamp goes back or jumps past 2.5 frame durations.
• The frame.rate fact is the stream’s declared frame rate (PresentationInput.getPresentationFrameRate): its average frame rate, else its real frame rate when the average is missing or at least 1000, which configurePresentationColorPipeline carries as PresentationColorPipelineConfiguration.frameRate.
• Authorization renders the default chain through the same planner and executor, so a preset never changes the route.
• With HDR output enabled by the host, an HDR source presents extended-range values on an rgba16float canvas.

Deinterlacing

• An interlaced stream presents through Presentation deinterlace, the engine’s yadif pass (passes/builtin/DeinterlacePass.ts; see Decisions).
• The presenter asks for it (PlanRequest.deinterlace) once the stream’s facts or any frame call the video interlaced, while the mode is not off. A Dolby Vision route never asks, since its profiles are progressive.
• The planner places it before every stage: the signal renders into signal, a ring of three frames at the frame’s visible size in rgba16float, and the pass reads the frame it shows, the one before it, and the one after it, and writes deinterlace at the same size, which the rest of the spine takes.
• It deinterlaces the signal’s ‘source’ RGB per channel in f32, so one pass serves every route but Dolby Vision’s, 8 to 12 bits and every chroma format.
• Plane passes take their planes from deinterlace, through the signal’s transform, as mpv’s user shaders see the planes a deinterlacing filter wrote.
• A progressive frame passes through untouched, as does a frame beside a progressive one that repeats a field, as yadif’s deint=interlaced does.
• The presentation uniform’s fieldState word (PRESENTATION_FIELD_STATE) names what shows: whether the pass interpolates, the field shown, whether it is its frame’s second, whether the shown frame is the one before the newest, and whether the frame the signal reads now is interlaced.
• On the raw and frame-pool routes a deinterlacing plan’s signal reads an interlaced frame’s 4:2:0 chroma within each field, at MPEG-2’s interlaced siting, so a field never takes the other field’s colors. A native VideoFrame arrives as an external texture whose chroma the browser upsampled progressively, so its fields share chroma rows.
• Each frame carries its own fields (VideoFrameFields): a GPU decoder’s reportPictureFields report, else its parser’s parser_picture_fields word, else the container’s scan, Matroska’s FlagInterlaced and FieldOrder or an MP4 sample entry’s fiel box. A frame without fields takes the stream’s scan from the host’s facts, FieldOrder and then IsInterlaced; a frame of unknown scan presents as it is.
• DeinterlaceTimeline mirrors the ring and picks the draw for a clock time: a frame’s first field at its own time, and at double rate, the default, its second field halfway to the next frame’s time, rounded down to a whole microsecond. Single rate shows each frame’s first field only, and off presents frames woven, as decoded.
• Deinterlacing reads the frame after the one it shows, so the frame loop takes frames one frame ahead of the clock (getFrameLookaheadMicroseconds), and the presenter shows the frame before the newest. The clock and audio never move: decoding runs one frame further ahead. After a reset the first frame shows at once without a frame after it, and at the end of a stream, or while decoding falls behind, the newest frame shows once the clock reaches it, as FFmpeg’s yadif repeats its last frame.
• A tick that takes no frame draws the held frame’s next field from history (presentHeldFields), without a source, so passes that read the source are skipped.
• setDeinterlaceMode switches between double rate, single rate, and off; only a switch to or from off builds a new plan.

Display rate

• The presenter measures the display’s refresh rate (DisplayRefresh) from the animation frame times the frame loop hands recordAnimationFrame: a least-squares line through the times over refresh counts, which counts skipped refreshes and averages timer coarsening up to 1 ms. It reports an interval once at least 30 frames fit the line with a relative standard error of at most 0.02%, over a window of 240 refreshes. A frame more than 0.4 intervals off the line, as after a move to another display, starts the measurement over, and a gap over 500 ms, as a paused loop leaves, starts a new window; the last measurement holds until the next is precise.
• The display.rate fact is that rate, 0 until measured, and the presenter plans again when it moves more than 0.1% (DISPLAY_RATE_REPLAN_TOLERANCE) from the rate the plan read, while a preset pass’s when reads it.
• A pass with rate: display plans only while frames come from the frame loop (PlanRequest.displayRateAvailable), and not in a deinterlacing plan; otherwise it is left out as display-rate-unavailable.
• In a display-rate plan (PresentationPlan.displayRate) the pass that reads the source writes a texture rather than the canvas, and a display-rate color step gets a pass of its own, Presentation color.
• The frame loop takes frames one refresh ahead of the clock (getFrameLookaheadMicroseconds), so the frame that starts within a refresh is in history when the refresh draws.
• A tick that takes no frame redraws from history at its clock time (presentHeldFields), unless the last draw was at that time, from PresentationPlan.historyDrawStart, the first pass with a display-rate step: the passes that read the source and the passes before it are skipped, since no draw of the frame changes their outputs. After a new executor is installed, nothing redraws until a frame’s submission validated it.
• The presentation uniform’s refreshSeconds is the media time one refresh covers, the measured interval times the rate the clock advances at, which the frame loop hands over as 0 unless playing, and targetOffsetSeconds the clock time the draw presents at minus timestampSeconds, subtracted in double precision. The lookahead follows it, so a paused or seeking playback takes no frame ahead of the clock.
• The uniform’s frameDurationSeconds is the media time between frames (FrameInterval): the slope of a least-squares line through the last 24 new frames’ timestamps over their frame counts, which evens out millisecond-rounded timestamps to inside 0.1% within 8 frames, where a mean of 16 frames’ own durations strayed 0.2% on such a stream. The frames the loop skipped between two it drew count by the last frame’s own duration, else by the line, and a frame more than 0.4 frames off the line, as after a change of frame rate, starts the line over from the frame before it; a seek, a new generation, or a history discontinuity starts it over too.
• Telemetry’s presented media time is the frame the clock falls in: the one before the newest until the clock reaches the newest.
• A history reset clears the display-rate frames.

Black bars

• setBlackBarCropEnabled turns the black bar crop on or off, off by default; the choice outlives sessions, and a change starts detection over and builds a new plan.
• While it is on, plans end with Presentation black bars (passes/builtin/BlackBarAnalysisPass.ts), a compute pass over the route’s signal after the pass that writes the canvas, so the frame’s own passes never wait on it. It runs a workgroup for every row and every column of the frame’s source, getBlackBarLineCounts of the presentation uniform’s source size, and each reads 64 points evenly along its line, shifted along it from frame to frame. A point’s luma is the BT.709 weighting of its ‘source’ R, G, and B, where 0 is black in every transfer, kept signed, so ringing that swings both ways around black cancels in a line’s sum. A point is lit above 0.1, about 0.3 nits in PQ and 0.4 in SDR at 100 nits, and bright above 0.3, as a subtitle’s strokes are. Each line’s word holds how many of its points are lit, how many are bright, and the sum of their luma.
• Only a new frame with a source runs the analysis: the presenter hands the executor a readback buffer (PresentationFrameInput.blackBarReadback), the executor copies the statistics into it, and the presenter maps it once the GPU wrote it, through waitForGPUCompletion. Up to 3 readbacks map at once; a frame while all of them do goes unanalyzed.
• BlackBarDetector decides the bars from each frame’s words. A line holds picture when its mean luma passes 0.06, about 0.1 nits, as FFmpeg’s cropdetect reads a line, or when 2 or more of its points are bright, as a subtitle in a bar makes them. The line at each end of a frame’s picture that holds no bright point and less than half the level of the line inside it is chroma bled across the edge, not picture. An end of an axis is a sharp edge when a line within 3 lines of it at 1080 lines is lit across half of the points within the other axis’s picture. Each axis crops symmetric bars, the smaller of its two ends, so a dark region at one end never crops the other, and bars must leave at least 30% of the frame.
  • Bars crop at once on a frame that looks like a letterbox or a pillarbox: sharp edges at both ends of one axis, within 0.4% of the lines of each other, and a picture that spans the other axis. Otherwise a frame whose picture ends in a sharp edge where its bars end proposes them, and they crop once a frame of another shot agrees within 0.4% and no frame between them showed picture inside them; a shot starts after a black frame, or where the mean levels of the rows and of the columns move by more than 8 of 255 on average.
  • A frame whose picture reaches into the cropped bars, such as an IMAX scene or a subtitle in a bar, crops less at once. Once an axis cropped, it crops more only for bars agreeing shots support and no picture reached into for 30 s of playback, at the start of a shot; a step between frames counts as at most 250 ms of it, so a seek ahead counts as little.
  • A frame of another size starts over.
• The crop rounds to a standard aspect ratio (StandardAspectRatios.ts), from the table of the Detect Aspect Ratio plugin, when its picture is that close: the nearest ratio that moving the bars of an axis the crop cuts reaches, by cutting at most 0.5% more of the picture or by showing at most one line more of bar, half a line at each end. Moving the rows’ bars changes the picture’s height and the columns’ its width, by the picture’s shape on the display, its pixel aspect ratio included; where both axes could, the one that cuts picture moves, and the picture keeps its center. So a picture coded at 1920x804 presents at 2.39:1, one at 1920x816 stays as measured, since 2.35:1 would show over half a line of each bar, and one with a subtitle in a bar near no ratio stays as measured too.
• The crop applies to the layout: calculateTexturePresentationGeometry takes the part of the source the rounded bars leave as its source crop, which object-fit fits as the whole object, so a letterboxed film fills an ultrawide surface and cover zooms the picture rather than the bars. A change takes effect from the next frame; the presenter invalidates the layout and asks for a redraw, which a paused pushed session answers with a refresh.
• Telemetry’s blackBarCrop gives the bars in the analyzed frame’s lines, null while the crop is off, and blackBarAspectRatio the standard ratio the picture presents at, null while it presents as measured.

Transitions

• Seek: a new presentation generation, then controller.seek, which stops the running generation and starts a new one at the target in the same worker. The owned HEVC, AV1, and VP9 paths start at the preceding key packet, and drop the frames before the target except the one the target falls in, which presents first; a frame at the target itself presents first with nothing before it. The preroll decodes in fenced batches (see Video credits above). DTS and TrueHD use a 1 s preroll. The host drops stale seek results by its own revision.

• Audio track switch: eligibility runs again. If the source is no longer eligible, the session renegotiates; otherwise it restarts as a seek at the current time. Downmix gain changes apply live with a 20 ms ramp.

• Audio output layout switch (controller.reconfigureAudioOutput): a new device layout, force stereo, or a new downmix algorithm restarts only decoded audio while video keeps playing. The requested layout applies whole to a source a package renders as objects (audioRendersObjects); for any other source its floor is a ceiling applied to the decoded source layout, which the worker reports with audio-source-format: three channels, 5.0, and 5.1 take 5.1, 6.1 and 7.1 take 7.1 or fold into 5.1, and anything else mixes to stereo. A request that keeps the layout only records its downmix for later restarts, and a newer request replaces a pending switch. A binaural headphone change that keeps the layout switches live: session.updateAudioBinaural reaches the attempt’s package as renderBinaural before its next packet.

  1. The controller stops the output and picks a target 250 ms ahead of the clock (the current time when paused or waiting on an audio underflow).
  2. session.resyncAudio stops the old bridge and advances the audio epoch. BrowserCustomAudioOutput.reconfigure retires the worklet, sets destination.channelCount, and leases a worklet with the new channel count on the same context and sink.
  3. The worker gets resync-audio: it closes the audio decode worker’s attempt, whose channel to the replaced worklet closes at once, and opens a new one at the target with the new worklet’s channel and a rebuilt downmix, resampler, and limiter (DTS and TrueHD keep their preroll). Samples and credits are tagged with the epoch, so stale ones are dropped.
  4. Once the new epoch has buffered 100 ms, the output starts when the clock reaches the target. Audio telemetry does not move the clock in between. A clock that waited on the old output’s underflow resumes at this point, because the new output starts full and never reports a recovery. A pause holds the start, and resume counts down to the target again.

  A failure, or a fill that exceeds 5 s, falls back to HTML in the same session. The host triggers the switch, for example when the output reports sinkchange with a different channel count.

• Pause: the clock pauses and the worklet is gated to silence. The AudioContext keeps running; the pool suspends it only when idle.

• Playback rate (controller.setPlaybackRate): every generation of the controller keeps the requested rate, across seeks, track switches, play, and stop. A change applies live to the clock, the worklet’s stretcher or the <audio> element, and the presentation interval, unless it crosses the audible range’s boundary while the source has an audio track: then the generation restarts as a seek at the current time, with or without audio. A generation without audio keeps the requested audio selection aside, and seeks, track switches, and the restart back into the audible range take it up again; a downmix or layout change meanwhile only records itself in that selection. The decoded PCM startup and resync fill scales to 100 ms times the rate, at most 200 ms, the live layout switch’s lead to 250 ms times the rate in media time, and its start waits for the target at the rate.

• GPU device loss: one recovery per session. The presenter first tells its host’s device loss handler, its third constructor argument, the browser’s reason and message; a device it destroys itself is never reported. The new device must re-authorize the active HDR, Dolby Vision, or raw route before it draws. If that fails, device-recovery-failed falls back to HTML. The frame-pool route never recovers, because its decoder’s references lived on the lost device, so it falls back the same way.

• Audio sink change (WebGPUAudioOutputManager): the default sink is setSinkId(''). The fallback chain is the selected device, then the default, then the first enumerated device. A route whose sink ID is unchanged is left alone, because browsers ignore a same-ID setSinkId. Decode, worklet, and queues never restart for a sink change.

• Output device recovery: Chromium binds an AudioContext created with no output device to a fake output for its whole life. The pool detects such a context at creation and closes it on release instead of pooling it. While one is in use, or a context has reported error, the router polls enumerateDevices() once a second, because Chromium sends devicechange only with microphone permission. Each poll that lists an output runs a recovery pass, without a retry limit: it rebuilds the sink with setSinkId({ type: 'none' }) and then the requested sink, and resumes. Polling runs only while the page is visible; a hidden page stops it and probes again as soon as it becomes visible. The rebuilt sink reports sinkchange, so a surround device switches the session’s layout at once.

• Re-detect output (WebGPUAudioOutputManager.redetectAudioOutputs, offered by the host’s settings): rebuilds every AudioContext sink so the browser re-reads the device. Chromium moves a default output to a new device without telling the page, which leaves a stale channel count until a rebuild.

• End of stream: the audio decode worker flushes the resampler and limiter tails, and the worker posts ended. The controller emits ended once the bridge and worklet queues are empty, the output time has reached the end, and the video queues are empty. On the frame-pool route the worker’s last packets still decode on the page, so the session ends its video only once the GPU decoder drained its last frame.

• Audio ends first: the worker posts audio-ended (epoch-tagged) when an audio attempt finishes while video continues, and ends the run only once the video track itself has ended. The worklet’s final underflow releases the audio tail instead of starting an audio wait that would end in playback-stalled. Unlike the end of stream drain, video waits stay in force and an uncorrelated output does not pause the clock; once the tail is out, video starvation holds the clock. The end also completes a start or audio resync that has no PCM left to wait for. A native-media session sends endOfStream, so <audio> plays out, and the clock then runs without the element.

• Generation handoff: the session’s worker runs one generation at a time, and every run posts stopped last. A new start goes out right behind the previous run’s stop, and the worker starts it at once, while the stopped run unwinds beside it; session.stop() does not wait for stopped either. A stop closes the run’s decoders and audio attempt and cancels its reads, and the next run’s parser pipeline replaces the stopped run’s on the coordinator worker, so the next run never waits on the stopped one. A run’s stopped waits until the audio decode worker released the run’s decoded PCM attempts, and the page drops a retired run’s frames, packets, and audio as stale. A worker is replaced, never reused, after a generation fails, a message fails validation, or the worker crashes; a session’s start on another video decoder backend also gets a new worker, since the session’s earlier runs may still unwind in the old one. A retired run that does not post stopped within 10 s, or a retired run’s failure, replaces the worker at the next start, and such a worker terminates once it holds no run; a start that would leave more than 4 retired runs unwinding in the worker takes a new one. A lost audio decode worker fails its run’s decoded PCM as audio-output-failed, and the decode worker spawns another for its next run.

• Stop: controller.destroy stops the generation, releases the worklet and the sink lease (1.5 s cap), and ends the presenter session. Once the stopped run posted stopped, the decode worker, with its parser coordinator, parse workers, and audio decode worker, waits for the page’s next session, as the next item of a queue starts one, and terminates if none takes it within 30 s (releaseIdleCustomDecodeWorker ends the wait at once). A session on either video decoder backend takes it. The worker’s first run on another backend than its last run’s ends what that backend set up in the worker: the parser coordinator and its parse workers for the GPU decoder, or the decode fence’s device for the native one. A worker unfit for another generation terminates instead.

• Source prefetch: a host that knows the URL its next session will play, before it starts the session, calls prefetchCustomDecodeSource(source) (pipeline/CustomDecodeSession.ts), so the media’s first range read overlaps whatever comes before the start, such as the host’s own negotiation. The prefetch goes to the worker that session takes: the idle one, or a session’s worker, which waits for the prefetch until the session is destroyed and takes it then, behind its runs’ stop, or else a new worker, whose startup it overlaps too. A session on any video decoder backend takes that worker once it holds no run. In the worker, prefetch-source retains a UrlSource for the URL and starts its first read, the read at byte 0 that any run’s input begins with; a worker whose run still holds it ignores it, but a stopped run, whose reads were canceled, does not hold it while it unwinds. The session’s first run over the same URL and pattern acquires that source and continues on its response, so it sends no second request for the head. A run over another URL releases it, as does release-prefetch with the prefetch’s ID, which the handle’s cancel posts and which the worker ignores once a run took the source over; no run ever reads another URL’s bytes. A source no run takes within 5 s is released, a failed read releases it at once, a later prefetch replaces an earlier one, a prefetch that waits 5 s for a session to be destroyed gives up, and a session that takes no idle worker, as one that starts before the destroyed session’s worker parks, ends the prefetch at once.

Fallback and renegotiation

pipeline/CustomPlaybackController.ts:getFallbackDisposition decides what a failure does:

• The host carries out each disposition, and may renegotiate on every reason when its HTML player cannot play the source.
• Failures in the host’s wrapper (presenter, frame submission, rAF) are lifecycle-failed.
• Fallback never selects another player.

Every stale callback is dropped by a generation or revision check:

• The host: its own generations and revisions.
• The controller: activeGeneration and fallbackGeneration.
• The session: a generation record per generation, which its worker’s messages must match.
• The worklet: its flush generation.
• The presenter: deviceResourceEpoch, the color and layout revisions, and fallbackLatched.

Gotchas

• External textures of high-bit-depth frames are 8 bits per channel in Chromium (see Decisions). Authorization tolerances follow: external HDR 10/255, external Dolby Vision 8/255, raw 3/255.
• The native external HDR route rewrites the color of the hvcC SPS and of every in-band SPS, in delta packets too, to BT.709 at the stream’s own range, and the shader recovers the 10-bit codes (Y*876+64, C*896+512), a full-range code as its limited-range equivalent. A frame with a colorSpace that is not neutral, BT.709 at either range, latches decoded-frame-color-mismatch.
• The raw route checks each frame’s colorSpace against the metadata, and a null member is unspecified, so it never contradicts. SDR also matches a smpte170m transfer, and HLG a bt709 transfer on BT.2020 primaries.
• WebGPU presentation on the HTML path is SDR only: a <video> external texture is browser-converted sRGB.
• A deinterlaced native frame’s chroma is the browser’s progressive upsampling, which blends the two fields’ chroma rows; only the raw and frame-pool routes read chroma within each field.
• Worker track indices are container ordinals, not Jellyfin MediaStream.Index; the streams Jellyfin lists for external files (IsExternal) take none.
• Every time the page and the worker exchange is a playback time, which starts at 0 where the host’s positions start. A container on a program clock, MPEG-TS, M2TS, or an MPEG program stream, stamps its packets past zero, 600 s on a Blu-ray title, so the worker counts a run’s times from the first timestamp of the tracks it plays (demux/PlaybackTimelineOrigin.ts). It adds that origin to the start and every resync target before it reads the tracks, and takes it off every frame, packet, decoded PCM input, native audio fragment, startup progress report, and container duration it posts. The audio decode worker and the worklet run on the playback timeline; the queues the worker keys by packet timestamps, the Dolby Vision RPU and HDR10+ queues and the EL pairing, keep container times.
• Every seek, audio switch, and paused repaint starts a new run in the session’s worker, and each run opens its input again, over the last run’s URL source when it reads the same URL (demux/URLSourceRetention.ts). The source keeps its cache, with the container’s header pinned in it and its index once read, and its open responses, so a seek reads the header and index from memory and reads on near where the last run stopped. Matroska Cues are read at the first lookup past the start, so a run that starts at 0 never reads the end of the file. An MP4 or MOV movie box is read without the sample tables of a large track other than the first video and audio track, which that track reads at its first lookup and pins like the Cues, unless the source only reads forward. A run’s stop cancels the reads its input still waits for; a failed run’s source, or one no run takes within 5 s, is released, and only by the run, or the prefetch, that took it last. After the first preparation over a source, its read-ahead holds 5 s of media and its cache 15 s, at the media’s size over its duration, within 8 to 64 MiB and 32 to 192 MiB; a file’s cache stays 32 MiB.
• Decoded PCM gain (volume cubed times normalization) is applied after the limiter and is not capped, so a boost can clip; clipping is counted. The native-media route caps gain at 1.
• DecodedVideoGeometry locks the first decoded size of a run, with a 64 px tolerance. A later size change fails the session.
• The negotiated maximumCoded* is Jellyfin’s cropped Width and Height, while containers and decoders report block-aligned sizes (mkvmerge can declare 2080 lines for a 2076-line HEVC picture). Route checks allow the same 64 px through exceedsNegotiatedCodedSize.
• A media server’s cross-origin 206 may hide Content-Range, as Jellyfin’s stream endpoint does. Such a response passes only from a path the URL source’s hiddenContentRangePathPattern names, which the host sets, and only with a Content-Length within the requested range (demux/HTTPRangeResponse.ts); a source without a pattern needs every 206 to show its Content-Range. The Jellyfin plugin sets the pattern of /Videos/{id}/stream on the stream URL it plays.

Eligibility and routes

This chapter follows a source from the engine’s capability evidence to a concrete decode and presentation route. The host turns the same evidence into what it advertises to its server; the Jellyfin plugin’s book describes its device profile.

Host entry points

A host calls these, in this order, for each playback:

1. CustomPlaybackEligibility.hasPotentialCustomPlaybackVideoRoute, the metadata-only prefilter, when it chooses a player. The prefilter declines a source whose containers no route in CUSTOM_CONTAINER_CODEC_RULES carries, such as FLV, Ogg, or an MPEG program stream. It also declines a video codec no route decodes, such as Theora, rotated video, and interlaced HEVC and JPEG 2000, which code their fields apart (see Interlaced video).
2. CustomPlaybackRuntime.getCustomPlaybackRuntimeAvailability, which fails without a secure context, Worker, navigator.gpu, or VideoFrame.
3. probeCustomDecodeCapabilities(item), the native-media audio probe, and the GPU authorizations the item needs (5 s each), before it advertises anything; see Probes and caching.
4. hasEligibleCustomVideoRoute, to ask whether the runtime would present one item with the measured capabilities and authorizations. It takes the GPU decoder packages a playback could use as gpuVideoDecoderSummaries, since offers need the packages and the presenter’s device; see GPU decoders.
5. getCustomPlaybackEligibility, when the session starts, with the route flags and authorized keys of CustomPlaybackEligibilityOptions. allowRawSDR needs only a settled :sdr raw key; a key outside BT.709 authorizes on first use, so getCustomPresentationEligibilityOptions waits for the presented stream’s own key (getFirstUseRawSDRRouteKey). gpuVideoDecoders holds the GPU decoder packages the host offers, from PresentationSetup.getGPUVideoDecoderOffers, and allowFramePoolHDR lets them present HDR; see GPU decoders. audioDecoders holds the host’s audio decoder packages; see Audio routes.

Eligibility

getCustomPlaybackEligibility checks in order:

1. parsePlaybackSource: DirectPlay, not live, a custom container, and a media source: a File, or a credential-free http(s) URL, which it resolves against the page. A URL source keeps the host’s hiddenContentRangePathPattern, the paths whose cross-origin 206 hides Content-Range (see Architecture). A missing or zero RunTimeTicks is an unknown duration, not a rejection. Playback then ends with the streams, audio takes the decoded-PCM route because the native media backend needs a duration, and the worker reports the container’s duration on ready.
2. selectVideoStream, and a codec the engine names. A codec it knows but nothing decodes, Theora, reads video-codec-without-route; any other unnamed codec, such as H.263+ (h263p), reads codec-unsupported.
3. Rotation 0, and a codec that keeps an interlaced stream’s fields together in frames (codesInterlacedFieldsApart, from presentsInterlacedVideo).
4. selectVideoOutput (below), and a route that presents the stream’s scan (refusesVideoScan).
5. selectPlaybackAudio.
6. supportsCustomContainerCodecCombination.
7. The runtime budget.

An ineligible result carries its reason, and the host falls back.

The worker refuses what only the file shows, with a message that names the check: a track whose edit list Mediabunny cannot honor (getUnsupportedFeatures()), an AVI video track that outruns its index (avi-index-incomplete), a video track Mediabunny names no codec for that no frame-pool route takes (no decode route: fourcc: <FourCC> for an AVI or ASF track, codec-id: <ID> for another container’s), and a DTS Express audio track (no decode route: dtse). On the frame-pool route an MPEG-TS, M2TS, or MPEG program stream track of MPEG-1, MPEG-2, or VC-1 must carry the negotiated codec (container-codec), and takes its first sequence header’s extradata as codec private data (sequence-header-missing when none of its first 64 packets has one); the worker tells these MPEG system streams by their input format (isMPEGSystemStreamFormat). A VC-1 or WMV3 track must be a Matroska V_MS/VFW/FOURCC track, an ISO base media vc-1 sample entry (VC-1 only), or an ASF stream of WVC1 or WMVA (VC-1) or WMV3 (codec-id); an AVI stream of those FourCCs is refused, as no container rule places VC-1 in AVI. An ASF stream’s FourCC and extradata come from its BITMAPINFOHEADER, and its media objects, in decode order like a Matroska VfW track’s, are restamped in presentation order by VC1PresentationOrder.ts. An MPEG-4 Part 2, H.263, or MS-MPEG4 track must have a codec ID of the negotiated codec (codec-id): Matroska’s V_MPEG4/ISO/SP, ASP, or AP and ISO base media’s mp4v for MPEG-4 Part 2, which Mediabunny names mpeg4 (container-codec otherwise), s263 or h263 for H.263, V_MPEG4/MS/V3 for MS-MPEG4 v3, and Matroska’s V_MS/VFW/FOURCC, AVI, or ASF for any of them. A VfW, AVI, or ASF track’s FourCC must be one FFmpeg’s riff.c maps to the negotiated codec, compared upper-cased (fourcc), and Mediabunny leaves every such AVI or ASF track but MPEG-4 Part 2 codec-less, with the FourCC as its codec ID. A codec-less AVI or ASF track’s extradata is the one after its BITMAPINFOHEADER (demux/VFWStreamFormat.ts), as FFmpeg’s AVI and ASF demuxers pass it. The codec private data is a 40-byte BITMAPINFOHEADER of the track’s coded size and FourCC (mp4v, s263, or MP43 for a track without one), then the track’s extradata (see Decoder packages). AVI, ASF, and Matroska VfW stamp packets in decode order, so MPEG4PresentationOrder.ts restamps an MPEG-4 Part 2 track from them that FFmpeg outputs one VOP late: its video object layer, from the extradata or else the first key packet, clears low_delay in its vol_control_parameters, or has none and is neither Simple nor Advanced Simple, or has none and its first group holds a B-VOP. An I, P, or S-VOP then takes the time of the next coded VOP that is no B-VOP, passing over N-VOPs, for which FFmpeg outputs nothing, and a B-VOP keeps its own. A JPEG 2000 track must be an MJ2 mjp2 sample entry or a Matroska V_MS/VFW/FOURCC track (codec-id), whose FourCC must be one riff.c maps to JPEG 2000, compared upper-cased (fourcc), such as the mjp2 FFmpeg’s Matroska muxer writes. Its blocks are bare codestreams, every one a key packet stamped in presentation order, and the decoder takes no codec private data, as FFmpeg’s decoder reads no extradata. A Matroska attachment, such as a cover image, is no track, so it never shifts the video track ordinal. AVI audio takes the same routes as any container’s: Mediabunny reads a WAVE_FORMAT_EXTENSIBLE track’s codec from its SubFormat, and gives FLAC and Vorbis the decoder descriptions their strf extradata holds. A track whose container declares a WAVE channel mask, which Mediabunny’s getChannelMask reports for WAVE_FORMAT_EXTENSIBLE, is laid out by that mask: the worker takes its layout from CustomWaveChannelLayout.ts and passes the mask to the audio decode worker’s attempt, whose output stage binds Mediabunny’s decoded samples to it. So a three-channel 2.1 track feeds its third channel to the LFE, and 3.0(back) splits it across the surround pair, as FFmpeg’s decode does, where three channels without a mask are 3.0. A mask of 0, or none, lays channels out by their count; a mask the output stage has no layout for, or one naming more or fewer speakers than the track has channels, is refused as The selected audio channel mask is unqualified. Without a container mask, the host’s audioMetadataLayout (Jellyfin’s count and, where the metadata names a layout no count implies, its mask) screens the track: a metadata mask lays out the channels when it names the container’s count, and Jellyfin’s count replaces a container count that has no layout (demux/CustomAudioTrackMetadata.ts); a track neither lays out is refused as The selected audio channel layout is unavailable.

Probes and caching

• Main thread: WebCodecs configuration and decoded-output probes, raw copyTo fingerprints, the per-profile H.264 probe, and the MSE AC-3/E-AC-3 probe. The engine has no decoder of its own to probe: Mediabunny’s PCM decoding is always supported, and a codec only a package decodes reads unsupported with the reason no-engine-decoder.
• GPU: presentation authorization reads back renders of the production shaders. Results are cached per GPUDevice, canvas format, and shader signature, and dropped on device loss. A timeout or device-lost decision stays visible but is not kept: the next authorize call probes the route again (isRetriedAuthorizationDecision), since a deadline that counted time behind other runners is no verdict. Every runner, and the canvas readback, pushes and pops its validation error scope in the same task, around its submit (captureValidationErrors, submitAuthorizationFrames), and awaits the popped result later, because the runners at device creation share the device’s one scope stack and a late pop could take another runner’s scope.
• External HDR authorization opens a hardware decoder (HEVC hvc1.2.4.L120.B0 at 1920x1088, AV1 av01.0.08M.10 at 1920x1080, prefer-hardware), so its isConfigSupported check and its decode run as recorded steps of the page’s probe session, under the probes external-hdr-authorization:hevc and external-hdr-authorization:av1, with labels such as check hvc1.2.4.L120.B0 1920x1088 prefer-hardware. A crash in them is quarantined like a capability probe step’s, and a blocked or defective step rejects the route with the reason stopped-page or browser-defect. The decode step’s call decodes, imports, renders, and reads back within the manager’s step timeout (CapabilityProbeManager.timeoutMilliseconds), and returns decisions, never the frame. The routes of one codec and target format asked for in the same task, such as the HEVC PQ and HLG keys, share one check and one decode, and a device runs a codec’s batches one after another, so it never holds two of that codec’s decoders. A deadline that passes during the check opens no decoder.
• External Dolby Vision authorization returns video-frame-creation-failed before compiling anything when canHoldVideoFrameFormat('I420P10') is false, as on shipping Safari or without VideoFrame. The other authorizations open no decoder, so they are not recorded.
• selectCustomDecodeProbes(item) picks the probes an item needs. Every audio probe runs for every item, because a playing item can switch to any of its audio tracks: the stereo probes (native-audio), the 5.1 probes (native-surround-audio), and the 7.1 probes (native-seven-point-one-audio). The video probes follow the union of the video streams across the item’s sources: the codec picks its native probes, a range extension picks its own variants, an H.264 stream beyond 8-bit 4:2:0 picks every raw format variant its profile codes at its depth beside the per-profile probe, since a host advertises the profile at that depth only when all of them qualify, a stream beyond 8-bit SDR adds the raw-plane and native HDR probes, and a range that is neither SDR nor static HDR adds the Dolby Vision probe. JPEG 2000, MPEG-2, VC-1, MPEG-4 Part 2, H.263, and MS-MPEG4 streams pick none, since only a GPU decoder package decodes them. An HDR item also runs the HEVC and AV1 probes, because an HDR transcode targets those codecs and their ranges bound it. An item without stream metadata, or with a video stream that names no codec, runs every probe.
• A probe the item does not select reads not-probed, which no profile rule or eligibility check treats as support, and its H.264 profile result is absent; probeStates records which probes a result ran. At eligibility, the host keeps the negotiated result when hasProbedCustomDecodeSelection says it covers the played source, and otherwise probes that source.
• A range-extension or raw-format probe downloads its vector before its decode step joins the probe manager, under its own 30 s bound (capability/CapabilityAssetLoading.ts), so a slow link never reads as a slow decoder or trips a step’s timeout. A download that fails with a network error, 408, 429, or 5xx is retried after 250 ms and again after 1 s, inside the same 30 s bound; any other HTTP failure, or the bound ending, is final and reports asset-unavailable with an unknown status.
• Before its download, a raw HDR, range-extension, AV1 raw format, or H.264 raw format probe asks canHoldVideoFrameFormat(format) (capability/VideoFrameFormatSupport.ts), which builds a 2x2 VideoFrame of the format. A TypeError means the browser has no such pixel format, as every shipping Safari lacks the 10-bit and 12-bit formats, and the probe reads unsupported with the reason output-copy-unsupported without downloading or decoding.
• An audio probe copies each AudioData in its own format (copyAudioDataPlanes in capability/AudioDataPlanes.ts): one copy for interleaved data, one per plane for planar data, converted to float in JavaScript. It never asks the browser to convert a sample layout, since Safari 26.0 to 26.3 ends the page in copyTo(..., { format: 'f32-planar', planeIndex }) with a plane index of 2 or more on its interleaved multi-channel output (WebKit bug 302521).
• A step that matches an entry of capability/BrowserDefects.ts never runs and reads unsupported with the reason browser-defect. The table holds WebKit bug 308322: AAC decoding with more than 2 channels on WebKit before Safari 27, or of unknown version, may end the page, since AudioToolbox can return no format description for the raw AudioSpecificConfig. An iOS 26 user agent freezes the OS version at 18.x and a web view has no Version/, so such a WebKit counts as unknown.
• Each probe is one promise per page, so a later item runs only the probes no earlier item ran. It is dropped only when the store clears a stored verdict it holds, or replaced when a background rerun reaches a new verdict. capability/CapabilityProbeManager.ts runs the browser calls of every probe, and the MSE AC-3/E-AC-3 playbacks, side by side; see The probe manager.

The probe manager

Every browser call is a step: each isConfigSupported check, each decode, each H.264 profile’s check and decode, and each MSE layout playback. One CapabilityProbeSession per page (capability/CapabilityProbeSession.ts, from getDefaultCapabilityProbeSession() or a probe’s getSession()) owns the manager, and every probe’s steps go through its record(queue, probe) wrapper; see Probes that stop the page.

• capability/CapabilityProbeSteps.ts describes each step with a label naming the exact call and a cost { bytes, decoders }. Video labels read <check|decode> <codec> <W>x<H>, then the hardwareAcceleration unless it is no-preference, then the copy format; audio labels read <check|decode> <codec> <N>ch <R>Hz, and MSE labels play <codec> <N>ch <R>Hz. Labels are printable ASCII, at most 96 characters (CAPABILITY_PROBE_STEP_LABEL_MAXIMUM_LENGTH).
• A check costs 0 bytes and 1 decoder. A video decode costs the frame layout at coded size times the codec’s maximum reference frames plus one, plus one copy frame when it copies: 16 for H.264 and HEVC, 8 for VP9 and AV1, and 3 for VP8. An audio decode costs its decoded f32 PCM.
• Negotiation steps, and a promoted background run’s, run in parallel while the running decoders stay within the decoder limit and the running bytes within the byte limit; a step larger than the limits runs once nothing else does. The byte limit is CAPABILITY_PROBE_DEFINITION_BYTES, the largest decode cost among the steps of every probe definition (getLargestCapabilityProbeStepCost), the raw HEVC 4K decode, so parallel probing never holds more than the heaviest single test would alone. The decoder limit is navigator.hardwareConcurrency, or 1 without it. The session lowers each to the stored ceiling’s value in that dimension when it has one (CapabilityProbeCeiling, each of bytes and decoders a number or null for uncapped), and the decoders to 1 while suspects remain; it reads the limits again once the last suspect settles.
• Negotiation steps, and a promoted background run’s, go before background steps; among them, a step that does not fit never holds back a later one that does. A negotiation’s video decodes join the manager only once its audio probes settle (manager.after), so a video timeout never costs audio its verdicts; checks join at once.
• A step of a background run that is not promoted, read at admission, runs alone: it starts only when nothing runs and no negotiation step waits, and no other background step joins it, checks included. A negotiation step may join a running background step, so playback never waits on the warm-up, but a background crash still names exactly one call when no negotiation runs. A promotion takes effect at once (CapabilityProbeBackgroundRun.promotion): the run’s waiting steps join the negotiation steps without waiting for a running call to settle.
• A step enqueued with { alone: true } runs by itself: it starts once nothing runs and no step ahead of it waits, and nothing joins it. A negotiation’s other steps go ahead of it and it goes ahead of background steps, except that a background step that runs alone goes last.
• A decode step may pass confirm(value), which accepts a negative result. A call that overlapped any other call and settles before its timeout with a value confirm accepts runs again alone, timed from its own admission, and its caller gets that run’s result. A call that ran alone, a timeout, and an error are never rerun. A hardware decoder pool may refuse a decoder only because other calls hold theirs, and a refused decode would otherwise be stored as unsupported for the fingerprint. Every decode step passes it: native video and Ultra HD, HDR and Dolby Vision frames, raw HDR and the exact raw formats, audio, each H.264 profile decode, and each MSE playback; configuration checks do not.
• A step holds its reservation until its call settles, not until its timeout, so work the browser still does keeps counting. Checks and decodes (check, runTimed) time out 2 s after they start, so queue time never counts, and resolve CAPABILITY_PROBE_TIMEOUT. A timed call still unsettled one more timeout after its own stops the page’s probing: every waiting and later step reads as a timeout until the page reloads.
• The H.264 profile output probe also closes its own decoder after 2 s (H264_PROFILE_OUTPUT_PROBE_TIMEOUT_MILLISECONDS), since the manager cannot close a decoder.
• An MSE playback (NATIVE_MEDIA_AUDIO_PROBE, native-media-audio) joins through the untimed run, since each playback bounds its own stages at 3 s; its cost is its buffered f32 PCM.
• The raw HDR output probe waits for a pending copyTo before it settles, and closes a frame only after its copy settles.

Background probes

A host may start the probes before its first negotiation with warmCustomDecodeCapabilities(), and tells the engine when media plays with configureCustomDecodeCapabilityBackground({ isPlaybackActive, refreshAfterUse }). Each call replaces the previous configuration; refreshAfterUse, which defaults to true, decides whether a stored verdict a negotiation used reruns in the background (see Stored verdicts).

• The warm-up runs every probe without a page result or stored verdict, one probe after another, in the order a full run starts them. A client whose navigator.connection.saveData is set, or whose navigator.hardwareConcurrency is below 4, gets no warm-up.
• Before a background probe starts, and before each of its run and runTimed steps joins the manager, it waits for its turn: a requestIdleCallback with a 2 s timeout (BACKGROUND_PROBE_IDLE_TIMEOUT_MILLISECONDS), then no media playing, no negotiation under way, and none settled in the last 10 s (BACKGROUND_PROBE_NEGOTIATION_HOLD_MILLISECONDS); a held step looks again after 1 s. Its checks skip the wait, but still run one at a time like its other steps. Without requestIdleCallback, as on WebKit, the wait is the whole 2 s. The wait comes before a step joins the manager, so it never counts against the step’s timeout, and it never holds back a negotiation’s steps.
• A background probe downloads its range-extension vector with fetch(..., { priority: 'low' }), retries included.
• probeCustomDecodeCapabilities promotes a background probe it needs: its remaining steps skip the wait and take a negotiation’s admission at once, and its later downloads take the default priority; a download already under way keeps its priority. A probe no background run started runs at once, as without a warm-up, so a negotiation shares the limits only with the one background step that may be running.
• A background step that times out marks only its run and halts the background for the page: no later background step runs, and the run’s result, like any background result with a probe-timeout part, is discarded rather than kept as the page’s result. A negotiation that needs the probe runs it again.
• A background run of a probe that holds suspect steps runs after the others; see Probes that stop the page.
• runCustomDecodeBackgroundTask(task) runs a host’s own work in the same background: after every background probe queued ahead of it, one task at a time, each at a turn a probe step would take. No negotiation promotes a task, a task that rejects counts as run, and the promise resolves whether the task ran; a page whose background halted drops it. warmGPUVideoDecoders runs its compiles this way; see Decoder packages.

Stored verdicts

capability/CapabilityProbeStore.ts keeps verdicts across page loads and browser restarts, in one localStorage entry, webgpu-player-capability-probe-verdicts.

• A host names its build with configureCapabilityProbeStore({ buildKey }), as it calls configureEngineAssets; until then nothing is read or written.
• The entry holds a schema version (CAPABILITY_PROBE_STORE_SCHEMA_VERSION), the fingerprint, the time of its last write, and each probe’s result by its CustomDecodeProbe name.
• The fingerprint is the build key, the default adapter’s info, its sorted features, and its limits, navigator.userAgent, and Chromium’s getHighEntropyValues(['fullVersionList', 'platformVersion']), compared as JSON with sorted keys. A fingerprint that cannot be read within 2 s disables the store for the page.
• An entry of another schema or fingerprint is a miss, and the next write replaces it, so an update leaves no other entry behind.
• A result is written only when every status in it is supported or unsupported; one with any unknown or not-probed part, from a timeout, an exception, or a missing API, is not. Writes merge, so the video probes of each item a page negotiates join the entry.
• CustomDecodeCapabilityProbe reads a probe’s stored verdict before running it. A hit answers at once and schedules the probe to run again in the background, which overwrites the stored verdict and the page’s result when it reaches a verdict; a rerun that reaches none leaves both. With refresh after use off (refreshAfterUse: false, or the probe’s refreshAfterUse option and setRefreshAfterUse), a hit schedules no rerun, so a negotiation it serves makes no browser call and no write. Turning it off schedules no further rerun, but a rerun already queued still runs; turning it back on reruns the hits from then on.
• A start that falls back with decode-failed or startup-timeout before its first frame clears the entry and the stored verdicts the page holds, when the page used any (clearAfterStartFailure, from CustomPlaybackController), so the renegotiation probes again. Driver updates, OS codec packs, the hardware acceleration setting, and Chrome’s software fallback after a GPU process crash change no fingerprint.
• clearStoredCapabilityProbeVerdicts() clears it, quarantine included, and readStoredCapabilityProbeVerdictState() returns the stored fingerprint, write time, age, probe names, blockedSteps, suspectSteps, ceiling, and crashedProbes, the blocked and suspect steps grouped by probe, each a CrashedCapabilityProbe with its probe name and step labels, sorted by probe.
• onCapabilityProbeCrash(listener) calls the listener on the page that finds a stopped page, once per finding, after the entry records it, with each probe and the steps it had in flight; it returns the function that removes the listener. A listener that throws stops neither the recording nor the other listeners. A store of its own takes listeners with onCrash.
• The MSE AC-3/E-AC-3 probe and the GPU presentation authorizations are not stored: the first is not a CustomDecodeProbe, and the second are measured on the presenter’s device, which keeps them for that device.

Probes that stop the page

A probe whose browser call stops the page, as a WebContent crash on WebKit does, never returns, so it reaches no verdict and the next page would run it again. The store therefore keeps a run journal beside the verdicts, and quarantines the steps that were in flight when a page stopped, per fingerprint.

• CapabilityProbeSession.record(queue, probe) (capability/CapabilityProbeSession.ts) is the one wrapper every probe step goes through, the MSE playbacks under the probe native-media-audio included. A step that matches a declared browser defect, or whose label the quarantine blocks, never joins the manager and resolves a CapabilityProbeBlock whose reason is browser-defect or stopped-page; the probe reads it as unsupported with that reason, which is a verdict the store keeps. A suspect step joins the manager as a suspect; any other step joins as usual. Once the manager admits a step, the wrapper records it on the store (markRunning(probe, step, cost)) with its label and cost, and ends the record once the call settles, even past its timeout. Downloads, idle waits, and steps waiting in the manager are not recorded, so a background run’s record covers only the moments the browser works on it.
• Every probe run, negotiation, background, or MSE, first calls the session’s readQuarantine(), which reads the store’s quarantine and sets the manager’s limits from it.
• Each page keeps its record, the probes it has in a call, each with the labels of the steps it has in flight, the in-flight totals of bytes and decoders, and its fingerprint, under its own random id in the localStorage entry webgpu-player-capability-probe-runs, and drops it once its last call ends. The page counts its open calls by probe and step, so concurrent calls of one step each appear once, and a step leaves the record once its last call ends.
• A hidden page records nothing, since a browser may end a hidden page without a crash: the page drops its record as it hides, and as it unloads or enters the back-forward cache (pagehide), whatever a later visibility change reads, and writes it again as it shows.
• A page holds a Web Lock of its own id while it has a record: it takes the lock before it writes the record and lets go of it after it drops the record, so an idle page holds none. The browser frees a page’s locks when the page ends, however it ends, so a record whose lock is free is a stopped page’s.
• A page’s first quarantine read (readQuarantine on the store) takes the record of every other page whose lock is free, holding that lock while it does, so no page of that id writes one meanwhile, drops the stored verdicts of the probes in it, and calls the crash listeners; a record of another fingerprint goes with no crash, and a page whose lock is held still runs, so its record stays. Without Web Locks, or when the lock manager takes more than 2 s, the page records nothing and reads no crash.
• A stopped page with one distinct label in flight blocks that label: in every probe it reads unsupported with the reason stopped-page and never runs again for the fingerprint. A label already blocked only adds the probes that had it in flight.
• Several labels in flight become suspects. The next round lasts while suspects remain, with the decoder limit at 1, and retries the suspects alone: the session enqueues each with { alone: true }, so it starts only once nothing runs and no step ahead of it waits, and the background runs first the probes that hold no unsettled suspect (holdsSuspectSteps(probe)). A suspect’s call never overlaps another, so it is never rerun to confirm a result. Once the last suspect settles, the session restores the limits, with any new ceiling, even in the middle of the round. A suspect that stops the page alone is the culprit: it is blocked, and leaves the suspects; the others clear as they settle (resolveSuspect). A step that was no suspect and stops the page alone during the round is blocked too, but explains nothing of the suspects’ crash, so the round can still end in a ceiling.
• When the last suspect clears and no suspect was the culprit, the crash came from concurrency, and the store lowers the ceiling below that crash’s in-flight totals, each dimension on its own. A dimension the crash held some of is capped one below its total, never above an existing cap; a total of 0, such as the bytes of a crash with only checks in flight, is no evidence and leaves that dimension’s cap, or none. When a new crash comes before an earlier one is explained, the round keeps the lower of each total. A ceiling with neither dimension capped is null; the manager still runs one step alone whatever the ceiling.
• The store’s schema version is 2; an entry or record of version 1 is a miss.

Route catalog

Video route selection

selectVideoOutput tries these in order.

1. A Dolby Vision descriptor. PresentationInput.getDolbyVisionPresentationSelection accepts any integer profile with the base layer flag set, a valid bit depth, and any 4-bit compatibility ID (CCID) or none; the EL flag never rejects. The descriptor names the RPU route (reconstructionProfile): Profiles 4, 5, 7, and 8 as signaled; Profiles 10 (AV1) and 20 as 5 with CCID 0 or none and as 8 otherwise; none for Profile 9, the retired profiles, or a stream without an RPU. RPU reconstruction ignores the CCID, which only declares a base layer that can be shown on its own. The Dolby Vision routes, in order:
   1. Native external Profile 5: a 10-bit HEVC P5, or a P20 that reconstructs as P5.
   2. Native external AV1 Profile 10 (selectNativeAV1DolbyVisionOutput): a 10-bit Main P10 reconstructs as P5 or P8 from the hardware decoder’s frames, every sequence header neutralized to limited BT.709, whatever base the CCID declares. It needs nativeHDRAV1, the AV1 PQ external key, which proves the AV1 decoder’s frames import exactly, and the external key of the profile it reconstructs as: allowNativeDolbyVision for P5, allowNativeDolbyVisionProfile8 for P8. getCustomPresentationEligibilityOptions waits for the AV1 PQ key and the P8 key on first use.
   3. The native compatible base, HEVC only, when the CCID declares one: HDR10 (1) or Ultra HD Blu-ray (6) for P7 and P8, and HLG (4) for P8. The RPU and EL are ignored. This needs exact BT.2020 limited base metadata with explicit transfer, primaries, and matrix. A separate-track P7 base is a standalone HDR10 track, so it qualifies whatever CCID the EL track reports.
   4. Raw RPU reconstruction. The base layer decodes natively to raw-planes: I420P10 for HEVC Main 10 and for AV1 Main at 10 bits, or the exact format of any range-extension variant or AV1 raw format variant, such as I420 for AV1 Main at 8 bits. HEVC Main at 8 bits has no native raw route, so its RPU route is ineligible. Single-layer routes (P5, P8, P10, P20) discard a signaled EL. No decoder decodes an EL, so dual-layer routes (P4, P7, HEVC only) reconstruct from the base layer alone: MEL is exact, and FEL presents its base layer. The worker still pairs a separate EL track to read its RPUs, and the presenter keeps its FEL composition for an EL decoder, which no route reaches.
   5. The declared base layer (getDolbyVisionBaseColorMetadata), through the ordinary steps below, where a GPU decoder’s RPU reconstruction (step 6 of this list) comes before the base’s own GPU route. CCID 1 or 6 declares PQ, 4 HLG, and 2 SDR, never for P5 or a P20 that reconstructs as P5. An explicit ColorTransfer must agree with the CCID; VideoRange and VideoRangeType are ignored.
   6. Without a declared base, RPU reconstruction from a GPU decoder’s frame pool (see GPU decoders).
   7. Otherwise the source is ineligible, with the reason from the reconstruction step.
2. No color metadata: metadata-unsupported. PresentationInput.parseVideoStreamColorMetadata reads ColorTransfer, ColorPrimaries, ColorSpace, and ColorRange. A field that is unknown, unspecified, or reserved is absent and takes its transfer’s default: BT.709 for SDR, BT.2020 for PQ and HLG, and limited range. The BT.2020 10 and 12-bit transfers read as SDR, SMPTE 170M and SMPTE 240M primaries as smpte170m, BT.470 BG primaries as bt470bg, and the SMPTE 170M and BT.470 BG matrices as BT.601. Any other value is unrecognized and fails.
3. An HEVC range extension needs its exact variant capability and raw key, and so does AV1 outside Main 10 (High 4:4:4, Professional 4:2:2 or 12 bits), through av1RawFormats[variant]. So does H.264 beyond 8-bit 4:2:0 through h264RawFormats[variant]: a profile beyond High (High 10, High 4:2:2, High 4:4:4 Predictive, their Intra profiles, High 4:4:4, or CAVLC 4:4:4) whose pixel format is one of the five variants, I420P10, I422, I422P10, I444, or I444P10, and one the profile codes. The SDR keys go by matrix, primaries, and range, and 10-bit PQ or HLG takes the raw HDR key of the variant’s format. A range extension or a raw format variant never falls through to steps 4 and 5; a profile beyond High at 8-bit 4:2:0, 9, 12, or 14 bits, 4:0:0, or RGB has no variant and no other route, so it is refused.
4. SDR: native, or native Main 10 SDR. Then, for 10-bit 4:2:0 SDR that neither decodes (AV1 Main, VP9 Profile 2, or HEVC Main 10 without native decode), raw I420P10 through the raw SDR key of its matrix, primaries, and range. JPEG 2000, MPEG-1, MPEG-2, VC-1, WMV3, MPEG-4 Part 2, H.263, and MS-MPEG4 have no native route (CUSTOM_PACKAGE_VIDEO_CODECS).
5. PQ or HLG: native external first, through the stream’s codec’s key (external-hevc-main10-bt709-limited:<pq|hlg>-v1 for HEVC Main 10, external-av1-main10-bt709-limited:<pq|hlg>-v1 for AV1 Main at 10 bits), then raw (<I420P10|I420P12>:bt2020-ncl:bt2020:limited:<pq|hlg>). The worker rewrites the HEVC SPS VUI to BT.709 at the stream’s own range, or every AV1 sequence header, the codecs string, and an av1C description to limited BT.709, so Chrome converts the hardware frame without tone mapping; each rewritten HEVC SPS must declare BT.2020 with the non-constant-luminance matrix and the route’s transfer, at either range, and each AV1 header limited BT.2020 with them, or the attempt fails.
6. When steps 3 to 5 chose no route, a GPU decoder package the host offers takes the stream, if one does; otherwise the source is ineligible with the reason those steps gave.

GPU decoders

The gpu backend decodes on the presenter’s device with a GPU decoder package’s WGSL; the frame-pool output presents its pool slots without a copy. A package may carry a parser module (//! parser:), which the selection carries to the decode worker’s start as gpuVideoParser: the HEVC decoder project’s parser parses each packet into the GPU record its WGSL reads. PresentationSetup.getGPUVideoDecoderOffers offers the host’s packages that accept the stream’s codec and write its output format, in the host’s order, with the presenter’s buffer limit; it verifies nothing. selectGPUVideoOutput takes the first offer for the stream’s codec that meets each of these:

• The output format: a range extension’s own, else the one that holds the stream’s pixel format (getGPUDecoderPixelFormatHolding: yuv422p is I422, and 9-bit samples take the 10-bit format, shifted up), else 4:2:0 at the stream’s bit depth. An explicit BitDepth must agree with the pixel format’s samples, and a pixel format no frame pool holds, such as RGB or 4:1:1, has no route. JPEG 2000’s xyz12le, the Digital Cinema profiles’ X’Y’Z’, takes I444P12, into which the JPEG 2000 package writes it as limited-range BT.709 Y’CbCr, so it needs a stream whose color metadata is BT.709 SDR at limited range, as an untagged stream’s is.
• For a 4:0:0 pixel format, such as gray10le, which takes the 4:2:0 format of its depth: a decoder whose header declares monochrome.
• The decoder’s stream constraint, over the pool’s coded size and the format’s bit depth. The pool takes the container’s coded size rounded up to 64 (GPU_DECODE_POOL_CODED_SIZE_ALIGNMENT), the largest coding block, and the decoder’s header crops it, such as by a conformance window. A session may code larger, at the largest picture its parser declares (see How a session decodes), but eligibility checks the container’s size.
• The frame pool, the decoder’s state and buffer:s, and its packet ring each fit the device’s buffer limit (maximumBufferByteLength), the smaller of its largest buffer and its largest storage binding: the HEVC decoder project’s package holds 29 slots, which in I420P10 at 3840x2176 is about 727 MB. findGPUDecoderStreamFitFailure checks them.

Static HDR needs allowFramePoolHDR, which the host sets when it tone-maps.

A Dolby Vision stream with an RPU route reconstructs on the route too, with dolbyVisionProfile set, when the host authorized that route (allowDolbyVision, allowDolbyVisionProfile4, or allowDolbyVisionProfile7, from the raw route’s key of the BL format) and an offer writes the format the RPU route names: the range extension’s own, I420 for HEVC Main, or I420P10 for HEVC Main 10 and AV1 Main at 10 bits. allowFramePoolHDR plays no part in it. The worker parses each packet’s RPUs, as on the raw route, a separate EL track’s included, and each frame carries its packet’s RPUs through the GPU decode session to the presenter; a repeat of the last good frame carries that frame’s. The presenter reconstructs the slot through the raw route’s reconstruction (FramePoolDolbyVisionSignal.ts), gated by the same base key, and no frame pool holds an EL, so Profiles 4 and 7 reconstruct from their BL alone. getGPUVideoDecodeTarget names the RPU route’s format, so a host fetches packages for a stream without a declared base.

hasEligibleCustomVideoRoute holds no offers, so a GPUVideoDecoderSummary (src/decode/video/gpu/GPUVideoDecoderCatalog.ts) stands in for one, after the offers, and selects the route without a decoder. A summary names the package’s codecs and output formats, and may carry the package’s header and the buffer limit of the device a playback would decode on. isGPUVideoDecoderSummaryFit judges the stream as an offer would: by codec and format, then with a header by monochrome for a 4:0:0 stream and the accepts conditions, and with a limit by the frame pool, state, buffer:s, and packet ring. A summary without a header takes every stream of its codecs and formats, and leaves the other checks to the playback’s offers.

A package’s self-test vectors run only when a host calls verifyGPUDecoderPackage (see Decoder packages); eligibility never waits on them, and offering an unverified package is the host’s choice.

VP9, HEVC range extensions, and H.264 beyond 8-bit 4:2:0 present HDR only through raw planes, and AV1 keeps its raw route beside its native external one, so a host must settle the raw HDR keys for them whatever the external result. The AV1 external keys authorize on first use: getCustomPresentationEligibilityOptions waits for the key of an AV1 HDR stream’s transfer before eligibility, through WebGPUPresenter.waitForExternalHDRRouteAuthorization. The Dolby Vision keys a source needs on first use (Profile 4 in any format, or Profile 7 or single-layer reconstruction in a format other than I420P10) must settle before eligibility.

Interlaced video

Every route but Dolby Vision’s presents interlaced video through the deinterlacing pass (see Architecture), so eligibility refuses interlaced video only where no route can present it, as interlaced-video-unsupported:

• IsInterlaced true on HEVC, which codes interlaced video as field pictures, and on JPEG 2000, which codes a codestream per field: no decoder weaves their fields into frames, and the pass deinterlaces frames whose rows alternate between the fields. The check runs before route selection, and the prefilter declines the same streams.
• A Dolby Vision route, whose profiles are progressive and which never deinterlaces, unless IsInterlaced is false. A Dolby Vision stream whose declared base layer takes an ordinary route presents through that route’s deinterlacing.

H.264, MPEG-2, MPEG-4 Part 2, VC-1, and WMV3 decode interlaced video into woven frames, so they are eligible whatever IsInterlaced says, and a missing or malformed IsInterlaced is an unknown scan, which each frame’s own fields settle. MPEG-1, H.263, MS-MPEG4, VP8, VP9, and AV1 have no interlaced coding. presentsInterlacedVideo(codec) exports the per-codec answer, so a host’s negotiation offers interlaced streams of exactly those codecs, outside its Dolby Vision labels.

Route keys

• Raw: <I420P10..I444P12>:bt2020-ncl:bt2020:limited:<pq|hlg> and <I420..I444P12>:<matrix>:<primaries>:<full|limited>:sdr, with every matrix (bt709, bt470bg, smpte170m, bt2020-ncl) and primaries set (bt709, bt470bg, smpte170m, bt2020). The HDR keys and the BT.709 SDR keys are prewarmed; every other SDR key authorizes on first use, and its vector doubles the chroma excursion so the gamut conversion moves its samples beyond the tolerance.
• External HDR: external-hevc-main10-bt709-limited:<pq|hlg>-v1 and external-av1-main10-bt709-limited:<pq|hlg>-v1, each with its own codec’s vector, authorize on first use, since they are the only keys whose check decodes a frame. The two HEVC keys start together, sharing one decode, when an HDR negotiation waits for them (waitForExternalHDRAuthorizationPrewarm) or a playback that requests the external HDR route starts them (prewarmExternalHDRPresentationAuthorization); an AV1 key starts when its route is first used. Device creation prewarms only the raw and Dolby Vision keys named prewarmed here.
• Dolby Vision raw, each per BL format from I420 to I444P12: <format>:dovi-rpu-v1, <format>:dovi-profile4-base-v1, <format>:dovi-profile4-fel-v1, <format>:dovi-profile7-base-v1, and <format>:dovi-profile7-fel-v1. Only I420P10:dovi-rpu-v1 and the I420P10 Profile 7 keys are prewarmed; the others authorize on first use.
• Dolby Vision external: external-I420P10-bt709-limited:dovi-p5-rpu-v1, prewarmed, and external-I420P10-bt709-limited:dovi-p8-rpu-v1, which only AV1 P10 uses and which authorizes on first use; both check the same shader with their profile’s synthetic RPU.

Audio routes

• Track: DefaultAudioStreamIndex, else the first audio stream by index. The worker selects it by its ordinal among the container’s audio tracks, which skips the streams Jellyfin lists for external files (IsExternal), as the video ordinal does. A selected external audio stream is audio-track-external, since only the server can mux an external file in. Codec aliases: DCA is dts, EC-3 is eac3, TRUE-HD is truehd; WMA takes FFmpeg’s decoder names, wmav1, wmav2, wmapro, wmalossless, and wmavoice.

• Order: an E-AC-3 or TrueHD stream whose Jellyfin Profile names Atmos takes the first offered package that declares renderChannelMask when the host’s audioRenderTarget needs its objects: a height speaker layout, a floor wider than the declared channels (7.1 from a 5.1 core), or binaural headphones on a stereo layout; the declared layout must still pass the codec’s input routes, and the eligibility sets audioRendersObjects. Otherwise the native-media MSE bridge for AC-3 and E-AC-3 (2 or 6 channels, exactly 48 kHz) comes first. Otherwise decoded-pcm: through the engine’s own decoders when the codec capability and the input layout qualify, the browser’s for WebCodecs codecs and Mediabunny’s for PCM. A codec no engine decoder supports, AC-3, E-AC-3, DTS, TrueHD, MLP, and WMA among them, takes the first of the host’s audio decoder packages whose manifest accepts it.

• Input channels, as declared by Jellyfin. The decoded format is authoritative at runtime, and any decoded layout maps to the output by channel name:

  Codec	Channels
  PCM	1, 2, 3, 6 (3 needs a 3.0 layout)
  aac, flac, opus, vorbis	1, 2, 3, 6, 8 (3 needs a 3.0 layout, or 2.1 for FLAC, laid out by its mask; 6 needs the codec’s 5.1 verdict and 8 its 7.1 verdict)
  ac3	1, 2, 5, 6 (2/1 and 3/0 both arrive as 3.0, and no evidence covers either; 5 is 3/2, 5.0(side))
  eac3	1, 2, 6, 8
  mp3	1, 2
  dts	1, 2, 3, 5, 6, 7, 8 (4.1 and 5.0 arrive with 5, 6.1 with 7)
  truehd	1, 2, 6, 8 (8 only at 48 kHz)
  mlp	1, 2
  wmav1, wmav2	1, 2
  wmapro, wmalossless	1, 2, 3, 6, 8
  wmavoice	1

  Jellyfin keeps only the part of FFmpeg’s layout name before (, so 3.0(back) arrives as 3.0 and 7.1(wide) as 7.1. A package reports each block’s speaker mask, so the mask, not the declared layout, tells them apart at runtime: a DTS 3.0(back) plays with its back center on the surrounds, and a mask with no output layout fails the attempt as decode-failed, which renegotiates. DTS-ES’s XCh and a lossless 6.1 bed decode to the WAVE 6.1 mask, which plays with its back center split across the surround or back pair; a DTS 4.1 plays the same way, and a mask with no layout, such as 4.0, fails the attempt. Seven-channel WMA Pro and WMA Lossless still have no route, since their input table leaves 6.1 out. An 8-channel AAC, Opus, FLAC, or Vorbis track declares no mask, so it lays out by its count as WAVE 7.1 (CUSTOM_SEVEN_POINT_ONE_CHANNEL_LAYOUT), the order its 7.1 probe proved. FFmpeg’s encoder writes 7.1 AAC as AudioSpecificConfig channelConfiguration 7, which FFmpeg’s decoder, and so Chrome’s, reads as 7.1 with back and side pairs, not the specification’s 7.1(wide); either reaches the engine as 7.1.

• DTS and TrueHD play from Matroska and from MP4, M4V, and MOV (the DTS , dtsc, dtsh, dtsl, and mlpa sample entries); MLP plays only from Matroska.

• WMA plays from Matroska A_MS/ACM tracks and AVI and ASF streams, which Mediabunny leaves codec-less: the worker reads the WAVEFORMATEX (demux/WaveFormatAudioConfiguration.ts), names the codec by its format tag, and refuses a track whose tag names another codec than the package decodes it as. The package configures with that WAVEFORMATEX before its first packet (see Decoder packages).

• Rates: any positive integer, resampled to 48 kHz. Past 192 kHz the resampler’s kernel widens in proportion, so its band edge stays as sharp as at 192 kHz. A browser decoder judges its own rates per item, and a rate it refuses falls back.

• Output layout (audio/NativeMultichannelAudioOutput.ts): selectCustomAudioSpeakerLayout resolves the host’s speaker layout setting on destination.maxChannelCount: auto takes 7.1.4 from 12 channels, 7.1.2 from 10, 7.1 from 8, 5.1 from 6, else stereo; an explicit stereo, 5.1, 7.1, 5.1.2, 5.1.4, 7.1.2, or 7.1.4 needs a destination with its channels, else resolves as auto. selectCustomAudioOutputLayout then gives a source a package renders as objects the whole speaker layout, and every other source the floor rule within the layout’s floor: three channels, 5.0, and 5.1 take 5.1, 6.1 and 7.1 take 7.1 or fold into 5.1, everything else downmixes to stereo with the selected algorithm. A host may force stereo. A live layout switch applies the same rule to the decoded layout. The output plays 2, 6, 8, 10, or 12 channels in WAVE order (5.1 in Web Audio’s order); a decoded layout maps by channel name, a 5.0 bed fills 5.1 without its LFE and mixes to stereo as that 5.1, a height the output lacks folds into the floor pair beneath it, and a height layout mixes to stereo through its folded floor.

Gotchas

• Native decode uses the hint its probes measured (getCustomDecodeHardwareAcceleration). Only the routes that present the decoder’s opaque hardware output (native external HDR, the native Dolby Vision base, and external Profile 5) prefer hardware. Raw-plane AV1 and VP9 prefer software, because Chromium’s hardware decoders return opaque 10-bit surfaces, and so does raw-plane H.264, whose profiles beyond High Chromium reports unsupported with prefer-hardware. Every other native route uses no-preference, so a codec without a hardware decoder, such as VP8 in Chromium on Windows, decodes in software. One exception: 10-bit SDR HEVC has no probe of its own. The native HDR HEVC probe gates it with prefer-hardware, while the SDR route requests no-preference. Chromium decodes HEVC only in hardware, so both resolve to the same decoder.
• Eligibility checks no audio profile: a stream a package cannot decode, such as DTS-UHD in a DTS package, fails its attempt at runtime as decode-failed.
• A raw SDR stream outside BT.709 primaries converts its gamut to BT.709 in the raw shader (builtin.sdr-gamut), as the frame-pool route does, while the native VF-SDR route leaves the conversion to the browser.

HEVC and Dolby Vision support

This chapter says which HEVC and Dolby Vision sources, AV1 Profile 10 included, the custom pipeline plays, by which route, and on what evidence, and which AV1 sources fall back to a decoder package. It is the engine’s support matrix: what getCustomPlaybackEligibility accepts and the route it selects, whatever the host. A host may restrict it further; the Jellyfin plugin’s book has the matrix of what Jellyfin negotiates for each row. It comes from the source and the unit tests, not from playback. test/capability/CustomPlaybackEligibility.test.ts covers route selection, and the Jellyfin plugin’s support matrix tests assert every row, with its fallbacks, through getCustomPlaybackEligibility.

Read every “Yes” as conditional:

• the evidence listed for at least one of the row’s routes must pass on the running browser and GPU;
• the host must pass the route flags and authorized keys of CustomPlaybackEligibilityOptions for that evidence.

HEVC is eligible only in the containers that capability/CustomContainerCodecSupport.ts pairs with it (MP4, M4V, MOV, Matroska, MPEG-TS, M2TS, MTS). Resolution, level, frame rate, and bitrate are never gates. The route names are defined in Routes, and the probe and authorization names in Evidence.

Every HEVC row also falls back last to GPU, when the host offers a GPU decoder package whose header accepts the stream: the plain and range extension rows, and a Dolby Vision row’s declared base. A Dolby Vision row with an RPU route also falls back to GPU-DV, RPU reconstruction from the package’s frame pool, after its native and raw routes and its declared base’s native and raw routes, and before that base’s own GPU route; the authorization is the raw RPU route’s key of the BL format.

Plain HEVC

The SDR rows cover every transfer the engine reads as SDR: BT.709, SMPTE 170M, sRGB, and the BT.2020 10 and 12-bit transfers, which use the BT.709 curve. They take BT.709, BT.601 (SMPTE 170M, SMPTE 240M, or BT.470 BG), or BT.2020 primaries and matrix. The VF-SDR routes leave the color conversion to Chrome, while the raw SDR routes convert a gamut other than BT.709 themselves, through the raw SDR key of the stream’s matrix, primaries, and range, which authorizes on first use outside BT.709. Jellyfin’s HDR10+ flag does not change these rows: HDR10+ metadata applies only on a PQ transfer, so an SDR or HLG stream that Jellyfin flags as carrying it takes its SDR or HLG row, and its metadata is ignored.

The HLG row includes HLG-compatible streams, whose VUI signals the BT.2020 10-bit transfer and whose alternative transfer characteristics SEI names HLG (see Evidence).

Range extensions

The PixelFormat must be exact, and an explicit BitDepth must agree with it.

Dolby Vision

A variant is written as the Dolby Vision profile and compatibility ID (CCID), then the HEVC profile. Main 10 at 10 bits unless stated. The engine reads the Dolby Vision configuration and the color fields; it ignores VideoRange and VideoRangeType.

Dolby Vision over AV1

Profile 10 is single-layer: the RPU travels in an ITU-T T.35 metadata OBU of each temporal unit, and the base layer is AV1 in any profile and depth. VF-DV5 and VF-DV8 here decode Main at 10 bits in hardware through the engine’s own AV1 path, every sequence header neutralized to limited BT.709, and reconstruct from the external frames; RAW-DV decodes natively in software, into I420P10 for Main at 10 bits, or into the exact format of an AV1 raw format variant (see AV1 outside Main 10). Every AV1 track takes that path, whatever its route, so the declared base of a stream with HDR10+ metadata applies its HDR10+ metadata OBUs too (see Decisions). Only an RPU route loads the RPU parser; any other route removes the RPU OBUs without parsing them.

The external routes present 8-bit-quantized frames, as every external route does, so their reconstruction differs from RAW-DV by a few code values, more where the RPU’s mapping is steep.

The engine maps the MP4 dav1 sample entry of a 10.0 stream to AV1 itself (see Decisions).

Dolby Vision outside HEVC and AV1:

• Profile 9 (AVC, 8-bit) has no RPU route, because the engine owns no AVC decode path. Its declared SDR base plays through the H.264 routes.
• The retired Profiles 0 to 3 and 6 have no RPU route and present only a declared base.

AV1 outside Main 10

Native routes decode AV1 Main at 4:2:0: VF-SDR at 8 bits, and at 10 bits VF-PQ and VF-HLG where the AV1 external keys are authorized, RAW I420P10 through the raw SDR keys, and RAW I420P10 PQ or HLG. AV1 outside Main 10 takes raw planes in the exact format of its raw format variant, which its Jellyfin Profile and PixelFormat name, through av1RawFormats[variant], a 192x192 key and inter temporal unit decoded with prefer-software. Every AV1 stream falls back last to GPU, in its own format, when the host offers an AV1 decoder package whose header accepts it. The AV1 package takes pictures up to level 6.x, 16384x8704 and 35,651,584 samples, 8K included, on a device whose largest binding holds half its frame store and its packet ring: at 8192x4352 two buffers of 535 MB and a ring of 626 MB at 4:2:0 10-bit, two of 1.07 GB and a ring of 1 GiB at 4:4:4 12-bit (see WebAssembly build); eligibility skips it with the failed check otherwise.

Chrome’s software AV1 decoder decodes and copies the 8 and 10-bit variants, and refuses 12-bit AV1 at decode, so the 12-bit variants read unsupported there and take GPU. hasPotentialCustomPlaybackVideoRoute keeps AV1 outside Main 4:2:0 potential for these routes. On GPU the decoder receives the track’s av1C (see Codec private data).

H.264 beyond 8-bit 4:2:0

Native H.264 decodes Constrained Baseline, Baseline, Main, and High at 8-bit 4:2:0 to VF-SDR, each profile through its own h264Profiles probe. A profile beyond High takes raw planes in the exact format of its raw format variant through h264RawFormats[variant]: an I, a P, then a B access unit at 192x192, decoded with prefer-software, whose B frame must output before the P frame. The profile must code the variant’s chroma format: High 10 and High 10 Intra code 4:2:0, High 4:2:2 and High 4:2:2 Intra up to 4:2:2, and High 4:4:4 Predictive, High 4:4:4 Intra, the removed High 4:4:4, and CAVLC 4:4:4 up to 4:4:4. The PixelFormat names the variant, and FFmpeg’s yuvj names of a full-range 8-bit stream count as their yuv formats. No GPU decoder package decodes H.264, so a variant without its capability or raw key is refused.

Chrome’s software H.264 decoder is FFmpeg’s, which outputs the same planes as the reference decode the vectors pin, while prefer-hardware reports these profiles unsupported. The worker decodes them through Mediabunny’s sample path, as native H.264, which orders B frames, reads avcC or Annex B as the container gives it, and takes the presentation times the fork’s AVI demuxer gives each picture.

Interlaced video

The deinterlacing pass presents interlaced frames on every route but Dolby Vision’s (see Architecture and Eligibility and routes). This table covers every codec, beyond this chapter’s HEVC and Dolby Vision.

A stream whose metadata calls it interlaced but whose frames report progressive presents them untouched, and a frame without any fields takes the stream’s scan. The decoder’s and the parser’s reports override the container’s: an MPEG-2 or VC-1 stream whose Matroska header calls it progressive still deinterlaces each picture it reports interlaced.

Rejected

Routes

A route is the eligibility output: videoOutputMode, videoDecoderBackend, rawVideoFrameFormat, dolbyVisionProfile, and nativeHDRTransfer, plus gpuVideoDecoder on GPU. The host maps it to a presenter input mode.

Eligibility and routes gives the order in which they are tried.

HDR10+ dynamic metadata is applied on VF-PQ, DV base PQ, RAW PQ, and GPU PQ, and on the RPU routes (VF-DV5, VF-DV8, RAW-DV, and GPU-DV) to a frame whose RPU has no L1. HEVC carries it in SEI, and AV1 in ITU-T T.35 metadata OBUs, whose trailing bits the engine removes as dav1d does: it drops the trailing zero bytes, then the byte holding the trailing one bit, which must be 0x80, or the frame’s metadata is malformed. VP9 has no metadata of its own, so Matroska and WebM carry each frame’s ITU-T T.35 message beside it in a BlockAdditional, and VP9 in MP4 carries none. The engine recognizes the message by its T.35 header whatever its BlockAddID, so WebM, which has no BlockAdditionMapping, reads like Matroska, which maps FFmpeg’s BlockAddID 4 to the ITU-T T.35 type. A frame’s scene peak and average drive every tone map operator; only the ST 2094-40 operator follows a frame’s Bezier curve, adapted from its targeted display, and without a curve it builds libplacebo’s brightness-matching curve from the scene’s knee. A frame without a curve, such as every profile A frame, never has its targeted display read. A frame without HDR10+ metadata of its own, or whose payload fails to parse, takes the last metadata before it in decode order, as in FFmpeg (see Decisions). Conflicting metadata, unsupported metadata, and a curve with a targeted display of 0 tone-map their frame statically and end the carried metadata until the next valid payload. Each decode attempt and each seek starts without carried metadata.

An RPU route tone-maps each frame from its RPU, as libplacebo ranks the metadata: from its L1 scene peak and average, else from HDR10+, else statically from the peak of the mastering display the RPU names (source_max_pq), which an RPU without display management metadata leaves at FFmpeg’s default of 4000 nits. An L1 whose maximum or average is 0, as an RPU whose CM v2.9 section codes no L1 presents, counts as none.

Evidence

Probes, all in capability/CustomDecodeCapabilities.ts:

GPU has no probe: a package’s own vectors run only when the host verifies it (see Decoder packages).

GPU authorizations, all in validation/:

I420P10:dovi-rpu-v1 and the I420P10 Profile 7 keys are part of the default Dolby Vision prewarm. The Profile 4 keys in any format, and the Profile 7 and single-layer keys of other formats, authorize on first use, so a host must wait for them before it advertises or starts a stream that needs one. The external HDR keys authorize on first use too: an HDR negotiation waits for the HEVC pair (waitForExternalHDRAuthorizationPrewarm), which shares one decode, and an AV1 HDR stream waits for its own key.

Other conditions:

• Native VideoFrame HDR and the DV base routes also need explicit ColorTransfer, ColorPrimaries, and ColorSpace values.
• The SPS parser never rejects a VUI color description. It maps each code to a WebCodecs name or to unspecified (null), and an SPS without VUI has unspecified color. Only the native HDR route check is strict: limited range, BT.2020 primaries, the BT.2020 non-constant-luminance matrix, and the route’s transfer.
• The transfer that check reads is the key access unit’s alternative transfer characteristics SEI value (payload type 147) when one is present, and the VUI transfer otherwise. Without the SEI, VF-HLG and DV base HLG also accept the BT.2020 10 and 12-bit VUI transfers (14 and 15) of HLG-compatible streams. An SEI naming another transfer rejects the HLG route, and a malformed SEI counts as absent, as in FFmpeg.
• P20 is MV-HEVC. NAL units with a nuh_layer_id above 0, the second view, are dropped before decode, and SEI of other layers is ignored.
• A separate-track P7 (a base track and an EL track) selects the same routes. test/capability/CustomPlaybackEligibility.test.ts covers it.
• An RPU the parser rejects ends the custom session, which then falls back. The parser follows FFmpeg’s dovi_rpudec.c: a display-metadata extension block that runs past the data or falls short of its level’s fields, an L2 ms_weight below -1, or more than 7 static or 25 dynamic blocks reject the RPU, an L1 in the CM v4.0 section counts as an L1 of zeros, and unknown, other misplaced, or padded blocks are skipped. Leftover bits before the CRC32, or an AV1 EMDF wrapper whose size or protection bits FFmpeg refuses, also reject it, while a CRC32 mismatch and bytes after the 0x80 terminator do not, as in FFmpeg. It also reads what FFmpeg does not: polynomial and MMR pieces mixed within one component, polynomial linear interpolation (see Decisions), and any mapping color space or chroma format.

Not supported

• HDR10, HDR10Plus, or HLG at 8 bits, under any profile.
• Profile and bit depth contradictions: Main at 10 bits, Main 10 at 8 bits, Main 10 at 12 bits.
• 14-bit and 16-bit range extensions.
• Monochrome range extensions outside GPU.
• High Throughput 4:4:4 profiles, Screen Content Coding (Screen-Extended) profiles, Main Still Picture, and named Intra aliases such as Main 4:4:4 10 Intra. Generic Rext with an intra constraint is the ordinary variant.
• Interlaced HEVC of any profile or range: HEVC codes interlaced video as field pictures, which no decoder weaves into the frames the deinterlacing pass takes (see Interlaced video).
• Dolby Vision:
  • the configurations in Rejected;
  • RPU reconstruction for AVC Profile 9, and for AV1 Profile 10 in a raw format variant the browser does not qualify, such as 12 bits in Chrome;
  • RPUs the parser rejects: a linear interpolation piece next to an MMR piece, an RPU format extension, missing sequence information, display metadata compression above method 1, and a header from which no profile is inferred while the container declares none.

Known issues

• Firefox on Windows has no WebCodecs HEVC (see Decisions), so HEVC plays there only through GPU. Dolby Vision then plays through GPU-DV, Profile 5 included, when the host offers the HEVC package and authorized the RPU route’s key. Firefox decodes 10-bit AV1 to 8-bit BGRX, so no raw AV1 route, P10 included, qualifies there.

Module map

One entry per file or family. [main], [worker], and [worklet] mark the thread where it is not obvious. Each source file’s tests are at the same relative path under test/, and integration suites that span several modules sit beside their main module.

src/

• EngineAssets.ts: the typed manifest of every runtime asset path, the engine’s workers, its own WebAssembly modules, and the native range extension probes’ streams, and URL resolution against the host’s asset base or, inside a worker, the worker’s own URL.
• EngineConfiguration.ts: the feature flags a host can set.
• GPUCompletion.ts: waitForGPUCompletion, through which every wait on a device’s queue work goes, onSubmittedWorkDone() or a mapAsync(); it polls the device with empty submits while the wait is pending, since Firefox completes queue work only when the device is polled.
• MediaTime.ts: branded integer microseconds and conversions to Jellyfin ticks.
• TimeMath.ts: safe-integer microsecond math.
• TimingTrace.ts: the opt-in playback timing trace: the page’s ring buffer, the decode worker’s batches stamped on the shared epoch clock, long tasks, and the JSON export; every hook is one null check until a host starts a trace.
• style.css: presenter and overlay styles, imported by the host; plain CSS, so a host needs no preprocessor.

presentation/

• WebGPUPresenter.ts: the GPU device and canvas; rVFC and pushed-frame submission; each route’s built-in chain, planned and recorded through passes/; HDR and Dolby Vision authorization; device-loss recovery; the latched fallback. It also holds the engine API for presentation: the host’s preset of user passes, live parameter changes, the plan report, HDR output for HDR sources, the deinterlacing mode, and the black bar crop. While a plan deinterlaces it shows the frame before the newest, tells the frame loop its lookahead, and draws held fields from history. While a plan holds a display-rate pass it measures the display’s refresh rate from the frame loop’s animation frames, takes frames one refresh ahead, and redraws from history at each tick’s clock time. While the black bar crop is on it reads back each new frame’s black bar statistics and fits the picture the detector’s bars leave, rounded to a standard aspect ratio near it.
• BlackBarDetector.ts: the black bars a presentation crops, decided frame by frame from the black bar analysis’s statistics: symmetric bars per axis, cropped at once for a letterbox or a pillarbox and else once two shots agree, given up at once for picture inside them, and cropped again only after 30 s of playback.
• StandardAspectRatios.ts: the film and television aspect ratios of the Detect Aspect Ratio plugin, and the rounding of a black bar crop to the nearest one its bars reach by cutting at most 0.5% of the picture or showing at most a line of bar.
• DisplayRefresh.ts: the display’s refresh interval, a least-squares fit of animation frame times over refresh counts, reported once it is precise and restarted when the display changes.
• FrameInterval.ts: the media time between a stream’s frames, frame by frame: a least-squares fit of the last 24 new frames’ timestamps over frame counts, counting skipped frames and restarted when the frame rate changes.
• DeinterlaceTimeline.ts: the frames a deinterlacing plan’s history holds and the draw a clock time picks: the field shown, of which frame, and its time, at double or single rate, with yadif’s passthrough of progressive frames.
• PresentationInput.ts: MediaStream metadata to color metadata; Dolby Vision descriptors with their RPU route; declared and exact native base metadata; the presented video ordinal; the known-SDR gate; the stream’s fields from its FieldOrder or IsInterlaced and its size, and its declared frame rate, which PresentationSetup.ts hands the presenter.
• DolbyVisionGeometry.ts: the P4 and P7 EL coded size (half the BL when the BL is wider than 1920).
• PresentationSetup.ts: what a host calls around the eligibility, without state of its own: the play options’ presentation input, the authorizations that gate the eligibility’s presentation routes, the offers of the host’s GPU decoder packages for a stream (getGPUVideoDecoderOffers) and the factory of their sessions on the presenter’s device, the presenter’s color pipeline for the chosen route, Dolby Vision bases included, and static HDR metadata updates.
• PresentationGeometry.ts: pure object-fit and object-position math to viewport and texture transforms, fitting a crop of the source, the black bar crop’s, as the whole object.
• RenderSettings.ts: versioned (v7) HDR-to-SDR settings, the four tone map operators, and their 144-byte uniform, with the dynamic fields of HDR10+ and Dolby Vision L1.
• WebGPUConstants.ts: the WebGPU values presentation and GPU decode share: canvas configuration, texture formats, features, and default limits.

presentation/passes/

The presentation pipeline: each route’s fixed signal and default color track, user WGSL passes, the planner that arranges them, and the executor that records frames.

• PassTypes.ts: the contracts of the pipeline: pass manifests, built-in chains, plans, and frame input.
• PassLanguage.ts: the names that headers, the prelude, and the planner share: the plane and image stages, HOOKED, history accessors, texture block accessors, entry points, offset: align, the prelude’s functions, and plan-time facts.
• PassExpression.ts: the size and condition expressions of pass and decoder headers.
• PassManifest.ts: parses a pass file, whose passes each start with a //! header as in mpv’s user shaders, into manifests in file order, each with the texture blocks it binds; a failure names its line in the file.
• ContentIdentity.ts: the content identity of a texture block, a 53-bit cyrb53 hash computed synchronously, which keys its upload.
• PassResources.ts: the engine-owned bindings, their WGSL declarations and layout entries, the byte layouts of the uniforms the engine writes, and a parameter value’s normalization to its type and range.
• PassPrelude.ts: the WGSL around pass code: input accessors, each with its input’s plan-time offset, texture block accessors, constants, the prelude’s linear light and planes transform, entry stages, and the renaming that keeps each user body’s names apart.
• PresentationPlanner.ts: arranges a route’s built-in chain and the user passes on the stage spine and the color track, then generates each pass’s module. A pass that cannot run is left out with a reason code and a terse detail. Without user passes every route plans one pass, which reads the source and writes the canvas. It reads the signal’s planes apart for plane passes, from the source or a deinterlaced image, merges and converts the planes they replace, and gives every image its offset and extent, which the scaler corrects.
• PresentationExecutor.ts: compiles a plan, owns its textures, history rings, buffers, and bind groups, binds the caller’s uploads of its texture blocks, and records each frame without submitting; a draw from history records the passes from the plan’s history draw start on, except those that read the source. It runs a plan’s black bar analysis only for a frame with a readback buffer, a workgroup per row and column of the frame’s source, and copies its statistics into that buffer.
• PresentationPipelineCache.ts: the passes each device keeps compiled, a bounded least recently used cache keyed by WGSL, bind group layout, and target format, which the executor compiles through.
• PassTextureCache.ts: the uploads of texture blocks on one device by content identity, made when a plan first binds a block and released when no plan or preset holds it.
• PlaneTextureSource.ts: the plane route’s ingress: layout validation that names the failed check, one integer texture per plane, and the uploads.
• builtin/: the built-in signals (BuiltinSignals.ts, FramePoolSignal.ts, which reads a pool slot of any output format, as its layout says, for the frame-pool route’s chain, and FramePoolDolbyVisionSignal.ts, the raw Dolby Vision reconstruction over that read), color functions (BuiltinColorFunctions.ts), and route chains (BuiltinChains.ts), the deinterlacing pass (DeinterlacePass.ts, yadif over the signal’s ring), and the black bar analysis (BlackBarAnalysisPass.ts, a compute pass over the signal that writes how many of 64 points along each row and column are lit and bright, and the sum of their signed luma). A deinterlacing plan’s raw and frame pool signals read an interlaced frame’s 4:2:0 chroma within each field. Every signal reads 4:2:0 chroma at the stream’s declared location, left where it declares none. Each signal offers its planes to plane passes: the raw and frame pool signals their decoded planes, with chroma’s siting, range, and matrix, the external signals planes derived from RGB through the stream’s own matrix and range at its chroma subsampling, and the Dolby Vision signals none. Every signal with planes also offers their transform from ‘source’ RGB and back, which deinterlacing plans and the prelude use. BuiltinColorFunctions.ts also gives each chain its transfer function for the prelude’s linearize and delinearize, renamed from the transfer decode’s code.

The optional user passes a host may offer are pass projects in wgpup/, outside src/ (see Bundled passes); the presenter tests run some of them.

color/

• ColorMetadata.ts: the input color schema, validation, and SDR, PQ, and HLG factories.
• ColorPipeline.ts: the CPU reference of range, matrix, transfer, gamut, tone mapping, and display controls, and the luminance, YUV matrix, gamut, and IPT tables for BT.709, BT.2020, and BT.601 that the built-in color functions share.
• DolbyVisionColorTransform.ts: RPU reshaping (polynomial and MMR, chosen per segment), the FEL NLQ residual, and reconstruction to BT.2020 PQ, as a CPU reference and as WGSL.

capability/

• CustomContainerCodecSupport.ts: the single container and codec matrix.
• CustomPlaybackEligibility.ts: per-session route choice, the player-selection prefilter, hasEligibleCustomVideoRoute for item-scoped negotiation, the codecs whose interlaced streams a route presents (presentsInterlacedVideo), and the video and audio codecs a playback decodes (getGPUVideoDecodeTarget, getPlaybackAudioCodec), which a host fetches packages for.
• CustomDecodeCapabilities.ts: the cached orchestrator of every capability probe, the per-item probe selection, and the codec lists, with the video codecs only a decoder package decodes (CUSTOM_PACKAGE_VIDEO_CODECS).
• CapabilityAssetLoading.ts: the bounded, retried download of a probe’s served vectors.
• CapabilityProbeStore.ts: the probe verdicts kept in localStorage across page loads, their fingerprint, the clear a failed start or a host calls, the run journal that names the steps in flight and their costs when a page stopped, the step quarantine (blocked steps, suspects, and the concurrency ceiling), and the crash event a host listens to.
• CapabilityProbeManager.ts: the parallel runner of every probe’s browser calls within byte and decoder limits, with negotiation priority, reservations held until a call settles, and the stop rule for a call that never settles.
• CapabilityProbeSession.ts: the page’s probe session, which owns the manager, sets its limits from the quarantine, and wraps every probe step (record): it blocks defect and quarantined steps with a CapabilityProbeBlock, enqueues suspects, and records admitted calls on the store.
• CapabilityProbeSteps.ts: the step descriptors, a label naming each browser call and its cost in decoded bytes and decoders, from the codecs’ maximum reference frames.
• BrowserDefects.ts: the declared table of known browser defects whose matching steps never run, and the browser identity it matches.
• AudioDataPlanes.ts: the copy of an AudioData in its own format, converted to float planes in JavaScript.
• VideoFrameFormatSupport.ts: the check that the browser holds a raw VideoFrame format, and a format’s byte length.
• CustomPlaybackRuntime.ts: runtime feature detection with failure reasons.
• H264ProfileCapabilities.ts: the per-profile H.264 decoded-output probe and Jellyfin’s profile names.
• HEVCRangeExtensionCapabilities.ts: the nine range-extension variant definitions and the stream metadata resolver.
• AV1RawFormatCapabilities.ts: the eight AV1 raw format variant definitions outside Main 10 (High 4:4:4, Professional 4:2:2 and 12 bits, and 8-bit Main for Dolby Vision) and the stream metadata resolver.
• H264RawFormatCapabilities.ts: the five H.264 raw format variant definitions beyond 8-bit 4:2:0 (10-bit 4:2:0, and 4:2:2 and 4:4:4 at 8 and 10 bits), the profiles beyond High that FFmpeg names and the chroma formats each codes, and the stream metadata resolver.
• NativeMediaAudioCapabilities.ts: the MSE AC-3/E-AC-3 route probe and its selection.
• NativeAudioChannelTones.ts: the channel order check of the 7.1 probes, which finds each decoded plane’s strongest source tone with a windowed Goertzel measure.

capability/vectors/

Hand-made vectors: known inputs with known answers. Qualification vectors run in the browser before a route is advertised; test vectors feed only tests and generators. The TypeScript modules are embedded in the bundle; the binary files are served or read as files.

• HEVCExactCapabilityVectors.ts: the single-frame black HEVC access units the native HEVC probes decode, Main at 1080p and Main 10 at 4K.
• NativeMediaAudioCapabilityVectors.ts: fMP4 AC-3/E-AC-3 vectors.
• NativeAudioCapabilityVectors.ts: stereo silence packets per WebCodecs codec.
• NativeSurroundAudioCapabilityVectors.ts: 5.1 silence packets.
• NativeSevenPointOneAudioCapabilityVectors.ts: 7.1 packets of one sine per source channel, and each channel’s tone in WAVE 7.1 order.
• NativeVideoCapabilityVectors.ts: 64x64 VP8, VP9, and AV1 keyframes.
• NativeUltraHDVideoCapabilityVectors.ts: 4K 8-bit HEVC, VP9, and AV1 keyframes.
• RawHDRCapabilityVectors.ts: 4K 10-bit VP9 and AV1 keyframes with plane fingerprints.
• ExternalHDRAuthorizationVector.ts: the neutralized Main 10 access unit and its expected samples, for external HDR authorization.
• ExternalAV1HDRAuthorizationVector.ts: a lossless AV1 Main 10 BT.2020 PQ temporal unit and its exact samples, for the AV1 external HDR keys.
• DolbyVisionAuthorizationVector.ts: synthetic RPU snapshots for Dolby Vision authorization.
• HDR10PlusVectors.ts: deterministic HDR10+ HEVC access units for the dynamic HDR tests.
• Base64.ts: the base64 decoder the inline vectors share.
• qualification/: a stream that no script generates, the VC-1 stream (vc1/), which the vc1 decoder project’s vector holds.
• test/: inputs only tests and generators read: DTS samples, a TrueHD Matroska remux, Dolby Vision RPU payloads, and an hdr10plus_tool HDR10+ stream.

pipeline/

• CustomPlaybackController.ts [main]: lifecycle, generations, the clock, the startup (20 s without progress, the source bytes read before readiness counting as progress, 60 s ceiling), stall (10 s), and lag (2 s) policy, the live audio output layout switch, the end-of-stream and ended-track drains, the presentation timing counters, and the fallback disposition. It also holds the playback rate: live changes, the restart that crosses the audible range, the audio selection kept aside outside it, the presentation interval it hands the session above 1x, and the forward keyframe resync above 1x.
• CustomPlaybackRate.ts: the playback rates custom playback takes, 1/16x to 16x, and the audible range, 0.25x to 4x.
• CustomPlaybackControllerTypes.ts: states, events, fallback reasons, and dispositions.
• CustomPlaybackFrameLoop.ts [main]: the frame loop a host runs per playback: each animation frame hands the presenter its timestamp and the clock’s rate, passes the presenter’s measured refresh interval to the controller when it changes, then presents the controller’s current frame, taken the presenter’s lookahead ahead of the clock, and acknowledges its credit, or draws again from history when no frame is due, and a hidden page drains decoded video every 25 ms instead.
• CustomDecodeSession.ts [main]: the session’s one decode worker, which the first start takes from the page’s idle worker or creates, each generation takes at once while the stopped runs unwind beside it, a failure, an unacknowledged stop, or too many unwinding runs replace, and destroy leaves idle for the page’s next session for 30 s once its runs are over; prefetchCustomDecodeSource, which starts the next session’s head read in the worker that session takes; and per generation, the frame queue, credits, raw buffer recycling, the frame-pool route’s GPU decode feed, readiness, audio-only resync epochs, the decoded source format, and an ended audio track completing a start, a resync, or a native-media stream. It also applies the playback rate to native-media audio bridges and scales the decoded PCM fill threshold by it, and keeps the video presentation interval for every later start.
• CustomDecode.worker.ts [worker]: one current run, which a start replaces at once while the stopped run unwinds beside it, each ending with stopped: demux, decoder dispatch, raw copy, Dolby Vision and HDR metadata, decoded PCM as restartable audio attempts in its audio decode worker, fMP4 remux, and credit waits; between runs, the source prefetch and its release. On the frame-pool route it posts compressed packets, or the records of the package’s parser, with their HDR10+ state, and the track’s codec private data on ready; after a parser’s recoverable failure it posts held packets up to the next key packet and restarts the parser there. Above 1x it decimates video by the presentation interval, skips owned HEVC pictures that can never present, and resyncs video forward to the next keyframe on resync-video with keyframe: 'next'.
• CustomAudioDecode.worker.ts, AudioDecodeWorkerRuntime.ts [worker]: the audio decode worker, which a decode worker spawns for its decoded PCM runs and keeps for its life, or until it is lost, when its next run spawns another. Each attempt has an audio decoder package’s instance, or none for samples Mediabunny decoded, an output stage, and a WorkletPCMProducer; it renders one batch at a time under the producer’s credit window and credits each batch back once rendered. A close shuts the attempt’s worklet channel at once and is acknowledged once the attempt released the rest, and a failure reports the kind the page acts on.
• AudioDecodeWorkerClient.ts [worker]: the decode worker’s side: the audio decode worker’s spawn, each attempt’s open, input credits, outcome, and close, and the packet and sample batch builders. A worker that fails to start, throws, or answers out of contract fails every attempt as audio-output-failed.
• AudioDecodeWorkerProtocol.ts: the messages between the two workers, their validators, the batch shapes and their transfers, the batch and credit constants (4 batches, 40 ms or 64 inputs), and the 1 MiB bound of the codec private data a package attempt carries.
• DecodeWorkerProtocol.ts: messages, the source prefetch requests that belong to no generation, validators, credit constants (4 frames, 2 raw buffers, 8 GPU decode packets, 8 audio samples), and the backend and output literals; a start’s and set-video-presentation-interval’s presentation interval, and resync-video’s forward keyframe choice.
• ConcurrentDecodeStreams.ts: concurrent decode streams that cancel each other on the first failure and all drain before the worker reports that the generation stopped.
• MediaClock.ts [main]: the generation-tagged clock; synchronize re-anchors it.

demux/

The first stage: the media a playback reads, its container input and track selection, codec configuration records, packet shaping and restamping, and the bitstream metadata that travels with packets. All worker code unless marked.

• CustomDecodeMediaSource.ts [main and worker]: the media a playback reads, a URL that the worker fetches in byte ranges, with the host’s pattern for the paths whose cross-origin 206 hides Content-Range, or a File that crosses to it by structured clone, and its validator.
• MediaFetchPolicy.ts: retries only transport errors and 408, 429, and 5xx.
• HTTPRangeResponse.ts: validates Range responses, a cross-origin 206 that hides Content-Range included, from the paths the source’s pattern names.
• URLSourceRetention.ts: keeps the last run’s URL source, or a prefetch’s, for the next run over the same media, released only by the run or prefetch that took it last, and sizes its read-ahead and cache by the media’s bitrate.
• MatroskaBlockAdditions.ts: reads a packet’s Matroska or WebM BlockAdditionals but alpha, such as VP9 HDR10+, from the fork’s packet side data.
• ContainerVideoScan.ts: a video track’s scan as its container declares it: Matroska’s FlagInterlaced and FieldOrder, which the fork’s Matroska demuxer reports as the track’s field order, or an ISO base media sample entry’s fiel box.
• CustomDecodeTrackSelection.ts: track lookup by ordinal within one media type.
• CustomAudioTrackMetadata.ts: the QuickTime DTS and Matroska MLP tracks whose codec Mediabunny does not map (DTS , A_MLP), DTS Express detection, declared-rate recovery for the DTS sample entries, the declared channel layout a track is screened with from its container and the host’s metadata layout, the native media route’s bed, and the decoded audio timestamp tolerance.
• WaveFormatAudioConfiguration.ts: the WAVEFORMATEX of a Matroska A_MS/ACM track or a codec-less AVI or ASF audio stream, written back from the fork’s parsed stream format fields, and the WMA codec its format tag names as FFmpeg’s RIFF tags do, its SubFormat’s for WAVE_FORMAT_EXTENSIBLE.
• VFWStreamFormat.ts: the FourCC and extradata of an AVI or ASF video stream’s BITMAPINFOHEADER, from the fork’s parsed stream format, for a stream Mediabunny names no codec for.
• AVIIndexCoverage.ts: whether an AVI track’s indexed packets end well before its stream header’s declared length, as a truncated file’s do, or an OpenDML file’s whose OpenDML index cannot be read.
• UnsupportedTrackFeatures.ts: the first feature, such as an edit list, that Mediabunny reports it cannot honor for a track.
• VideoChromaFormat.ts [main and worker]: a video track’s chroma from its decoder configuration, which the worker’s ready response names and every frame carries to presentation: its subsampling from an avcC (its closing chroma format or its first SPS), an hvcC, an av1C, or an AV1, VP9, or VP8 codecs string, and its 4:2:0 location from the first SPS’s VUI of an avcC or hvcC, or an av1C’s or AV1 codecs string’s chroma_sample_position; an H.264 or HEVC track without a record, or whose record holds no SPS, takes both from its first key packet’s in-band SPS, and MPEG-1 is center-sited 4:2:0.
• NALUnitFraming.ts: splits H.264 and HEVC data into NAL units at Annex B start codes or at an avcC’s or hvcC’s length prefixes, without copying them.
• AudioStartPacket.ts: the packet an audio attempt starts from, without Mediabunny’s proof scans when the track starts late or the start is at the playback timeline’s origin, and a decoder’s preroll, which never reaches before that origin.
• PlaybackTimelineOrigin.ts: which containers stamp their packets on a program clock, MPEG-TS, M2TS, and MPEG program streams, and where a run’s playback timeline starts in such a container: the first timestamp of the tracks it plays.
• TrackIteration.ts: a track’s packets in decode order, read about a second ahead, and its decoded video or audio samples from a start time, as iterators that end at once on return().
• ISOBaseMediaSampleEntryChildBox.ts: reads a child box of an ISO base media track’s first sample entry, such as fiel, from the moov bytes Mediabunny keeps.
• PresentationOrderTrack.ts: a track whose packets carry presentation timestamps for one that stamps them in decode order, as AVI and Matroska Video for Windows tracks do, by a codec’s reader of each packet’s picture order: an anchor takes the time of the packet FFmpeg outputs it at, and a B picture keeps its own; a key packet lookup steps back to the anchor that presents by the time.
• BitmapInfoHeader.ts: the BITMAPINFOHEADER of Video for Windows codec private data: its FourCC, printable or as hexadecimal bytes, and the header of a coded size and FourCC the engine builds before a codec’s extradata.
• JPEG2000TrackConfiguration.ts: the tracks the JPEG 2000 route reads, an MJ2 mjp2 sample entry or a Matroska Video for Windows track of a FourCC FFmpeg’s riff.c maps to JPEG 2000, which pass no codec private data.

demux/av1/

• AV1OBUParser.ts: the OBU walk of one temporal unit (headers, extension headers, leb128 sizes, a last OBU without a size field), frame header detection, metadata types and ITU-T T.35 messages, and the removal of a payload’s trailing bits.
• AV1SequenceHeaderParser.ts: the sequence header, parsed as the AV1 specification defines it, with the bit offsets of its color_config and trailing bits.
• AV1SequenceHeaderColorRewriter.ts: the bit-exact rewrite of a sequence header’s color description and range, and of every sequence header OBU of a temporal unit or av1C configOBUs.

demux/dolby-vision/

• DolbyVisionProfiles.ts [main and worker]: the dual-layer profile set (P4, P7), and the 1 us tolerance within which a base and an enhancement layer timestamp pair, as packets and as decoded frames.
• DolbyVisionHEVCSplitter.ts: NAL unit headers over NALUnitFraming.ts, with their temporal ID; the BL, RPU (62), and EL (63) split; RASL handling; in-band SPS neutralization; drops NAL units of layers above 0, the MV-HEVC second view.
• DolbyVisionAV1Splitter.ts: removes the Dolby Vision T.35 metadata OBUs from an AV1 temporal unit and returns their messages; other metadata, HDR10+ included, stays.
• DolbyVisionEncodedMetadata.ts, DolbyVisionEncodedMetadataProtocol.ts: per-packet RPU parsing for HEVC and per-temporal-unit parsing for AV1, which removes the RPUs unparsed on a route without Dolby Vision, over one PTS-keyed window, and the transferable schema.
• DolbyVisionEncodedPacketPairer.ts: BL and EL packet pairing, within the tolerance of DolbyVisionProfiles.ts.
• DolbyVisionRPUParser.ts, DolbyVisionRPUParserSession.ts, DolbyVisionRPUDataLayout.ts: the libdovi WASM parser with its HEVC and AV1 T.35 entry points, one instance per worker with a context for each parser, its per-run session, and the packed 3232-byte snapshot layout (schema 2: a mapping method per segment).
• WorkerWASMInstanceCache.ts: the one instance of a WebAssembly module that a worker’s users share, one cache per kit with the source it loads from as its version, discarded after its code traps; the libdovi parser uses it.
• MatroskaDolbyVisionHVCE.ts, ISOBaseMediaDolbyVisionConfiguration.ts, MPEGTransportStreamDolbyVisionConfiguration.ts: find a P4 or P7 EL configuration in Matroska; in MP4, as a separate EL track or as hvcE beside the BL hvcC of one interleaved track; or in MPEG-TS and M2TS (any descriptor version).

demux/hdr/

• HDR10PlusMetadata.ts: ST 2094-40 parsing from a frame’s ITU-T T.35 messages, and from an HEVC access unit’s SEI.
• AV1HDR10PlusMetadata.ts: the HDR10+ messages of an AV1 temporal unit’s T.35 metadata OBUs, without their trailing bits, parsed by HDR10PlusMetadata.ts.
• HEVCStaticHDRMetadata.ts, AV1StaticHDRMetadata.ts: the MDCV and CLL of an HEVC access unit’s SEI and of an AV1 temporal unit’s metadata OBUs, and the AV1 PQ sequence header check that gates its scan.
• StaticHDRMetadata.ts [main and worker]: the MDCV and CLL luminance schema, the codec-neutral startup scan, the scan-result validators shared across the worker boundary, and the static tone mapping source peak.

demux/hevc/

• HEVCDecoderConfiguration.ts: reads an hvcC record: its profile, chroma format, bit depth, NAL length size, and its base layer’s parameter sets.
• HEVCSEI.ts, HEVCSPSParser.ts: SEI extraction and the alternative transfer characteristics value; SPS parsing within the progressive Main and Main 10 bounds, the VUI color mapping to WebCodecs names, and the VUI rewrite with its native HDR route check, and the chroma format, VUI chroma location, and temporal sub-layer count of any profile’s SPS.
• HEVCTemporalSubLayers.ts: the highest temporal sub-layer every SPS of a track declares, from its hvcC record and in band, and whether an access unit is a picture no later picture references.

demux/mpeg/

• MatroskaMPEGVideoConfiguration.ts: a Matroska V_MPEG1 or V_MPEG2 track’s CodecPrivate, the sequence header FFmpeg hands its decoder, or none.
• MPEGTransportStreamVideoConfiguration.ts: the engine codec of an MPEG-TS, M2TS, or MPEG program stream MPEG-1, MPEG-2, or VC-1 track, and its codec private data, the extradata of its first packet with a sequence header, as FFmpeg’s extract_extradata takes it.

demux/vc1/

• MatroskaVFWVideoConfiguration.ts: reads a Matroska Video for Windows track’s FourCC and extradata as FFmpeg does, and from them or an ASF stream’s (readVFWVC1DecoderDescription, readVFWWMV3DecoderDescription) VC-1’s extradata after a WVC1 or WMVA header, and WMV3’s STRUCT_C behind a new header of the track’s coded size, since STRUCT_C holds no frame size.
• ISOBaseMediaVC1Configuration.ts: a vc-1 sample entry’s dvc1 sequence header and entry point, Advanced profile only, which the fork hands on as the track’s description.
• VC1TrackConfiguration.ts: the codec private data and packet track of a VC-1 or WMV3 track, per codec and container.
• VC1PresentationOrder.ts: reads a VC-1 or WMV3 picture’s order from its picture header, for PresentationOrderTrack.ts to restamp a Matroska VC-1 or WMV3 track by FFmpeg’s rule: an I or P picture takes the next I or P picture’s time, and a B picture keeps its own.

demux/mpeg4/

• MPEG4TrackConfiguration.ts: the codec private data and packet track of an MPEG-4 Part 2, H.263, or MS-MPEG4 track: its codec ID per container, a Video for Windows or AVI FourCC of the negotiated codec as FFmpeg’s riff.c maps it, and a BITMAPINFOHEADER of the track’s coded size and FourCC before its extradata.
• MPEG4PresentationOrder.ts: reads a video object layer’s low_delay, vol_control_parameters, and time increment width, and a VOP’s coding type and vop_coded, for PresentationOrderTrack.ts to restamp an AVI or Matroska VfW MPEG-4 Part 2 track FFmpeg outputs one VOP late, a layer without vol_control_parameters by whether its first group holds a B-VOP, passing over N-VOPs.

demux/vp9/

• VP9FrameParser.ts: splits a packet at its superframe index and tells from the frame headers whether it shows a frame.

decode/

The second stage: the WebCodecs decoders the engine owns, the GPU decoder package route, the audio decoders, and the package loading that video and audio share. The video modules are worker code unless marked.

decode/video/

What both video routes share.

• VideoPresentationInterval.ts: whether a frame presents on a presentation interval’s grid: some multiple of the interval falls within it.
• RawVideoFrameCopy.ts: copies a VideoFrame into aligned plane buffers (NV12, and I420 to I444P12). A Dolby Vision pair is a BL in any format from I420 to I444P12 and an I420P10 EL, in one compound buffer. It also names each format’s bit depth and the BL formats Dolby Vision reconstruction samples, which the presenter and the host check.
• RawFrameBufferPool.ts: a run’s spare raw buffers, recycled by the page or left by frames closed unposted, which drained frames and raw copies take; it allocates when no spare fits and keeps at most 2 of each byte length.
• DecodedVideoGeometry.ts [main and worker]: the first-decoded-size lock.
• VideoFields.ts [main and worker]: a stream’s scan, a frame’s fields, and the picture fields word GPU decoders and parsers report them in.
• HDR10PlusFrameMetadataQueue.ts, HEVCDynamicHDRMetadataQueue.ts: PTS matching of each packet’s HDR10+ result to its decoded frame, for any codec, with the last metadata carried in decode order to frames without their own, and the HEVC packet step that feeds it.

decode/video/native/

The owned WebCodecs paths.

• OwnedVideoDecodeStream.ts: the codec-neutral state of one owned decode attempt: decoded outputs matched with their packets’ metadata, the pre-start rule, decimation by the presentation interval, BL and EL pairing, frame credits, and the packet pump, which waits briefly after each packet for the decoder’s next output, skips packets a path marks as never presenting, and waits for the GPU’s decode of the batch before once a preroll’s or decimation’s unshown frames fill a batch. It pairs a path’s Dolby Vision and HDR10+ queues into the metadata its frames take (createOwnedVideoFrameMetadataSource). It also runs a single-layer attempt (runOwnedSingleLayerVideoStream), given a per-packet metadata step.
• VideoDecodeFence.ts: the decode worker’s fence over native frames, a WebGPU device of its own whose one-texel read of a frame completes once the GPU decoded it.
• OwnedNativeVideoDecoder.ts: an owned WebCodecs decoder for packets that decode unchanged, used for AV1 and VP9.
• OwnedAV1VideoStream.ts: one attempt of the owned AV1 path, which every AV1 track takes: each temporal unit’s Dolby Vision RPU and HDR10+ metadata, paired with its frame, after an optional color neutralization.
• OwnedVP9VideoStream.ts: one attempt of the owned VP9 path, which every VP9 track takes: the HDR10+ in each packet’s container side data, paired with its frame.
• OwnedNativeHEVCVideoDecoder.ts: the engine’s own WebCodecs HEVC decoder, built on OwnedNativeVideoDecoder: NAL order fix, leading RASL drop, repair of an SPS without default_display_window_flag, optional SPS neutralization.
• NativeHDRHEVCColorNeutralizer.ts: rewrites the hvcC SPS and the decoder colorSpace to BT.709 at the stream’s own range for the native external HDR route, and repairs an hvcC SPS without default_display_window_flag on any route.
• NativeHDRAV1ColorNeutralizer.ts: the AV1 counterpart: the decoder colorSpace, codecs string, and av1C, and each temporal unit’s sequence headers, each checked against the route’s limited BT.2020 color first.
• DolbyVisionFramePairQueue.ts: BL and EL frame pairing, within the tolerance of demux/dolby-vision/DolbyVisionProfiles.ts.

decode/video/gpu/

The video half of decoder packages: WGSL compute decoders, which write decoded frames into a pool on the presenter’s device, and their optional CPU parsers. The frame-pool route plays them: GPUVideoDecodeFeed.ts feeds a session the worker’s packets, and presentation/PresentationSetup.ts offers the host’s packages and creates sessions on the presenter’s device. The engine holds no decoder; the decoder projects in wgpud/ build packages.

• GPUDecodeTypes.ts [main]: the decoder manifest, stream facts, packets, output events, and coded failures.
• GPUVideoDecoderCatalog.ts [main]: the offer of one video decoder package for one output format, with the device’s buffer limit, which the eligibility selects from, and the summary of a package a host has not loaded, which judges a stream by the same fit check when it carries the package’s header.
• GPUDecoderStreamFit.ts [main]: whether a decoder fits a stream on a device (its accepts on the picture size, pool, state, scratch buffers, and packet ring), which eligibility checks at the container’s size, and the pool a session codes at: a first session’s at the largest picture its parser declared when the decoder fits it, a stream change’s at its sequence’s requirement or the old pool grown to the picture.
• GPUDecoderManifest.ts, GPUDecoderPrelude.ts [main]: parse a decoder’s //! header, its error codes’ recoveries among it, or the header alone as a server’s catalog carries it, and generate its bindings and helpers, the status a packet reports into among them.
• GPUDecoderEntryPointModules.ts [main]: splits a decoder’s module into one module per entry point, of the declarations it reaches, each kept character at its line and column, or leaves code it does not understand whole.
• GPUFramePool.ts [main]: the pool buffer, its slot layout, and slot lifetimes, with counted holds on a frame a resync repeats.
• GPUVideoDecodeSession.ts [main]: runs a decoder: the packet ring, dispatches with each picture’s repeat counts, submitted whole or, when the host turns slicing on, in slices that a reset or a resync stops, every packet’s status readback, which its frame waits for, a recoverable error’s recovery, reordering or a parser’s output events, slot release, and the visible rectangle its first header or each picture reports.
• GPUDecoderPipelineCache.ts [main]: the compiled decoders a device keeps for later sessions, by package id and output format, with the package’s content hash as their version, so a package’s new version replaces its old one’s.
• GPUVideoDecodeFeed.ts [main]: the frame-pool route’s page side: it creates or reuses a GPU decode session per generation, hands a picture its session cannot hold to a session made for it (a stream change), keeps up to two slices of decode work in flight and submits the next as one completes or a status readback applies, returns packet credits, turns decoded frames into presentation frames, counts the session’s recoveries, and reports its failure and its drained end.
• GPUDecodeFormats.ts [main], GPUDecodeLimits.ts [main and worker]: the planar formats a decoder may write and the FFmpeg pixel formats each holds, and the route’s bounds.
• GPUDecoderParser.ts: one instance of a package’s parser module, with its configure, plan, plan_next, parse, finish, and decode calls, each picture’s record, output events, visible size, repeat counts, output format, and sequence maximum, and whether a failure resyncs.
• GPUDecoderParserPipeline.ts: an attempt’s parser on its coordinator instance: it plans each packet’s pictures, splits a picture’s units into parse jobs, retries a blocked plan or plan_next, and finishes the pictures in decode order.
• GPUDecoderCoordinator.worker.ts, GPUDecoderParse.worker.ts: a video decoder package’s coordinator worker, which a decode worker starts with its first attempt that needs one and keeps for its life, and its parse workers, which the coordinator worker keeps while the package’s id and content hash stay; they run GPUDecoderCoordinatorWorkerRuntime.ts and GPUDecoderParseWorkerRuntime.ts.
• GPUDecoderWorkerCoordinator.ts, GPUDecoderCoordinatorHost.ts, GPUDecoderCoordinatorProtocol.ts, GPUDecoderCoordinatorWorkerRuntime.ts: the decode worker’s coordinator worker, shared by its attempts, with its 30 s watchdog; its messages; and its runtime, which runs one pipeline at a time.
• GPUDecoderParseWorkerPool.ts, GPUDecoderParseProtocol.ts, GPUDecoderParseWorkerRuntime.ts: the parse workers, one per core beyond two, from one to six; the pool sends each job to the worker with the fewest outstanding and stops on a worker error or the watchdog.
• GPUDecoderVectorSet.ts [main]: reads and writes a package’s vectors section.
• GPUDecoderVectorCheck.ts, GPUDecoderVerification.ts [main]: one vector’s check on a device, and verifyGPUDecoderPackage, which runs it, through the package’s parser on a worker of its own, only when a host calls it.

decode/audio/

The audio half of decoder packages.

• AudioDecoderManifest.ts: parses an audio package’s //! header.
• AudioDecoderPackageInstance.ts [worker]: one decoder on its own instance of a package’s module, which drives version 1 of the audio decoder interface and copies each block out; the selection of a package and codec, and its validators across worker boundaries.
• PackageSoftwareAudioDecoder.ts [worker]: owns one package decoder in the audio decode worker, configures it with the track’s codec private data before its first packet, checks each block’s layout and rate, and stamps blocks without a timestamp after their packet’s earlier ones.

decode/packages/

The package file that both halves share (see Decoder packages).

• DecoderPackage.ts [main]: loadDecoderPackage, which compiles a package file and checks it in order, failing with a DecoderPackageError code and detail; the video and audio package types.
• DecoderPackageFormat.json: the section names and the vectors section’s magic and version, which the engine and tools/decoder-package-format.mjs share.
• DecoderHeader.ts: the //! header syntax both kinds of manifest share, the kind a source names, and error names.
• DecoderPackageIdentity.ts [main and worker]: a package’s identity, the form of the id: its header declares and the version, its id with the content hash, that the engine keys what it keeps for a package by.
• DecoderPackageBinary.ts: reads what the JavaScript API does not reflect from a module binary: its imports with their memory limits, and whether it defines a memory.
• DecoderPackageImports.ts [main and worker]: the import whitelist, the memory bound (16384 pages), the inert WASI stubs, and the instantiation that runs a reactor’s _initialize first.
• DecoderPackageStore.ts [main]: a host’s IndexedDB store of package files by package id, which loads each file with the engine’s checks, so a package’s new version replaces its old one.

audio/

• CustomAudioCodec.ts: the WebCodecs, Mediabunny PCM, and package codec lists; the package codecs are those the engine has no decoder of.
• CustomAudioSampleRate.ts: the sample rate contract, any positive integer.
• CustomCompressedAudioRoute.ts: the E-AC-3, DTS, and TrueHD route tables that the input channel counts derive from, the DTS profiles, and the TrueHD rate rule.
• CustomAudioOutputPolicy.ts: input channels per codec, the 3.0 layout requirement for decoders without a speaker mask, and the 48 kHz 2, 6, and 8-channel output contract with its 2 s ring.
• NativeMultichannelAudioOutput.ts: the speaker layout setting (selectCustomAudioSpeakerLayout) and the output layout rule (selectCustomAudioOutputLayout: objects take the whole layout; otherwise three channels, 5.0, and 5.1 to 5.1, 6.1 and 7.1 to 7.1 or 5.1, else stereo).
• AudioNormalization.ts: TrackGain and AlbumGain decibels to linear gain.
• AudioSampleWindow.ts: PCM windowing at a seek.
• SafeIntegerValidation.ts: the positive safe integer check the audio modules share.

audio/processing/

• StreamingAudioResampler.ts, StreamingAudioLookaheadLimiter.ts, StreamingAudioOutputPipeline.ts, StreamingAudioDownmixSettings.ts [worker]: the 48 kHz resampler with input timestamp reconciliation within 2 s (larger deviations throw, and a listener sees every correction) and a continuation across source rate changes, the 100 ms lookahead limiter, the pipeline that rebinds its source rate and enables the limiter late, and generation-scoped live gains that follow the decoded rate. The resampler’s and the limiter’s own storage and rendering are the JavaScript reference; given the output stage module, a kernel stores and renders instead, with the same bytes.
• AudioOutputStageModule.ts [worker]: the audio-output-stage WebAssembly module, loaded once per audio decode worker on its first attempt, with a once-warned fallback to the reference when it does not load, and its resampler and limiter kernels, freed on finalize, on close, or after their owner is collected.
• PCMChannelPool.ts [worker]: the spare channel buffers the worklet returns, each sized for the largest chunk, which the output stage writes later chunks into.
• DecodedAudioOutputStage.ts [worker]: binds one audio attempt’s output stage to the decoded rate and layout, checks them against the decoded PCM routes, and reports each bound format; close frees a stopped attempt’s kernels.
• CustomAudioDownmix.ts, CustomAudioDownmixAlgorithm.ts, CustomAudioChannelLayout.ts, CustomWaveChannelLayout.ts: downmix matrices per algorithm (three-channel beds have their own), algorithm IDs, layout tables with the output layouts (stereo, 5.1, 7.1, 5.1.2, 5.1.4, 7.1.2, 7.1.4) and by-name mapping and height folds into them, a 5.0 bed’s mix through 5.1, and WAVE mask mapping.

audio/output/

• AudioWorkletController.ts [main], AudioWorkletProcessorSource.ts [worklet], AudioWorkletProtocol.ts: the worklet node, the inline processor (which renders silence from a flush up to the first chunk, and plays the ring at the playback rate), and their messages, the rate among them.
• AudioWorkletTimeStretchSource.ts [worklet]: the WSOLA time stretcher the processor inlines, which plays media-rate PCM at a playback rate with its pitch kept, allocating nothing as it renders.
• AudioOutputDevicePresence.ts [main]: whether enumerateDevices() lists any audio output.
• BrowserAudioContextPool.ts, BrowserAudioContextPrewarm.ts, BrowserAudioWorkletPool.ts, BrowserAudioOperation.ts [main]: the shared 48 kHz context (never pooled when created without an output device), the prewarm lease, the leased worklet node, which forwards the playback rate, and timeouts.
• BrowserCustomAudioOutput.ts [main]: the production output: context, sink lease, worklet lease, the playback rate, the physical output time correction scaled to media time by the rate, in-place layout reconfiguration, and sinkchange reporting.
• CustomDecodeAudioBridge.ts [main]: each generation’s flush and its new channel for the producer, the producer’s progress against the processor’s consumption, which the end-of-stream drain reads, and the processor’s overflow and stale-generation failures.
• WorkletPCMProducer.ts [worker]: feeds one worklet processor over its own channel: it owns the credit window, checks each chunk’s layout, bounds, and continuity before posting it, and passes the buffers the processor returns to the channel pool.
• WebGPUAudioOutputManager.ts: the page-wide sink router (setSinkId), with its fallback chain, devicechange handling, output recovery poll and sink rebuild, picker, and UI snapshot.

audio/native/

AC-3 and E-AC-3 through a hidden <audio> and MSE.

• CustomDecodeNativeAudioBridge.ts, OwnedNativeMediaAudioBackend.ts [main]: the bridge and the owned media element, which parks at a late first fragment, delays play() by the gap, and advances by a late timer’s overshoot.
• NativeMediaAudioFMP4Remuxer.ts [worker]: the fMP4 remux of the selected audio track.
• NativeMediaAudioLimits.ts: the segment caps (2 s, 2 MiB) and the pending-append caps (16 segments, 4 MiB).

validation/

The authorization vectors are in capability/vectors/.

• RawHDRPresentationAuthorization.ts: raw route keys, readback of the route’s default plan, and the per-device registry.
• ExternalHDRPresentationAuthorization.ts: external PQ and HLG route keys per codec, HEVC Main 10 and AV1 Main 10, each checked with its own neutralized vector, its check and decode recorded as probe session steps and batched per codec and target format.
• DolbyVisionPresentationAuthorization.ts, ExternalDolbyVisionPresentationAuthorization.ts: raw Dolby Vision keys (per raw format: single-layer, the P4 and P7 bases, and the P4 and P7 FEL) and the external P5 and P8 keys, one registry each, checked with synthetic RPU vectors.
• GPUAuthorizationDeadline.ts: the shared 5 s timeout, cancelled on device loss.
• GPUCanvasReadback.ts: bounded GPU canvas pixel readback (5 s).
• GPUErrorScope.ts: the validation error scope the authorizations and the canvas readback push and pop in one task around their submit (captureValidationErrors), and its cleanup.

Outside src/

• test/helpers/enginePaths.ts: the engine root and the node_modules location, independent of the test runner’s working directory, and the folders from tools/constants.json.
• test/helpers/dolbyVisionAV1ITUTT35Payload.ts, test/helpers/dolbyVisionMixedRPUVector.ts: wrap an HEVC RPU in the AV1 EMDF T.35 container, and build RPUs with mixed and linear pieces.
• test/helpers/libraryAssets.ts: the asset build’s tables from scripts/library-assets.mjs, and a served binary read as bytes from the file the build copies, since tests have no server for its URL.
• test/helpers/decoderPackages.ts: video decoder packages for tests, assembled with the engine’s package tool or built in memory, and the decoder projects’ sources, vectors, and built packages.
• test/helpers/postprocessingPasses.ts: the pass files of the pass projects, one by project and name, or all of them as npm run passes copies them.
• test/helpers/audioDecoderPackageStubs.ts, test/helpers/scriptedAudioDecoderModule.ts, test/helpers/audioDecoderPackageResults.ts: the smallest audio package the loader accepts; a module of the audio decoder interface scripted in JavaScript, found again after a structured clone by a custom section; and the packets and results the decoder projects’ tests use.
• test/helpers/webAssemblyModuleEncoder.ts: encodes small WebAssembly modules without a toolchain, for the package loader’s and the interfaces’ tests.
• test/helpers/fakeIndexedDB.ts: an in-memory IndexedDB of the parts the package stores use: versioned opens with upgrades, keyed object stores, and transactions that apply their writes only when every request succeeds.
• test/helpers/vc1Media.ts: synthetic VC-1 and WMV3 sequence and picture headers, Matroska Video for Windows files, and MP4 vc-1 files.
• test/helpers/mpegVideoHeaders.ts: synthetic MPEG-1 and MPEG-2 sequence headers, sequence extensions, and I picture packets, for tests that read a stream’s sequences without decoding its pictures.
• test/helpers/mpeg4Bitstream.ts: synthetic MPEG-4 Part 2 video object layers, extradata, and VOP headers of each coding type, coded or not, for tests that order VOPs without decoding them.
• test/helpers/mpeg4Vectors.ts: the media of the mpeg4 decoder project and the DivX packed AVI vector, with each one’s codec, FourCC, size, extradata, and frame order as FFprobe reports them, and a packet credit return for frame-pool runs.
• test/helpers/matroskaElements.ts, test/helpers/isoBaseMediaBoxes.ts: EBML elements and ISO base media boxes, and a file of one video track in each, with any codec ID or sample entry, CodecPrivate or none, and boxes ahead of ftyp.
• test/helpers/av1MetadataOBUs.ts: AV1 OBUs, metadata OBUs included, that the AV1 tests build temporal units from.
• test/helpers/hdr10PlusVectors.ts: the HDR10+ messages of HDR10PlusVectors.ts, and what the HDR10+ AV1 and VP9 vectors share: the expectations.json reader, the coded values in the engine’s units, and the check of a posted frame’s HDR10+ result.
• test/helpers/av1HDR10PlusVectors.ts, test/helpers/vp9HDR10PlusVectors.ts: each set’s files and its codec’s known answers, such as the AV1 static HDR metadata and the VP9 BlockAddID.
• test/helpers/ownedVideoStreamFakes.ts: a stream run, a packet iterator, and a decoder that outputs each frame at once or holds its frames until a flush, in decode or presentation order, for the owned decode path tests.
• test/helpers/decodeWorkerHarness.ts: loads a fresh playback worker in a stand-in browser: a global scope that plays the session’s part and waits for each run’s stopped, so one worker can run several generations, range responses for the media it plays, a WebCodecs video decoder, and the audio decode worker run in-process (inProcessAudioDecodeWorker.ts) on a channel.
• test/helpers/hevcMatroskaVector.ts: a one-frame Matroska HEVC Main 10 track muxed from the x265 HDR10+ sample, as Annex B with in-band parameter sets or length-prefixed with an hvcC.
• test/helpers/inProcessGPUDecoderWorkers.ts: a package’s coordinator worker and parse workers run in-process on channels, for the frame-pool route’s parser tests.
• test/helpers/fakeGPUVideoDecode.ts: a GPU decode session without a device, which decodes each packet into a frame at once, and a queue whose submitted work completes when a test says so.
• test/helpers/fakeWorkletProcessor.ts, test/helpers/decodedAudioMedia.ts: the worklet processor’s end of a producer channel, which records each chunk and returns it played, held, or dropped; and a Matroska file of the VP9 vector’s video and a stereo PCM tone, which Mediabunny decodes in Node.
• wasm/: the engine’s own WebAssembly modules and their build. See WebAssembly build.
• wgpud/: the submodule of the decoders repository, webgpu-player-decoders: the decoder projects, each of which builds one decoder package, with its tests in test/, and their shared build rules. See WebAssembly build.
• wgpup/: the submodule of the passes repository, webgpu-player-passes: the pass projects, whose pass files a host offers as optional passes, and their upstream license texts in licenses/. See Bundled passes.
• vendor/: the FFmpeg and dcadec submodules (update = none), which make -C wgpud sources fetches.
• scripts/build.mjs: assembles bin/libraries/, including the esbuild worker bundles, and bin/build-info.json. scripts/library-assets.mjs maps each served file to its source.
• scripts/pinned-releases.mjs: installs the pinned decoder and pass releases into bin/wgpud/ and bin/wgpup/, recording them in bin/installed-releases.json, and with --pin pins a release in tools/pinned-releases.json. See Pinned releases.
• scripts/decoder-packages.mjs: builds the decoder projects into bin/wgpud/, or checks them against it.
• scripts/pass-files.mjs: copies the pass projects’ pass files into bin/wgpup/, flat.
• scripts/codec_vector_assets/: the Python codec vector generators, their tests in test/, and the local playback media generators. See Codec vectors.
• tools/: browser probes, the decoder package tools, the mpv shader converter and the .cube LUT converter with their tests in test/ and the texture block writer they share, constants.json, and pinned-releases.json, the pinned decoder and pass releases, file by file with their SHA-256. See Tools.
• bin/wasm/: the engine modules’ builds from make -C wasm, libdovi/ and audio-output-stage/, served from libraries/; ignored.
• bin/wgpud/: the decoder packages of the pinned release, or those npm run decoders builds; ignored.
• bin/wgpup/: the pass files of the pinned release, or those npm run passes copies from the pass projects; ignored.
• bin/installed-releases.json: the files scripts/pinned-releases.mjs installed, with their SHA-256; ignored.
• bin/codec_vector_assets/: the generated codec vectors, the one committed folder in bin/:
  • truehd/TrueHDExactCapabilityVectors.ts: TrueHD and MLP access units and their expected outputs, which the truehd project’s tests import as #codec_vector_assets/*, and the synthetic streams the module embeds.
  • hevc-range-extension/: the nine range extension streams, which the native probes fetch from bin/libraries/ and the hevc project’s vectors hold.
  • av1-raw-format/: the eight AV1 raw format streams, which the native probes fetch from bin/libraries/ as webgpu-player/av1-raw/.
  • h264-raw-format/: the five H.264 raw format streams, which the native probes fetch from bin/libraries/ as webgpu-player/h264-raw/.
  • dolby-vision-av1/: the Profile 10 test vectors and their expectations.json.
  • hdr10plus-av1/: the HDR10+ AV1 test vectors and their expectations.json.
  • hdr10plus-vp9/: the HDR10+ VP9 test vectors, in WebM and Matroska, and their expectations.json.
  • downmix-reference/: the deterministic 7.1-to-stereo reference.

Embedding the engine

A host imports the engine’s TypeScript, serves its prebuilt workers and WebAssembly modules, and configures it once at startup. Decoders are not among them: a host hands the engine the decoder packages it loads, at runtime. Read the engine’s folders from tools/constants.json (sourceDirectory, scriptsDirectory, libraryOutputDirectory, buildInfoFile) rather than restating them.

You need

• The engine as a git submodule of the host.
• The toolchain of the engine’s modules from WebAssembly build, for the first build of each checkout: Emscripten and rustup.

Steps

1. Make the sources resolvable. Map webgpu-player/* to the engine’s src/* in the host’s TypeScript paths and in its bundler alias. The sources import nothing through the imports field of the engine’s package.json; only the decoder projects’ tests do.

2. Install the engine’s dependencies. Add the engine to the host’s npm workspaces, so npm installs them, esbuild for the asset build included. Without a workspace, install the engine’s dependencies and esbuild in the host.

3. Build the engine’s modules once per checkout, and again after their sources change: make -C <engine>/wasm all. The asset build reads them from <engine>/bin/wasm/ and stops when that folder is missing.

4. Build the served assets before bundling: node <engine>/scripts/build.mjs, with --production for a production build. It bundles the four engine workers with esbuild as classic workers. It copies the engine’s modules, their licenses, and the native range extension probes’ streams into <engine>/bin/libraries/. It also writes <engine>/bin/build-info.json, whose assetKey hashes every served file.

5. Serve <engine>/bin/libraries/ unmodified, and do not minify it again. The default asset base is libraries/ beside the page. Workers live in its webgpu-player/ folder and resolve the other assets relative to their own URL, so keep the layout intact.

6. Configure the engine at startup:

   import { configureEngineAssets } from 'webgpu-player/EngineAssets';
   import { configureEngineFeatureFlags } from 'webgpu-player/EngineConfiguration';
   import 'webgpu-player/style.css';
   
   // assetKey comes from <engine>/bin/build-info.json, written by the asset build
   configureEngineAssets({ cacheKey: assetKey });
   configureEngineFeatureFlags({ isHDRToneMappingEnabled: () => Promise.resolve(true) });
   

   Pass baseURL to serve the assets from somewhere other than libraries/.

How assets are named and served

• src/EngineAssets.ts names every runtime asset, scripts/library-assets.mjs maps each served file to its source, and the EngineAssets test keeps the two in agreement. scripts/build.mjs fails when an asset it maps is missing, which also stops a host build that runs it.
• Asset URLs are stable. The cache key is appended as ?v=, so a browser never mixes files from two builds.
• Engine workers are served at libraries/webgpu-player/<Name>.worker.js, not as host bundler chunks.
• The range extension probes’ streams are served with a .bin extension, because some static file servers reject unknown extensions. They come from bin/codec_vector_assets/.
• The served folder holds each module’s license beside it. The engine serves no LGPL code; whoever distributes the LGPL decoder packages offers their source (see WebAssembly build).

Gotchas

• Reload the page fully after deploying a new build. Open pages keep their old bundles; engine workers and modules change their ?v= key with every build.
• A bundler dev server that runs the asset build at configuration load must be restarted after an engine worker changes.
• WebGPU needs a secure context. Over plain HTTP on a LAN address, a host that still selects the WebGPU player plays HTML pass-through only, which proves nothing.
• The presenter places its canvas over the surface’s video element and fits frames by that element’s CSS object-fit, so the element needs a layout box. Hide it with visibility: hidden or cover it, but never with display: none or the hidden attribute, which leave every frame undrawn as surface-empty.
• While the black bar crop is on (WebGPUPresenter.setBlackBarCropEnabled), the presenter fits the picture the bars leave rather than the whole frame, so the picture’s rectangle on the surface differs from the frame’s. Layers a host draws over the video, such as subtitles, place themselves over the element’s box and know nothing of the crop; the presenter’s telemetry gives it as blackBarCrop.

Decoder packages

A decoder package is one .wasm file that adds a video or an audio decoder to the engine. The engine has no decoder package of its own and names none: a host hands it the packages it loads, at runtime, and the engine reads what each one decodes from the package itself. Packages are a fallback beside the browser’s decoders: a native WebCodecs route comes first, and a package plays only what the browser cannot. A package carries no JavaScript, so a host may load packages from anyone.

The package file

A package is a WebAssembly module with custom sections; src/decode/packages/DecoderPackageFormat.json names them for the engine and its tools.

The header’s decoder: directive makes the package a video or an audio decoder. A video package’s module is its parser, when its header declares parser:, or a module without code; an audio package’s module is the decoder.

Identity

A package’s id is the one its header’s id: directive declares, and every version of the package keeps it. Its content hash, the lowercase hexadecimal SHA-256 of the whole file, tells its versions apart. src/decode/packages/DecoderPackageIdentity.ts holds the rules.

• Every header, video or audio, declares id: once.
• An id is 1 to 64 characters: lowercase ASCII letters and digits, in runs joined by single hyphens, such as av1, jpeg2000, or hevc-main-10.
• A header without one fails as directive-missing, a malformed one as id-syntax, and a longer one as id-length, so the package fails to load as package-manifest.
• A host’s store keeps one file per id, so a new version replaces the old one.
• The engine keeps what it derives from a package, such as compiled pipelines and parse workers, by id with the content hash as the version, so a new version replaces what the old one left, and nothing counts or ages entries out.

loadDecoderPackage(bytes) in src/decode/packages/DecoderPackage.ts compiles the module and checks the package in this order. It rejects with a DecoderPackageError whose message is the failed check’s code and a terse detail.

Imports and memory

A package’s module may import only these:

• env.memory, once: the memory the engine creates for each instance.
• env.emscripten_notify_memory_growth.
• From wasi_snapshot_preview1: args_get, args_sizes_get, clock_time_get, environ_get, environ_sizes_get, fd_close, fd_fdstat_get, fd_read, fd_seek, fd_write, proc_exit, and random_get.

The engine’s WASI calls are inert: arguments and the environment are empty, output goes nowhere, reads and seeks fail, the clock reads zero, randomness is zero bytes, and exit traps the call. src/decode/packages/DecoderPackageImports.ts holds the whitelist and the stubs.

A parser and an audio decoder must import their memory as env.memory; a video package without a parser imports none. Each instance gets a new memory with the module’s declared initial pages, which may not exceed 16384 pages (1 GiB). Its maximum is the module’s declared maximum, or 16384 pages when the module declares none or more. A module that exports _initialize, as a WASI reactor does, has it run before any other export.

Video decoders

A video decoder is WGSL: a //! header, then a body of compute entry points. The engine compiles the body behind a prelude of its own for the output format a session decodes into, and runs the header’s dispatch sequence once per packet on the presenter’s device. Decoded frames stay in a frame pool, one storage buffer of slots, which presentation reads in place.

The header

A header is the source’s leading lines that start with //!, one name: value directive per line; a bare //! line is skipped. src/decode/video/gpu/GPUDecoderManifest.ts parses it, and a malformed header fails with a code, a detail, and the line.

The codecs in accepts are the engine’s codec names, which are FFmpeg’s: av1, h263, h264, hevc, jpeg2000, mpeg1video, mpeg2video, mpeg4, msmpeg4v1, msmpeg4v2, msmpeg4v3, vc1, vp8, vp9, and wmv3. A size is a byte count with an optional B, KiB, or MiB, or an expression the session evaluates for each stream. Sizes, workgroup counts, and repeat counts are expressions over these facts, in the presentation passes’ expression language: numbers, + - * / %, comparisons, && || !, and abs, ceil, floor, max, min, and round. The accepts conditions read the stream and picture facts, and no output facts; the picture facts are theirs alone.

The dispatch sequence runs in order for every packet. A direct dispatch whose workgroup count is zero for the stream is skipped, and an indirect dispatch reads its counts from the named buffer, which an earlier entry point wrote. A repeat suits a wavefront, which dispatches once per wave: its entry points find their iteration in the decoder’s own state, and a count of zero skips the run. A parser may report a smaller count for each picture, such as the waves its picture needs, and the picture’s record runs the repeat that many times. A packet’s sequence holds at most 4096 dispatches once its repeats are expanded.

//! id: example-intra
//! name: Example intra decoder
//! decoder: video
//! accepts: codec in [jpeg2000], stream.width <= 4096, stream.height <= 4096, stream.bitDepth == 8
//! output: I420
//! packet: 8 MiB
//! state: 64 KiB
//! buffer: sliceCounts, 16, indirect
//! error: 1, sliceHeader
//! dispatch: parsePicture, 1
//! dispatch: decodeSlices, indirect sliceCounts
//! repeat: ceil(stream.height / 16), 1
//! dispatch: predictRow, ceil(stream.width / 16)
//! dispatch: reconstruct, ceil(stream.width / 16), ceil(stream.height / 16)

@compute @workgroup_size(1)
fn parsePicture() {
    // Reads the picture header with readPacketByte and writes the slice count into sliceCounts
}

The prelude

The engine compiles each output format’s module as its prelude followed by the decoder’s body, so one body serves every format it lists. src/decode/video/gpu/GPUDecoderPrelude.ts generates the prelude, and compile messages map back to the body’s lines. The body declares no bindings, no @group or @binding, and no enable directives, and its scratch buffers take none of the prelude’s names.

Each entry point compiles a module of its own, which src/decode/video/gpu/GPUDecoderEntryPointModules.ts makes of the whole one.

• A module holds the declarations its entry point reaches through the names they use, of the prelude and the body alike, with every directive and const_assert; the rest is blank.
• Every kept character stays at its line and column, so compile messages map back to the body as the whole module’s do.
• The split reads WGSL’s module scope: comments, which may nest, attributes, directives, and the fn, struct, const, override, var, alias, and const_assert declarations.
• Code it does not understand, a name declared twice, or an entry point no function declares, compiles whole for every entry point.
• When an entry point’s module reports an error, the whole module compiles: its errors are the decoder’s, as before, and a whole module without errors serves every pipeline.
• A declaration no entry point reaches compiles nowhere, so its errors go unreported.

Every binding is in group 0.

Each buffer binds whole, and the device must hold each one within its buffer limit, so a decoder whose working set can pass a device’s largest storage binding spreads it over several buffer: lines, as AV1 spreads its frame store over two, and its WGSL picks the buffer by index. A device holds at least eight storage buffers per stage, the prelude’s five among them, so a decoder may declare three storage scratch buffers on every device; more need a device that holds more, or the session fails as storage-buffer-count.

An error word holds an error code the header names in its high byte and the failing value in its low 24 bits: (code << 24) | value. A code without a recovery fails the session with decoder-error and the word, which the route describes by the code’s name and the value, such as sliceHeader: 12. A code with one recovers, as Errors and recovery describes.

Codec private data

The config binding holds what the decode worker reads from the track for its codec, as FFmpeg’s demuxers hand it to their decoders; configByteLength is 0 when the track has none.

How a session decodes

src/decode/video/gpu/GPUVideoDecodeSession.ts runs one decoder for one stream and output format.

• Each entry point’s pipeline compiles from its own module, as The prelude describes.
• The modules and their pipelines depend only on the package and the output format, so the device keeps each compiled decoder by package id and output format, with the package’s content hash as its version, in src/decode/video/gpu/GPUDecoderPipelineCache.ts. A playback’s session of a kept decoder compiles nothing, sessions created at once share one compile, and a failed compile is not kept. A package’s new version replaces every decoder its old one compiled, while sessions of the old one keep their own pipelines; nothing else drops a decoder. A self-test vector’s check compiles a decoder of its own.
• A session is created only once the worker has opened the file and described the track, and it waits for every pipeline of its decoder, since every packet dispatches every entry point. GPUVideoDecodeSession.precompile compiles a package’s decoder for an output format into that same cache before the stream is known, so a session created later compiles nothing and one created meanwhile joins the compile. A failed compile is left to the session, which fails the same way. Three calls in src/presentation/PresentationSetup.ts start it, earliest first:
  • warmGPUVideoDecoders(presenter, videoPackages) compiles, in the background, the decoders later playbacks would take from packages the host already holds. Only codecs no browser decodes count, since no native route comes before their packages, and each such codec and output format takes the first package that writes it, in the host’s order. Each package format is one runCustomDecodeBackgroundTask, so it waits behind the background probes, for an idle period, for no negotiation near, and for no playback; the warm-up ends at a dropped task or a device without a frame pool, and skips the rest of a package that fails to compile. Beyond the device’s own cache, it leaves the browser’s shader cache warm, so a later page’s session compiles in milliseconds.
  • precompilePredictedGPUVideoDecoder(presenter, offers, options) compiles the decoder predictGPUVideoDecoder(options, offers) in src/capability/CustomPlaybackEligibility.ts predicts while the host still negotiates the playback. It predicts only a stream of a codec no browser decodes, in a container the engine plays that carries the codec, with an SDR color and a scan and rotation a route presents, and takes the first offer that decodes and fits it, as eligibility would; predictGPUVideoDecodeTarget(options) gives its codec and format first, so a host loads only the packages that match. The audio and the presentation are not judged, so a playback they refuse has compiled a decoder it never uses.
  • precompileGPUVideoDecoder(presenter, eligibility) compiles only the frame-pool route’s chosen decoder, as soon as the host has the eligibility.
• Sizes and counts evaluate once per session, from the stream facts; nothing evaluates per packet, and a picture’s own repeat counts only lower a repeat’s count.
• A first session’s stream facts come from the container’s size, or from the larger of it and the largest picture the parser read in the codec private data, such as an AV1 sequence header’s maximum frame size, when the decoder’s accepts takes that picture size and the device holds its pool, state, scratch buffers, and packet ring; a stream whose frames grow past the container’s size, as AV1 resolution changes do, then fits the pool. selectGPUDecodeSessionStream in src/decode/video/gpu/GPUDecoderStreamFit.ts decides, and eligibility checks the container’s size the same way, so a stream never becomes ineligible for its declared maximum.
• The pool holds the decoder’s references, its reorder depth, three slots for presentation, and two spares when the decoder keeps references.
• Each packet goes into a ring of two packets at a word-aligned offset, and its packet uniform names the slot it decodes into. With a parser, each of the session’s packets is one picture’s record, with its packet’s timestamp, duration, and keyframe flag.
• The first packet after a start or a seek carries reset; the engine clears the state and status buffers before it, and the decoder drops what it held.
• A decoder without references frees a slot when presentation releases it. A decoder with references holds every output slot until it calls releaseReferenceSlot; the engine reads the freed slots back a few frames behind, which the spare slots cover.
• Without a parser, frames present in timestamp order once reorder frames wait; a frame earlier than one already presented is dropped, as frames-dropped.
• With a parser, its output events present or drop each frame, as the next section describes.
• A frame presents its visible rectangle scaled into the stream’s display area: the visible size its picture reported, at the slot’s top left, or the stream’s visible rectangle.
• Every packet’s status reads back asynchronously, and the packet’s frame presents only once it has, as the next section describes.
• A picture the session cannot hold, of another output format or beyond its pool, goes to a session of its own, as Stream changes describes.

Errors and recovery

A damaged packet need not end a playback: a decoder that can go on past an error says how in its error: lines, and the engine counts each recovery.

• The session submits a packet only when a status readback buffer is free for it, and copies the status into it after the packet’s dispatches.
• The fatal error word and the header’s stream facts last until a reset clears the status; the packet’s block, its freed slots, its picture’s error word, and its picture fields, holds one packet’s reports, and the engine clears it after each copy.
• A frame presents only once its packet’s readback applied, so a recovery always applies before its frame could present.
• reportDecodeError keeps the packet’s most severe recoverable word: resync over drop over conceal, then the highest word.
• A resync replaces each frame of the damaged packet, and of the packets after it that already decoded, with a repeat of the last good frame: of the earlier packets’ frames that the session or presentation still holds, the one with the latest timestamp. Each packet up to the next key packet repeats it too, at its own time, without running the decoder. The repeats share that frame’s slot, on which the pool counts holds; without a good frame they drop. The decoder’s references end with the old epoch, and the key packet decodes with reset.
• With a parser, the session numbers repeated pictures as it numbers decoded ones, so the parser’s events keep naming them; an event for a repeated picture changes nothing.
• A parser call that fails with a code its header gives any recovery resyncs, since the failed call made no picture and left its instance unusable. The decode worker closes the parser, posts the failed packet and those before the next key packet as held packets, which carry no data and repeat the last good frame, and opens a new parser at the key packet, whose first picture carries reset. The session presents the frames the old parser left undecided in timestamp order, since no event will decide them. Without a later key packet, the packets up to the track’s end hold, and the track ends. A failed call whose code has no recovery, or a failed check of the interface, ends the attempt.
• A recovery’s detail is the error’s name and value, parser before a parser’s error name, such as coefficients: 90112 or parser sliceHeader: 3. The feed’s telemetry counts recoveries since the page loaded as recoveries, one entry per code, concealed, dropped, or resynced, and error name, with the latest detail.
• A self-test vector that recovers fails its check as decoder-recovered, since a vector is undamaged.
• Without a recovery for its codes, a decoder ends the attempt on every error, as before recoveries existed; the route then falls back as any failed route does.

Stream changes

A stream may change midway in a way its session cannot follow, as an AV1 sequence header that raises the maximum frame size or changes the bit depth or chroma subsampling does, or an MPEG-2 sequence header of a larger size, as a broadcast stream that switches resolution has. GPUVideoDecodeFeed in src/decode/video/gpu/GPUVideoDecodeFeed.ts checks each picture the parser reports against the request its current session was made for and that session’s pool, in getGPUVideoDecodeStreamChange.

• A picture needs a new session when it reports another output format, or a requirement its pool does not hold: the larger of its sequence’s largest picture and its own visible size on each axis, against the coded size of the pool its pictures decode into.
• The new session decodes only the new sequence’s pictures, so its pool takes the requirement alone, rounded up to 64, even below the pool before it, when the decoder’s accepts takes the requirement and the device holds the pool and the decoder’s buffers.
• Otherwise the pool takes the fallback, the pool before it grown to the picture’s own visible size, as for a declared maximum far beyond the frames, such as 16384x8704 for frames of 640x360.
• A picture its pool holds starts no session when its requirement is the maximum its session asked for already, or one the decoder’s accepts refuses, since its pool would be the fallback, the current one.
• The new session’s request keeps the decoder and the codec private data, and takes the picture’s format, the requirement as its maximum, and the fallback; chooseGPUDecodeSessionStream in src/decode/video/gpu/GPUDecoderStreamFit.ts chooses its pool.
• The feed creates it while the current session decodes on; the picture and every packet after it wait for it.
• Pictures queued while a session is created, the first one or a change’s, wait unjudged, and are judged against its pool once it exists, since the device may give it a smaller pool than its request asks for.
• The current session hears that its packets are complete, with the picture’s events that name earlier pictures, and presents its last frames.
• Once the current session decoded its packets and its decode work completed, the new session takes over, numbering its pictures on from the picture’s index so the parser’s events keep naming them, and its first packet clears its state.
• A replaced session stays until presentation releases its last frame, and is destroyed then.
• Presentation reads each frame’s sample size, bit depth, and chroma subsampling from its slot’s layout, and binds each pool as its frames arrive, so frames of every output format and pool present through one chain.
• A change can come only where the decoder needs no earlier picture, since the new session holds no references of the old one: at an AV1 key frame after a new sequence header, or at an MPEG-2 sequence header of a larger size or another chroma format, after which FFmpeg drops every reference too.
• A seek configures the session from the track again, and a picture after it that the session cannot hold changes the session again.
• A change whose requirement and fallback the decoder’s accepts both refuse fails the attempt as accepts: picture 4096x2160 in pool 4096x2176 at 8 bits, the fallback’s picture size, pool, and bit depth.
• A new session that cannot be made fails the attempt with its creation failure, such as format-unsupported: I444P12 not in I420, I420P10 for a format the decoder’s output lacks.

The parser interface

A parser is the package’s module: CPU code that turns each packet into what the WGSL reads, such as a picture’s parsed syntax, where entropy decoding is serial. It implements version 2 of this interface, and its header declares parser: 2. A packet may hold several pictures, such as an AV1 temporal unit’s hidden frames beside its shown one, and each picture has its own plan, parse jobs, finish, GPU record, output events, visible size, and repeat counts. src/decode/video/gpu/GPUDecoderParser.ts drives it.

Every export but the first two takes the handle parser_create returned; pointers, lengths, and counts are u32, and statuses i32.

A failure’s code decides what follows: a code the header gives any recovery resyncs at the next key packet with a new parser, as Errors and recovery describes, and any other ends the attempt. A parser that can go on past damage, as FFmpeg’s decoders conceal it, outputs the picture’s record with the damage marked and lets its WGSL conceal it and report a conceal code, rather than fail.

The engine calls plan_next after every 3 until a call returns 0, 1, or -1, and plans no later packet before; a packet holds at most 64 pictures. A plan without units parses nothing, so its finish receives no jobs. parser_decode and parser_decode_next run a packet on one instance, as a decoder project’s tests do; every earlier picture finished there, so a blocked decode fails.

A picture’s visible size is its top-left width x height samples, which must lie within the pool’s coded size, or the session fails as visible-rectangle; a report that gives one side without the other fails the attempt. Presentation scales each frame’s own visible rectangle into the stream’s display area, so a stream whose frame size changes, as AV1’s may, plays at one size; a picture without a report keeps the stream’s visible rectangle. A repeat count above the directive’s count for the stream runs the directive’s count, so a parser without a smaller one returns 4294967295, and 0 skips the run. The largest picture the codec private data declares sizes the session, as How a session decodes describes; the decode worker opens its first attempt’s parser before the track is ready, so the ready response carries it. A picture’s format and its sequence’s largest picture decide whether it needs a session of its own, as Stream changes describes; a format no output format holds, such as 4:2:0 at 9 bits, fails the attempt as parser picture format 0x109. A picture’s fields word travels with its record to the frame, which the deinterlacing pass reads (see Architecture); a parser without parser_picture_fields reports none, so its frames keep their container’s fields, and a word that sets a bit beyond the four, or any without bit 0, fails the attempt as parser picture fields 0x12. The engine copies each output and event list out of the instance’s memory before its next call on that instance, and fails the attempt when a pointer and length leave the memory.

The engine runs a parser on these instances, each with its own memory:

• A coordinator instance on the decode worker’s coordinator worker, webgpu-player/GPUDecoderCoordinator.worker.js, which configures, plans each packet’s pictures as the packet arrives, and finishes pictures one at a time in decode order. Every start and seek opens a new coordinator instance.
• One instance on each parse worker, webgpu-player/GPUDecoderParse.worker.js: one worker per core beyond two, from one to six. A picture’s units split into at most that many nearly equal, contiguous groups, which parse in parallel, and each parse call receives the serialized plan. The coordinator worker keeps its parse workers for later attempts while the package’s id and content hash stay the same.
• The decode worker reads packets until their pictures run eight ahead of its credits, a packet still parsing counting as one, so the workers stay busy while the page decodes.
• A worker that answers nothing for 30 seconds is ended, and every pipeline on it fails.

Each picture reaches the page as a packet of its own: its record with its packet’s timestamp, duration, and keyframe flag, its own run of the dispatch sequence into its own output slot, and one video credit. Only the first picture after a start or a seek carries reset, and the first picture of a packet carries the packet’s HDR10+ state.

An output event is a u32 word: bits 0 to 29 hold a picture’s index, counted from 0 in decode order since the instance started, modulo 2^30, every picture of a packet counting. Bit 31 drops the picture instead of presenting it, as for a picture its stream marks not for output or an AV1 temporal unit’s hidden frame, and bit 30 is reserved and must be zero. A picture’s plan or plan_next events and then its finish’s travel with its record; the events of a call that planned no picture wait for the next picture, or for the end of the track. The session holds each decoded frame until an event names it, and a picture an event drops frees its slot at once. An event for a picture that already left changes nothing, and an event for a picture not yet decoded fails the attempt as output-event. When more frames wait undecided than the header’s reorder, the earliest presents, so the pool never fills. At the end of a track the waiting events apply first, then the frames still undecided present in timestamp order. A picture’s record carries at most 256 events, its own and those carried to it.

Self-test vectors

A video package may carry known-answer vectors of its own in its webgpu-player.vectors section, at most one per output format. The engine runs one only when a host calls verifyGPUDecoderPackage(device, package, format) in src/decode/video/gpu/GPUDecoderVerification.ts; playback never runs them, and nothing caches the result. getGPUDecoderVectorFormats(package) lists the formats a package carries vectors for.

A check decodes the vector’s packets in a session on the device, through the parser when the package has one, reads back frames firstFrame to lastFrame in presentation order, and compares their SHA-256 with the vector’s. The hash covers each checked frame’s luma plane, then its two chroma planes, cropped to its visible rectangle, rows tightly packed and 16-bit samples little-endian: FFmpeg’s rawvideo output of those frames in the vector’s format, such as yuv422p10le. A frame’s visible rectangle is the visible size its picture reported, at the top left, or the vector’s, so a vector whose frame size changes checks each frame at its own size. A check returns passed with the hash, unverified when the package carries no vector for the format, or the check that failed: parser-failed, decoder-recovered with the first recovery, such as decoder-recovered: concealed coefficients: 90112, frames-dropped, frames-missing, slots-exhausted, hash-mismatch, or the session’s failure code.

The section is little-endian u32 words, names and data padded to whole words; src/decode/video/gpu/GPUDecoderVectorSet.ts reads and writes it.

Audio decoders

An audio decoder’s module decodes packets to planar float PCM, which the engine downmixes, resamples to 48 kHz, and plays like any decoded audio.

The header

An audio package’s decoder section is its manifest, a //! header like a video decoder’s; any text after the header is ignored. src/decode/audio/AudioDecoderManifest.ts parses it.

//! id: eac3
//! name: E-AC-3
//! decoder: audio
//! accepts: codec in [eac3, ac3]
//! packet: 2 MiB
//! attribute: 1, libraryVersion
//! error: 1, input
//! error: 2, send

Codec private data

The decode worker reads a track’s codec private data for its codec, as FFmpeg’s demuxers hand it to their decoders, and sends it with the attempt’s package selection to the audio decode worker. There PackageSoftwareAudioDecoder hands it to audio_decoder_configure once, before the attempt’s first packet; a track without any makes no configure call. A configure call that fails ends the attempt with the package’s named error.

The audio decoder interface

The module implements version 1 of this interface, which wgpud/include/audio_decoder_interface.h declares for C. src/decode/audio/AudioDecoderPackageInstance.ts drives it, on the playback worker’s audio decode worker.

audio_decoder_send returns 0 when it accepted the packet, 1 when it dropped it, 2 while it waits for a sync point, or -1 when it failed. A dropped packet, and one before a sync point, decode to nothing and fail nothing. After an accepted packet the engine calls audio_decoder_receive until it returns 0, and copies each block out of the instance’s memory: its channels’ planar floats, its layout, its timestamp, and every attribute the manifest names. A packet’s blocks number at most its byte length, and their frames at most 65536 per byte. A block’s channel mask must be one the engine mixes: mono, stereo, 3.0, 2.1, 3.0 with a back center, 5.1 with back or side speakers, 6.1, 7.1, or one of FFmpeg’s height layouts, 5.1.2 (0x503F), 5.1.4 (0x2D03F), 7.1.2 (0x563F), and 7.1.4 (0x2D63F), or 5.1 with side speakers and the top front pair (0x560F), which FFmpeg’s E-AC-3 decoder reports for a dependent stream with heights. A height layout’s channels follow its mask bits too: the floor’s, then top front left and right, then top back left and right. An E-AC-3 or TrueHD package’s block may have a height mask whatever channel count the track declares; any other package’s blocks must fit its codec’s input routes (see Audio routes). Every start and seek is a new attempt on a new instance, which audio_decoder_destroy ends, so a decoder never sees a jump in its packets.

A module may also export int audio_decoder_drain(handle), which returns 0 or -1. The engine calls it once at the end of a track, after the attempt’s last packet, then calls audio_decoder_receive until it returns 0, so a decoder that holds frames back, as FFmpeg’s do, outputs them; at most 64 blocks follow. A drained block without a timestamp of its own continues from where the last block ended, or, when no block came before it, as after a seek onto a track’s last packet, starts at the last packet’s time. A module without the export ends the track at its last packet’s blocks, and findMissingAudioDecoderExports never asks for it.

Object audio

A package that renders object audio, such as Dolby Atmos in E-AC-3 and TrueHD, declares two options and one attribute by these names; their keys are its own.

The audio decode worker sets both options with audio_decoder_set_option after audio_decoder_configure and before the attempt’s first packet (PackageSoftwareAudioDecoder.setRenderTarget), and sets renderBinaural again between two packets when the host switches binaural rendering live. A live change of the output layout starts a new attempt on a new instance, which takes the new mask before its first packet. A package that declares neither option is never asked, and one that declares only renderChannelMask gets no renderBinaural. The package renders object audio into exactly the mask’s layout, or into binaural stereo (0x3) with renderBinaural 1, and reports its blocks with that mask. Where the target adds nothing to what its stock decode outputs, the package outputs the stock decode with its own mask instead, and a stream without objects always does; the engine plays either, folding or mixing it to the output layout by channel name. The worker reports each change of the blocks’ objectCount to the page, whose decode telemetry shows it as decodedAudioObjectCount; a count above 1024 fails the attempt. A block with objects passes through the output stage’s peak limiter, since the package sums them unlimited.

Hosts

Loading and keeping packages

• loadDecoderPackage(bytes) returns a VideoDecoderPackage or an AudioDecoderPackage: its id, content hash, name, manifest, compiled module, memory limits, and, for video, its vectors section. The compiled module crosses to the engine’s workers, which instantiate it with memory of their own.
• DecoderPackageStore in src/decode/packages/DecoderPackageStore.ts keeps package files in IndexedDB by package id, under a database name the host chooses. add(fileName, bytes) loads a file with the engine’s checks before it stores it, remove(id) deletes one, and loadAll() loads every stored file, oldest first, with a failure for each one that no longer loads. load(id) loads the one stored file of an id, or null when the store holds none, so a host that caches packages it fetches compiles only the ones a playback needs. A file of an id the store holds is a new version of that package: it replaces the stored file and keeps its place in the order. The store’s schema is version 2, which keys files by id; a database of an earlier version opens empty, without its files.
• Where a host gets package files is its own choice: the engine fetches none. The decoder projects in the decoders repository, checked out at the engine’s wgpud/, build packages a host may offer, and that repository’s releases publish them (see WebAssembly build).

Offering packages to a playback

• getGPUVideoDecoderOffers(presenter, videoPackages, options) in src/presentation/PresentationSetup.ts returns the packages whose accepts names the stream’s codec and whose output lists its format, in the host’s order, with the device’s buffer limit, the smaller of its largest buffer and its largest storage binding. The host passes them as the eligibility’s gpuVideoDecoders.
• The host passes its audio packages, in its order, as the eligibility’s audioDecoders. With them it passes audioRenderTarget, the speaker layout it chose for the destination and its binaural headphone setting, so that a Dolby Atmos stream whose objects the target needs takes a package that declares renderChannelMask ahead of the browser’s own decoder (see Audio routes). The eligibility then sets audioRendersObjects, and the host plays the track to the whole speaker layout.
• A host that fetches packages only when a playback needs them asks first what the playback decodes: getGPUVideoDecodeTarget(options) in src/capability/CustomPlaybackEligibility.ts names the video’s codec and frame pool format, and getPlaybackAudioCodec(options) the codec of the audio track the eligibility selects.
• Before a playback, a host passes what its video packages decode as hasEligibleCustomVideoRoute‘s gpuVideoDecoderSummaries: their codecs and output formats, and where it has them, each package’s header and the device’s buffer limit, so the summary runs the offers’ fit check; see GPU decoders. parseGPUDecoderHeader in src/decode/video/gpu/GPUDecoderManifest.ts reads a header without its body, such as a server’s catalog carries, with every check of the manifest parser but the body’s.
• The eligibility prefers a native WebCodecs video route; otherwise it takes the first offer whose accepts conditions take the stream and whose pool, state, buffer:s, and packet ring each fit the buffer limit, on the frame-pool route with the gpu decoder backend. findGPUDecoderStreamFitFailure in src/decode/video/gpu/GPUDecoderStreamFit.ts checks them at the container’s size. An HDR stream takes a package only when the host tone-maps frame-pool HDR, the allowFramePoolHDR presentation option.
• For audio it prefers passing the track through to the browser, then the browser’s own decoder, then the first package whose accepts names the track’s codec, on the decoded PCM route.
• The output format of a stream is its HEVC range extension format when it has one, otherwise the format that holds its pixel format, such as I422 for yuv422p, or 4:2:0 at its bit depth when the stream names no pixel format. A 4:0:0 pixel format, gray, gray9, gray10, or gray12 in either byte order, takes the 4:2:0 format of its depth, and only a decoder whose header declares monochrome is offered it. A JPEG 2000 stream in xyz12le takes I444P12, which a JPEG 2000 decoder writes as limited-range BT.709 Y’CbCr of FFmpeg’s conversion of its X’Y’Z’ to R’G’B’.
• The controller takes gpuVideoDecodeSessionFactory: createGPUVideoDecodeSessionFactory(presenter, { slicing }), where slicing is the host’s choice of sliced decode submits, and play() takes the eligibility’s gpuVideoDecoder and audioDecoder.
• The host calls precompilePredictedGPUVideoDecoder(presenter, offers, options) without waiting as its negotiation starts, and precompileGPUVideoDecoder(presenter, eligibility) without waiting before the rest of its setup, so the decoder compiles while the negotiation, the presentation, the controller, and the worker’s open of the file proceed.
• A host may call warmGPUVideoDecoders(presenter, videoPackages) after its background probe warm-up with the packages it already holds; it downloads nothing.
• An audio track switch passes the new track’s audioDecoder from its own eligibility to setAudioStreamIndex, with its audioMetadataLayout (getAudioMetadataLayout); the previous track’s package and layout never carry over.

Verifying a package

Verification is the host’s choice and never the engine’s. A host that wants a package checked on the user’s device calls verifyGPUDecoderPackage for each format getGPUDecoderVectorFormats lists, before it offers the package, and decides what a failure means. Each call decodes again, so a host that keeps results keys them itself, by the package’s id and content hash and the device, since a new version of the id needs its own.

Failures

Every failure carries a code and a terse detail with the failing value.

• Loading: the DecoderPackageError codes above, such as package-manifest: id-syntax: 'AV1' (line 1).
• A video session: its failure code, such as compile-error with the body’s line, dispatch-count, decoder-error with the name and value of an error without a recovery, output-event, packet-size: ring 1073741824 > 268435456, or visible-rectangle: picture 1920x1088 exceeds 1280x768, which ends the route’s attempt as any failed route does.
• A recovery is no failure: its code and detail, such as concealed coefficients: 90112 or resynced parser sliceHeader: 3, count in the feed’s telemetry, and the playback goes on.
• A stream change: accepts: picture 4096x2160 in pool 4096x2176 at 8 bits when the decoder takes no pool for the picture, or the new session’s creation failure, such as format-unsupported: I444P12 not in I420, I420P10, which ends the attempt as the first session’s would.
• A parser: parser <error name>: <value>, or the step that failed, such as parser reserve_input refused 4096 bytes, parser events 300 > 256, parser pictures 65 > 64 in one packet, parser plan_next status 7, parser picture format 0x109, or parser maximum picture size 1920x0.
• An audio decoder: audio decoder <error name>: <value>, which ends the audio attempt.

Bundled passes

wgpup/ is a submodule of the passes repository, alchemyyy/webgpu-player-passes, which holds the pass projects: WGSL passes the engine bundles for a host to offer as optional user passes. Its name marks them as this player’s own module type, as wgpud/, the decoders repository’s submodule, marks the decoder projects. The engine never loads a pass on its own: a host adds one to its users’ presentation preset through WebGPUPresenter.setPresentationPreset, as it adds any pass file a user picks.

Projects

A pass project is a folder of wgpup/ whose top-level WGSL files are its pass files. Each file holds one or more passes in the pass file format, which run in file order, and no two projects use the same file name.

Every pass file keeps the rules that test/presentation/passes/PostprocessingPasses.test.ts checks: each pass parses, defines its shape’s entry point, reads exactly the parameters it declares, reads each input through its accessors, and reaches engine data only through the presentation globals. The presenter, manifest, and pipeline graph tests run the four simple passes as well, and the presenter tests run smooth motion.

Build and offer

A checkout installs the pass files of the pinned passes release into bin/wgpup/ with npm run pinned-releases, beside the served assets but outside their asset key (see Pinned releases). npm run passes (scripts/pass-files.mjs) copies every pass file of the wgpup/ submodule there instead, flat, replacing what was there; it fails on two files of one name, or when the submodule holds no pass file, as before git submodule update --init wgpup. The passes repository’s release workflow collects the same files, flat, with the same check. A host serves that folder and offers its files, and the sandbox’s shader panel lists them through its dev server (see Sandbox).

Anime4K

wgpup/anime4k/glsl/ holds the two upstream mpv shaders, copied unmodified from bloc97/Anime4K at commit 7684e9586f8dcc738af08a1cdceb024cc184f426. Its PROVENANCE.txt lists their SHA-256, which the pass tests check, and wgpup/licenses/Anime4K-LICENSE is the upstream MIT license. The generators beside them turn a shader’s convolution and depth-to-space passes into a pass file, with the weights copied verbatim:

• generate_anime4k_wgsl.py fuses the whole network into one compute pass: each workgroup evaluates a 16x16 source tile in workgroup memory, with a halo of one pixel per layer, and writes the 2x image with the bilinear-upscaled input added. It takes networks of 8-channel hidden layers whose tiles fit the default 16 KiB of workgroup storage, as L’s do.
• generate_anime4k_wgsl_chain.py writes one compute pass per layer into one file: each layer stores its 4-channel parts side by side in an rgba16float texture, and the last layer does the depth-to-space step and adds the bilinear-upscaled input. UL’s eight layers become the eight passes of its file.

The generated pass files are committed, so a build needs no Python. tools/convert_mpv_shader.py converts these shaders too, pass for pass (see Migrate an mpv shader to a wgpup); the generators fuse each network into fewer passes instead. To regenerate them, run from the engine root:

python wgpup/anime4k/generate_anime4k_wgsl.py wgpup/anime4k/glsl/Anime4K_Upscale_Denoise_CNN_x2_L.glsl wgpup/anime4k/anime4k-l.wgsl
python wgpup/anime4k/generate_anime4k_wgsl_chain.py wgpup/anime4k/glsl/Anime4K_Upscale_Denoise_CNN_x2_UL.glsl wgpup/anime4k/anime4k-ul.wgsl

Smooth motion

In smooth-motion.wgsl, each display refresh shows the frames it spans, the newest and the one before it, blended by the share of the refresh each covers, as libplacebo’s oversample mixer, which mpv documents as SmoothMotion, does. A refresh within one frame shows that frame untouched, and so does every refresh while the clock stands still, so a paused picture shows one frame. The blend runs in linear light of the display’s sRGB encoding, extended past 1 and mirrored below 0 on an HDR canvas.

Whether frames blend is decided on every draw, from the refreshes a frame lasts: the media time between the stream’s recent frames over the media time a refresh covers at the clock’s rate, so a playback rate, a new frame rate, or another display takes effect within frames. Its mode constant picks when frames blend:

• 0, the default: only where there would be motion judder without it, when a frame lasts no whole number of refreshes within tolerance.
• 1: also where it does, when a frame lasts other than one refresh within tolerance.
• 2: always.

tolerance is relative and defaults to 0.002, so 23.976 fps at 24 Hz counts as a whole multiple. A refresh that need not blend shows the frame the clock is in, as it would without the pass; so does every refresh until the display is measured, and, in modes 0 and 1, before a frame’s duration is known.

• It hooks OUTPUT in the display domain, so each frame is tone mapped with its own metadata before it enters history.
• A deinterlacing plan leaves it out as display-rate-unavailable.
• It costs a ring of two output-size rgba16float images and one pass per refresh; the passes before it run once per frame.

NIS

wgpup/nis/glsl/ holds agyild’s mpv port of NVIDIA Image Scaling v1.0.2, NVScaler.glsl and NVSharpen.glsl, copied unmodified, CRLF line endings included, from the port’s gist at revision 05f00864228871ffd157daa9beb2db8fa7412cfa. The passes repository’s .gitattributes keeps those files and wgpup/licenses/NIS-LICENSE byte for byte, so their SHA-256, which PROVENANCE.txt lists and the pass tests check, stays upstream’s. NIS-LICENSE is the MIT license of NVIDIAGameWorks/NVIDIAImageScaling at commit 35e13ba316c98eeecf16f37eae70ce88019911f6, which the port’s headers carry.

• The port’s scaler writes 32x24 output pixels per workgroup with 21776 bytes of workgroup storage, more than the 16384 bytes WebGPU guarantees. generate_nis_wgsl.py converts it at a 32x16 block, 16208 bytes, where each of its 256 threads still writes two pixels; it changes the block height and the COMPUTE directive in memory and leaves the upstream file alone.
• The sharpener fits as it is, so tools/convert_mpv_shader.py converts it directly.
• The converter tests regenerate both passes and require the committed files, so a converter change that alters them shows up there.

To regenerate them, run from the engine root, with naga-cli 30.0.1 installed (see Migrate an mpv shader to a wgpup):

python wgpup/nis/generate_nis_wgsl.py wgpup/nis/glsl/NVScaler.glsl wgpup/nis/NVScaler.wgsl
python tools/convert_mpv_shader.py wgpup/nis/glsl/NVSharpen.glsl wgpup/nis/NVSharpen.wgsl

Create a new WebGPU pass

A pass is a WGSL file with a //! header that the presentation planner places into a route’s pipeline. This chapter is the reference for the pass file format: the header, the code the engine generates around a pass, and the reasons a pass is left out of a plan. src/presentation/passes/PassManifest.ts parses the header, PassPrelude.ts generates the code around the body, and PresentationPlanner.ts places the pass.

You need

• A checkout set up as in Set up a checkout, for the tests and the sandbox.
• A browser with WebGPU, to see the pass run in the sandbox.

Steps

1. Create wgpup/<project>/ and a pass file in it, <file>.wgsl, under a file name no other project uses. Start from the bundled pass of the same shape: sharpen.wgsl for a kernel, saturation.wgsl for a color function, catmull-rom.wgsl for a scaler, temporal-blend.wgsl for a history read, smooth-motion.wgsl for a pass that draws at the display rate, and the Anime4K files for compute passes.
2. Write the header, then a blank line, then the body with the entry point of its shape (see Shapes).
3. To try it before building, run npm run sandbox and drop the file on the shader panel. The panel shows a plan state per pass and an input per parameter, and the pipeline graph lists a pass the plan left out as reason: detail.
4. Add the project to EXPECTED_PROJECTS in test/presentation/passes/PostprocessingPasses.test.ts, and run npx vitest --watch=false test/presentation/passes.
5. Run npm run passes, which copies the file into bin/wgpup/, where a host serves it. The sandbox lists that folder on each request, so the file then shows in the shader panel’s bundled pass selector.
6. Add the project to the table in Bundled passes.

A pass a user supplies at run time follows the same format, and skips steps 4 to 6. To start from an mpv user shader instead, convert it (see Migrate an mpv shader to a wgpup). A host loads any pass file through WebGPUPresenter.setPresentationPreset, as a PresentationPresetFile of its source and its stored parameters values.

Check the result

• The pass tests check every bundled pass file: each pass parses, its body starts with a blank line and defines its shape’s entry point, it reads exactly the parameters it declares as parameters.<name>, it reads each input through its accessors, and it reads no @group, @binding, renderSettings, textureSample, or textureLoad. Of the presentation uniform it may read only the fields in the test’s PRESENTATION_FIELDS; add a field there when a pass needs it.
• The sandbox’s pipeline graph shows the pass as a node, or as a fused color step of a neighboring pass, and lists it beside the graph as reason: detail when the plan left it out.

Shapes

• A coordinate is a normalized position in the hooked input, and a position is the target pixel’s center in pixels.
• A compute pass’s id is the global invocation ID, and its dispatch rounds up, so check id.xy against TARGET_size() before storing.
• The input accessors read with textureSampleLevel or textureLoad, so they work in compute passes, where WGSL has no textureSample.
• WebGPU’s default compute limits apply: 256 invocations per workgroup, and 16 KiB of workgroup storage.

Stages and color domains

The hook names the stage a pass attaches to, in pipeline order:

• LUMA, CHROMA, then NATIVE, the plane stages, run on the signal’s planes before it converts them into RGB (see Planes).
• RGB, then PRESCALE, run before scaling, where HOOKED starts at the source’s size, or at luma’s when plane passes replaced a plane. A pass there may change the size, as the Anime4K passes double it, and the scaler takes whatever size it is given.
• SCALE holds the scaler: the first user scaler that can run replaces the built-in bilinear one, and every other scaler is left out as extra-scaler. Kernels and color functions on SCALE run after the scaler.
• OUTPUT runs at the output’s size, before the canvas.

Passes on one hook run in preset order, and a file’s passes in file order.

The domain names the color state a pass needs on its input; the planner runs the built-in color functions, transfer decode, tone mapping, and output encode, up to that state before the pass:

The planner only moves forward along the color track, so a pass whose domain the pipeline has already passed, such as source after any built-in step has run, is left out as domain-unreachable. Dither always runs in the pass that writes the canvas, since its noise must not pass through a scaler. The plane stages come before the color track, so a pass there takes the domain any; any other is left out as domain-unreachable.

Planes

The plane stages hold the planes as mpv’s do, as normalized code values: a code divided by the largest code of its bit depth, so a limited-range black is 16/255 at 8 bits.

• A pass that writes its plane stage’s image replaces that plane; the plan then merges the planes and converts them with the signal’s own range and matrix, reading a plane no pass replaced from the source, so it converts as the signal does.
• Chroma resampled onto luma’s grid, by NATIVE or the merge, is read bilinearly through both images’ offsets; a chroma upscaler that writes chroma at luma’s size with offset: align is read texel for texel.
• A plane is read apart from the source only when a pass reads it; a plan without plane passes reads the source as it always does.
• The planes hold the frame’s samples, so an anamorphic stream’s planes keep its non-square samples, and the image the planes merge into starts at luma’s size; the scaler takes it to the display’s shape.
• Chroma’s size is luma’s over the subsampling divisor, rounded up, and its extent on the reference grid is luma’s over the divisor exactly, as libplacebo sizes a plane’s rect; an odd luma size leaves chroma’s last pixel half outside, so chroma’s scale stays exactly the subsampling’s.
• The raw and frame-pool routes hold the decoded planes.
• The external routes import a frame the browser already converted to RGB, so they derive the planes from it through the conversion that made it: the stream’s own matrix and range, at its bit depth, which the merge inverts exactly. The external HDR route reads the codes its signal recovers from the browser’s BT.709 texture, the engine’s own labeling of the stream. Chroma takes the stream’s subsampling from its decoder configuration, read at each chroma sample’s site, so a chroma upscaler runs there too; a stream that declares none has chroma at the image’s size.
• A deinterlacing plan derives the planes from its deinterlaced image through the same conversion, chroma at the stream’s subsampling.
• Dolby Vision routes have no planes to offer, so a plane pass, and a pass that reads a plane stage, is left out there as planes-unavailable.

Display rate

A pass with rate: display draws at the display’s refresh rate rather than once per frame, as smooth-motion.wgsl does.

• The engine draws it on every refresh from history, and the frame loop takes frames one refresh ahead of the clock, so the frame that starts within a refresh is already in history when the refresh draws.
• HOOKED is the newest frame, and HOOKED[-1] the frame before it.
• presentation.refreshSeconds is the media time one refresh covers, and presentation.targetOffsetSeconds the clock time the draw presents at minus timestampSeconds; the newest frame’s share of the refresh is clamp((targetOffsetSeconds + refreshSeconds) / refreshSeconds, 0, 1). refreshSeconds is 0 until the display rate is measured, and while the clock stands still, as when paused, when a refresh shows the frame the clock is in: the newest once targetOffsetSeconds reaches 0, else the one before it. The frame loop’s tick after a pause redraws at the stopped clock, so a paused picture shows one frame.
• presentation.frameDurationSeconds is the media time between the stream’s frames, a line fitted through the recent new frames’ timestamps, and 0 before a frame’s time is known: the first frame after a seek is timed by its own duration, if it has one. frameDurationSeconds / refreshSeconds is the refreshes a frame lasts at the clock’s rate, by which smooth-motion.wgsl decides on each draw whether frames blend, so a playback rate or a new frame rate takes effect within frames.
• A draw from history skips the passes that read the source and every pass before the first display-rate pass, whose outputs no draw of the frame changes, so a heavy pass before it runs once per frame.
• The pass that reads the source then writes a texture rather than the canvas, and a color pass at the display rate gets a pass of its own rather than fusing into it.
• A plane stage reads the source, which a draw from history skips, so rate: display on LUMA, CHROMA, or NATIVE is a header error.
• Where frames are timed by the browser’s video element, or the plan deinterlaces, whose history holds fields rather than the frame after the shown one, the pass is left out as display-rate-unavailable, with the detail video-element or deinterlace.

Offsets

Every image the plan holds has an offset: the pixel of the image, in its own pixels, that the reference grid’s origin falls on, as libplacebo’s texture rects place them.

• The source’s luma and the signal’s RGB have the offset 0, and chroma has its siting: 4:2:0 chroma at the location the stream declares, and 4:2:2 chroma on its left luma column. Left-sited chroma, H.264’s and HEVC’s default and the engine’s where a stream declares no location, has (0.25, 0), half a luma pixel; top-left chroma, as BT.2020 sites it, has (0.25, 0.25).
• A kernel or compute pass’s output has the hooked image’s offset scaled by the output’s size over the hooked image’s, plus its declared offset, or 0 with offset: align.
• The engine never shifts a read: <P>_tex reads where the pass asks, and <P>_offset() gives the input’s offset, so a pass that aligns itself reads HOOKED_tex(coordinate + HOOKED_offset() * HOOKED_pt()), as libplacebo’s OFFSET ALIGN moves HOOKED_pos.
• The scaler corrects its image’s offset within its own resample, built-in or user, so an image that a pass doubled with offset: -0.5, -0.5 lands where the source did.

Header

A pass is a header, a run of lines that start with //!, and its body, which runs up to the next line that starts with //!. A file holds one or more passes, as mpv’s user shaders do, so a network of several passes ships as one file, and any texture blocks its passes bind (see Texture blocks).

//! name: Sharpen
//! shape: kernel
//! hook: OUTPUT
//! input: HOOKED
//! output: HOOKED.size, rgba16float
//! domain: sdr
//! param strength: f32 = 0.5 in [0, 1]

fn kernel(coordinate: vec2f) -> vec4f {
    ...
}

• A header line is //! <directive>: <value>, with any whitespace around the tokens, and \r\n line endings read as \n.
• Only input, bind, param, and const repeat; an unknown, repeated, or empty directive is an error, as is a directive that the shape does not take.
• Only blank lines may come before a file’s first header, and every line that starts with //! belongs to a header, so a body cannot hold one.
• Each pass of a file parses on its own, so a malformed header leaves the file’s other passes in place. An error carries the line of its directive, or the header’s first line for a missing required directive, counted from the file’s first line.
• An identifier is [A-Za-z_]\w*.

Inputs and outputs

• NAME is HOOKED, the image at the hook, a stage name, which reads that stage’s final image and must not come after the hook, or a name an earlier pass saved.
• A pass without inputs reads HOOKED, and a scaler reads nothing else.
• No input pairs a name and a history depth twice.
• A saved name is not HOOKED or a stage name, and no texture name is source, output, luma, or chroma, whose sizes are global facts, or ends in _history<digits>, which history accessors use.
• A compute pass’s input is never TARGET, the prefix of its storage target.
• In output, <NAME> is HOOKED, an input, source, output, luma, or chroma, and a comma inside parentheses belongs to a function call.
• A kernel’s output defaults to HOOKED.size, rgba16float, and its format is rgba16float, rg16float, r16float, rgba32float, r32float, rgb10a2unorm, or rgba8unorm.
• A compute pass writes a core storage format: rgba16float, rgba32float, r32float, or rgba8unorm.
• An output’s width and height round to whole pixels within 1 and the device’s largest texture dimension, and each dispatch count rounds up within 1 and 65535; otherwise the pass is left out as output-size or workgroup-count.

Expressions

• when, output, and dispatch take expressions of numbers, identifiers, parentheses, abs, ceil, floor, max, min, round, and the operators ! - * / % + < <= > >= == != && ||; a comparison or logic operator yields 1 or 0.
• Every expression reads the global facts, source.width, source.height, output.width, output.height, luma.width, luma.height, chroma.width, chroma.height, frame.rate, display.rate, hdr.source, hdr.output, and rgb.source, all fixed at plan time. The HDR facts are 1 or 0, and the luma and chroma sizes are the planes’ (see Planes), so chroma.width < luma.width holds for subsampled chroma. frame.rate is the stream’s declared frame rate, its average frame rate, else its real frame rate when the average is missing or at least 1000, and 0 when it declares none. display.rate is the display’s refresh rate in Hz, which the presenter measures from the frame loop’s animation frames, and 0 until measured; the presenter plans again when the measurement moves more than 0.1% from the rate the plan read, while a pass’s when reads it, and a pass that decides on each draw reads presentation.refreshSeconds instead. rgb.source is 1 where the source codes R, G, and B planes, where mpv fires its RGB hook rather than LUMA and CHROMA; no route presents such video, so it is 0 on every route.
• when also reads the pass’s own parameters and constants by name, at their current values, and the presenter plans again when one that a when reads changes.
• output and dispatch also read <NAME>.width and <NAME>.height for HOOKED and each input, by its base name for a history input.
• Anything else that varies per frame belongs in the body, which branches on a uniform.

Parameters

• A parameter’s name is a unique identifier, its type is f32, i32, or u32, and its numbers are decimals with an optional sign, fraction, and exponent.
• Its default lies within its range, every number fits its type, and a pass has at most 32 parameters and constants together.
• The body reads a param as parameters.<name>. It is a uniform, so WebGPUPresenter.setPresentationParameter changes one from the next frame without rebuilding a pipeline, rounding integers and clamping to the range.
• The body reads a const as <name>, a WGSL constant at its current value, so it may size an array or fold into other constants. A change builds a new plan, whose modules compile the new value, as does a change to a param that the pass’s when reads.
• The passes of a file share each parameter’s value, so passes that declare one parameter declare it alike, of the same kind.

Texture blocks

A file may hold texture blocks beside its passes, such as a 3D LUT or a table of weights, so a host keeps storing one file per pass. A block is a header that names a texture, then its data:

//! texture: LUT
//! size: 33, 33, 33
//! format: rgba16float
//! filter: linear
//! edge: clamp

AAA8AAA8AAA8ADw...

• The body is the texels’ bytes in base64, whitespace ignored: little-endian, packed tightly in the format, x fastest, then y, then z. It fills the size in the format exactly.
• A block is not a pass; a pass reads it after bind: <NAME>, under a name no input of the pass takes, and a file of blocks alone has no pass.
• A block that does not parse fails the passes that bind it, with its own line and message.
• The engine uploads a block once, when a plan first binds it, keyed by a content identity of its size, format, and data, so a new version of the file replaces the old upload once no preset holds it; there is no count or LRU limit.
• A block beyond the device’s largest texture of its dimensions, 2D for one and two, is left out as texture-size, and a linear filter on a 32-bit float format a device cannot filter as texture-format.
• tools/convert_cube_lut.py turns a .cube 3D LUT into a file of one color pass that grades through it (see Tools).

Generated code

The engine owns every binding, and generates around each body:

• For each input, the accessors <P>_tex(coordinate: vec2f) -> vec4f, a bilinear read at a normalized coordinate, <P>_texOff(coordinate: vec2f, offsetInPixels: vec2f) -> vec4f, <P>_pt() -> vec2f, the size of one texel, <P>_size() -> vec2f, and <P>_offset() -> vec2f, the input’s offset in its own pixels (see Offsets). <P> is the input’s name, or <NAME>_history<N> for <NAME>[-N], as in RGB_history1_tex. A format that cannot be filtered, such as r32float, reads with four loads and a manual blend, so the accessor behaves the same.
• For a compute pass, TARGET_size() -> vec2u and TARGET_store(position: vec2u, value: vec4f), which runs the color steps fused after the pass before it stores.
• For each texture block it binds, <NAME>_tex(coordinate), a read through the block’s filter and edge mode at a normalized coordinate, <NAME>_size(), and <NAME>_load(texel), one texel clamped to the block. A coordinate is f32, vec2f, or vec3f, and a texel i32, vec2i, or vec3i, by the block’s dimensions.
• The parameters uniform, one 32-bit scalar per declared param in order, and each const as a WGSL constant.
• For a body that calls them without declaring them itself:
  • linearize(color: vec4f) -> vec4f and delinearize(color: vec4f) -> vec4f, the source’s own transfer function, compiled from the code of the built-in transfer decode and its inverse, in mpv’s scale of linear light, where 1.0 is referenceWhiteNits() nits.
  • referenceWhiteNits() -> f32, the nits of mpv’s linear 1.0: the configured SDR white for an SDR source, BT.2408’s HDR reference white, 203 nits, for HDR. A pass of the linear domain reads nits, so it divides by it to work in mpv’s scale.
  • rgbToPlanes(color: vec3f) -> vec3f and planesToRGB(planes: vec3f) -> vec3f: ‘source’ RGB to the normalized Y, Cb, and Cr plane values of the source’s own matrix and range, and back, each the other’s exact inverse.
• The presentation uniform, whose frame globals are sourceSize, outputSize, and inputSize as vec2f, timestampSeconds, frameIndex, historyValid, skippedFrames, fieldState, refreshSeconds, targetOffsetSeconds, and frameDurationSeconds. historyValid counts the earlier frames history holds: 0 after a reset, rising by one per new frame to the plan’s deepest history, so a pass that reads history weighs it by historyValid, as temporal-blend.wgsl does. skippedFrames counts the frames dropped since the previous new frame, which a temporal pass may treat as a soft reset. refreshSeconds, targetOffsetSeconds, and frameDurationSeconds time a draw within the display’s refreshes and the stream’s frames (see Display rate). Architecture lists when history resets.
• The vertex stage and the fragment or compute entry point that call the body’s entry point.

The engine renames each body’s module-scope declarations, so two passes can declare the same helper. The renaming is lexical, so a struct member that shares the name of a module-scope declaration is renamed where it is declared but not where it is read; give members names of their own.

Why a pass is left out

A pass that cannot run is left out of the plan, playback continues without it, and WebGPUPresenter.getPresentationPlanReport and setPresentationPreset report it as a reason code and a terse detail.

WGSL is bounds-checked and cannot reach the network or the page, so a pass can only read the frames it is given and produce pixels.

Migrate an mpv shader to a wgpup

tools/convert_mpv_shader.py converts an mpv or libplacebo user shader, the //!HOOK format of mpv’s --glsl-shaders, into a pass file (see Create a new WebGPU pass). It splits the shader into blocks as libplacebo does, translates each pass’s GLSL to WGSL with naga, binds mpv’s texture and global names to the pass API, and validates every pass with naga before it writes the file. Its report names each place where the result differs from mpv.

You need

• Python 3.10 or later.
• naga-cli 30.0.1, which translates the GLSL: cargo install naga-cli --version 30.0.1 --locked. The converter refuses any other version, since its repairs answer that version’s GLSL frontend.

Steps

1. Create wgpup/<project>/, and copy the upstream shader into wgpup/<project>/glsl/ unmodified, with a PROVENANCE.txt of its source, commit, and SHA-256, and its license in wgpup/licenses/, as wgpup/anime4k/ does.

2. Run from the engine root:

   python tools/convert_mpv_shader.py wgpup/<project>/glsl/<shader>.glsl wgpup/<project>/<shader>.wgsl
   

   --name <name> names the passes that have no DESC, which otherwise take the file’s name and their position.

3. Read the report. An error leaves the file unwritten, a warning marks a difference from mpv, and a note records an emulation, a mapping, or a repair.

4. Continue from step 3 of Create a new WebGPU pass: try the file in the sandbox, add the project to the pass tests, build, and add it to the table in Bundled passes. A converted pass may read presentation.frameIndex, sourceSize, or outputSize; add those it reads to the test’s PRESENTATION_FIELDS.

A pass a user supplies needs only step 2; load its output in the sandbox’s shader panel, or through WebGPUPresenter.setPresentationPreset.

Check the result

• The converter validates each pass with naga against the pass API’s declarations, and exits with status 1, writing nothing, on any error.

• Run its tests from the engine root:

  python -m unittest discover -s tools/test -p "*_test.py"
  

  The conversion tests skip when naga-cli 30.0.1 is not installed.

• In the sandbox, the shader panel shows each pass’s plan state, and the pipeline graph lists the passes the plan left out as reason: detail.

How passes map

• Each mpv pass becomes one pass per hook it fires on, named with the hook when it has several, and the file’s passes are sorted into mpv’s pipeline order, so a MAIN pass runs after a LUMA pass wherever each sits in the file.
• A fragment pass becomes a kernel, and a COMPUTE <bw> <bh> [<tw> <th>] pass becomes a compute pass with workgroup: tw, th and a dispatch of bw by bh blocks over its output.
• DESC names the pass, cut to 64 characters and made unique within the file.
• SAVE becomes save, renamed when the pass API reserves the name, and BIND becomes the inputs the converted code reads, and every bound saved texture, read or not, since mpv skips a pass whose bound texture no pass saved. A bound TEXTURE becomes a bind of the file’s texture block (see TEXTURE blocks).
• WIDTH, HEIGHT, and WHEN, which mpv writes in RPN, become infix expressions. when reads plan-time facts and the pass’s parameters, so texture sizes there become the sizes the earlier passes leave, over luma, chroma, source, and output, and NATIVE_CROPPED is luma’s size.
• COMPONENTS gives a saved texture a smaller format and becomes components.
• OFFSET <x> <y> becomes offset: <x>, <y>, and OFFSET ALIGN becomes offset: align, which libplacebo’s texture rects mean (see Offsets), so nnedi3’s and ravu’s doubled luma lands where the source did. A pass after the scaler, or on CHROMA_SCALED, has no grid of its own to shift, so its OFFSET is reported as offset-ignored.
• A bound texture of an earlier stage, such as PREKERNEL read at POSTKERNEL, reads the final image of the engine stage it maps to; a plane’s name reads that plane’s final image, as in mpv.

mpv fires RGB on RGB sources and LUMA and CHROMA on YUV ones, so a pass that hooks RGB beside a plane hook keeps both, the RGB pass under when: rgb.source and the plane passes under when: !rgb.source, which the report notes as hook-by-source.

Planes

• LUMA, CHROMA, and NATIVE passes run on the engine’s plane stages (see Planes), which hold the planes as mpv’s normalized code values, chroma at its own size and siting. A pass that overwrites its plane writes the plane, and the engine converts the planes with the source’s own range and matrix.
• So a chroma upscaler such as KrigBilateral or the Chroma from Luma prediction shaders runs as in mpv: its WHEN CHROMA.w LUMA.w < reads chroma.width < luma.width, it reads chroma at its native size, and its OFFSET ALIGN output is the chroma the image converts with. On the raw and frame-pool routes the planes are the decoded ones; on the external routes and in deinterlacing plans the engine derives them from the RGB image at the stream’s own subsampling.
• Dolby Vision routes leave plane passes out as planes-unavailable.
• CHROMA_SCALED, chroma at the image’s size after mpv’s chroma scaler, is not held apart by the engine, so it is read from the RGB image through the prelude’s rgbToPlanes, the source’s own matrix and range, and new chroma joins the image’s luma through planesToRGB.

Linear light

• The engine’s linear light is in nits, while mpv’s 1.0 is SDR white. A LINEAR or SIGMOID pass divides what it reads by the prelude’s referenceWhiteNits(), the configured SDR white for an SDR source and the HDR reference white for HDR, and multiplies its result back.
• A SIGMOID pass also reads through libplacebo’s default sigmoid, center 0.75 and slope 6.5, and writes through its inverse.

Names a body reads

Parameters

• A PARAM of type float, int, or uint, DYNAMIC or not, becomes a param of type f32, i32, or u32, in each pass that reads it or whose WHEN reads it; a missing bound takes its type’s range.
• An ENUM int parameter becomes an i32 parameter over its labels, which read as their numbers.
• A WHEN that reads a parameter becomes a when that reads it by name, so the engine plans again when it changes. A WIDTH or HEIGHT that reads a tunable parameter takes its default, which the report notes as parameter-inlined.
• A CONSTANT parameter the code reads becomes a const, which the engine compiles at its current value. The GLSL declares it as a global, which naga reads by name, unless the code needs a constant expression of it, such as an array size; then naga may fold it into the code, which the converter finds by translating again at another value, and a folded constant keeps its default, reported as constant-folded.
• A DEFINE parameter steers the preprocessor, so the converter translates the pass once per value of the DEFINE parameters its code reads, each pass under when: <name> == <value>, and declares them as params, so changing one picks another pass. Past 16 passes the parameters keep their defaults, reported as define-variant-limit.
• A parameter named like a WGSL reserved word gains _value.

TEXTURE blocks

• A TEXTURE becomes a texture block of the file (see Texture blocks), in one, two, or three dimensions, with its FILTER and BORDER as filter and edge; each pass that binds it gets a bind, and the block follows the passes.
• An 8-bit format stays 8-bit, a 16-bit float format holds halves, as mpv’s GPU texture does, whether the shader wrote halves or 32-bit floats, and a 32-bit float format stays 32-bit. Three components take four, with alpha 1, and a 16-bit or 32-bit unorm format takes 32-bit floats, which the report notes as texture-format-mapped.
• Integer formats and STORAGE textures do not convert.
• A texture’s name that the pass API reserves gains mpv_, as texture-renamed.

Repairs

naga 30.0.1’s GLSL frontend rejects or mistranslates a few constructs that GLSL compilers accept. The converter rewrites them without moving any line, so naga’s line numbers still name the user shader’s lines:

• Adjacent array dimensions, which naga reverses, are reversed in declarations and constructors first.
• A for-loop condition that opens with a constructor, which naga reads as a declaration, is parenthesized.
• memoryBarrierShared() and the other memory barriers are dropped, and barrier() becomes workgroupBarrier() alone, since naga adds subgroupBarrier(), which needs an extension.
• After a naga failure, a function that can end without a return returns zero there, dot() of scalars goes through overloads, a swizzle of a scalar, which GLSL 4.20 allows, reads the scalar or a vector of it, and a global const that naga cannot fold, such as one set by a ternary, becomes a macro, with every const that reads it. Each of these is reported as glsl-repaired.

Report codes

Limits

• A BUFFER block does not convert, since a pass cannot declare buffers.
• A compute pass runs within WebGPU’s default limits, 256 invocations per workgroup and 16 KiB of workgroup storage, so NVScaler, which needs 21776 bytes, does not convert as it is. A shader whose block size sets its shared memory can fit by a smaller block, as the nis project’s generator converts NVScaler (see Bundled passes).
• A converted pass runs as mpv ran it, pass for pass. A generator that fuses a network into fewer passes, as the Anime4K project’s do (see Bundled passes), can go further than the converter.

WebAssembly build

The engine uses two kinds of WebAssembly. wasm/ holds the engine’s own modules: the Dolby Vision RPU parser and the decoded audio output stage. wgpud/ is a submodule of the decoders repository, alchemyyy/webgpu-player-decoders, which holds the decoder projects, each of which builds one decoder package; its name marks them as this player’s own module type, as wgpup/, the passes repository’s submodule, marks the bundled passes. The engine never loads a decoder project’s output by itself: a host supplies packages at runtime, and the engine names no project. A checkout installs the decoder packages from a pinned release of the decoders repository rather than building them (see Pinned releases). Both the builds and the installs write into bin/, which is ignored like everything in bin/ but bin/codec_vector_assets/.

You need

• GNU Make, Git, and a POSIX shell (Git Bash on Windows). No container is involved.
• Node.js: the Makefiles read the engine’s folders from tools/constants.json.
• Emscripten 4.0.13 (revision 2659582941bef14008476903f48941909db1b196), from the emsdk that EMSDK names, which emsdk_env sets. Without EMSDK, the Makefiles use the engine’s own emsdk in bin/emsdk/, and with neither, emcc from PATH. make -C wasm emsdk installs the engine’s emsdk: a clone of emsdk at the release tag of the pinned version, replaced when the pin moves, whose installer needs Python 3. The audio output stage and the C decoder projects need it.
• rustup, for libdovi and the Rust decoder projects. Their rust-toolchain.toml files select the same Rust, 1.96.1, with the wasm32-unknown-unknown target.
• To build the decoder projects: CMake, Perl, and GNU patch, which the AV1 project’s libaom build needs, and on Windows long path support (see Set up a checkout). The engine’s own build needs none of them.

The engine’s modules

Each is served from libraries/<kit>/ and has no JavaScript glue: the engine instantiates each .wasm itself. The playback worker makes one instance of libdovi when its first parser needs it (demux/dolby-vision/WorkerWASMInstanceCache.ts), and every later parser creates its own native context in that instance. The audio decode worker instantiates audio-output-stage once, when its first decoded audio attempt starts.

Run from the engine root:

make -C wasm emsdk                # Install the pinned Emscripten into bin/emsdk, once
make -C wasm all                  # Build both modules into bin/wasm
make -C wasm check                # Rebuild both and compare them with bin/wasm
make -C wasm audio-output-stage   # Rebuild one kit
make -C wasm clean

The build sets SOURCE_DATE_EPOCH, the locale, the time zone, and the Python hash seed, ignores compiler variables from the calling environment, and refuses any Emscripten other than 4.0.13. Panics embed source paths, so the libdovi build maps the cargo home, crate, and target directories to fixed names. With the rust-src component installed, rustc rewrites the standard library’s /rustc/<commit> paths to the toolchain’s local copy, so the build maps that copy back too. make check rebuilds both modules and compares them byte for byte with the existing bin/wasm/; copy another machine’s bin/wasm/ in first to check that two machines build the same bytes.

The Dolby Vision parser crate

wasm/libdovi/ is our crate. src/lib.rs wraps the dolby_vision crate and packs each RPU into the fixed snapshot layout the engine reads (demux/dolby-vision/DolbyVisionRPUDataLayout.ts, schema 2). It has two entry points over one parser state: dovi_parser_parse reads an HEVC RPU NAL unit (type 62), and dovi_parser_parse_av1_t35 reads an AV1 ITU-T T.35 metadata message, from the country code to the end of its OBU payload. dovi_parser_set_container_profile gives the state the RPU profile the container declares, which an RPU whose header reveals none takes. The playback worker loads dovi-rpu-parser.wasm only for an owned HEVC or AV1 attempt on a Dolby Vision route; any other route removes the RPUs without parsing them.

The dolby_vision crate is not fetched from GitHub. It is copied from dovi_tool rev 38adec0 into wasm/libdovi/vendor/dolby_vision/ and built as a path dependency, because the parser follows FFmpeg’s libavcodec/dovi_rpudec.c where upstream is stricter. vendor/dolby_vision/PATCHES.md lists every deviation from upstream; update it with any change to the copy. The Recipes cover a change.

From wasm/libdovi/:

cargo test --locked
cargo clippy --locked --all-targets

The audio output stage

wasm/audio-output-stage/audio_output_stage.c holds the decoded audio output stage’s two kernels: the resampler’s source history and windowed-sinc rendering, and the lookahead limiter’s history, attack envelope, and gain. Timeline reconciliation, timestamps, output chunking, and the downmix stay in TypeScript, in audio/processing/. Each kernel keeps its history in a ring, so a push copies only its own frames, and its SIMD (simd128) vectors hold channel pairs or neighboring frames, never neighboring taps. The kernels produce the same bytes as StreamingAudioResampler and StreamingAudioLookaheadLimiter, which stay as the reference and the fallback (see Decisions). Three rules keep them exact:

• -ffp-contract=off, so no multiply and add fuse.
• The filter table is built in JavaScript and copied in, because V8’s Math.sin and Math.cos need not match a C library’s.
• The limiter’s constants from Math.pow and Math.exp arrive as numbers, and its attack length reads the engine’s Math.log10 through the module’s one import, math.log10.

The kit links with -sSTANDALONE_WASM and emcc’s own --no-entry, so it has no JavaScript glue. It is a WASI reactor: _initialize runs the constructors that set up emmalloc’s heap, before any other export. It defines emscripten_notify_memory_growth itself, so math.log10 is its only import. audio/processing/AudioOutputStageModule.ts instantiates it, refuses a module whose audio_output_stage_abi_version differs from its own, and wraps each kernel; raise the version with any change to an export’s meaning.

After a change, rebuild the kit and run the equivalence suite, which compares both implementations byte for byte, from the engine root:

make -C wasm audio-output-stage
npx vitest --watch=false test/audio/processing/AudioOutputStageModule.integration.test.ts

Pinned releases

tools/pinned-releases.json pins one release of the decoders repository and one of the passes repository, file by file:

{
  "decoders": { "repository": "alchemyyy/webgpu-player-decoders", "tag": "v0.1.0", "files": { "<name>.wasm": "<sha256>" } },
  "passes": { "repository": "alchemyyy/webgpu-player-passes", "tag": "v0.1.0", "files": { "<name>.wgsl": "<sha256>" } }
}

Only the .wasm files of a decoders release and the .wgsl files of a passes release install; their license texts, source archives, and SHA256SUMS do not. npm run pinned-releases (scripts/pinned-releases.mjs) installs them into bin/wgpud/ and bin/wgpup/, and ./build.sh and ./sandbox.sh run it first:

• A file in place with its pinned SHA-256 stays.
• A missing file, or one the script installed before and that is unchanged since, downloads from https://github.com/<repository>/releases/download/<tag>/<file>, must match its pinned SHA-256, and replaces the old file whole. The script records what it installed, with each file’s SHA-256, in bin/installed-releases.json.
• Any other file there, such as a local build from npm run decoders or npm run passes, stays, and the script names it; --force installs the pinned files over local builds too.
• A file an earlier pin installed that the pin no longer names is removed, unless it changed since; with --force it is removed regardless.

It needs the network only for the files it installs. tools/constants.json names the two files as pinnedReleasesFile and installedReleasesFile, which tools/constants.mjs exports as PINNED_RELEASES_FILE and INSTALLED_RELEASES_FILE.

Release a repository

Each repository releases through its .github/workflows/release.yml, run by hand on main:

1. Commit and push the change on main of the decoders or passes repository.

2. In that repository, open Actions > Release > Run workflow, and give version three numbers, such as 0.2.0, which becomes the tag v0.2.0. Without a version the run is a dry run, which keeps the release files as the run’s artifact.

3. In the engine, pin the release and install it:

   node scripts/pinned-releases.mjs --pin decoders v0.2.0
   

   --pin <decoders|passes> <tag> reads that release’s SHA256SUMS, pins its .wasm or .wgsl files with their SHA-256 in tools/pinned-releases.json, then installs.

4. Move the submodule to the commit the release is tagged at, and commit the gitlink and tools/pinned-releases.json together in one engine commit, so the submodule and its pin always move together.

The decoders workflow checks out the engine at its engine-ref input, main by default, puts the decoders repository at the engine’s wgpud/, installs the pinned Emscripten with make -C wasm emsdk and Rust, and runs the engine’s own steps: make -C wgpud sources, npm run decoders, which loads every package with the engine’s loader, and make -C wgpud source-archives. Its release files are every <project>.wasm, the license texts of licenses/, the three source archives, which are the corresponding source of the LGPL packages, and SHA256SUMS. The passes workflow needs no engine checkout: it collects every WGSL file at the top level of every project folder, flat, and fails on two files of one name, as npm run passes does; its release files are those, the license texts of licenses/, and SHA256SUMS.

After an install, npm run decoders -- --check compares the submodule’s build with the pinned release in bin/wgpud/, so it checks that the submodule reproduces the release.

Decoder projects

A decoder project is a folder of wgpud/ with a decoder.json. It is a minimal project that builds one package and runs nowhere on its own.

decoder.json names the project’s decoder source, the WGSL file or audio manifest the package carries, under decoder. Each project’s header declares the project’s name as its package id, such as //! id: hevc, so a rebuilt package replaces the one a host stored before. A WGSL decoder may name a list of files instead, which the build joins in order into the one source the package carries: each later file follows a // Decoder source: <file> line, so a compile message leads back to its file, and only the first file may hold header lines. A video project lists its self-test vectors under vectors, each with its output format, the layout key of its root folder in tools/constants.json, its path there, optionally the FFmpeg decoderOptions of the reference decode, and optionally codecPrivate, the layout of the codec private data the vector passes the decoder. That is the stream’s extradata, or with bitmap-info-header a 40-byte BITMAPINFOHEADER of the stream’s size and FourCC before it, as the engine passes the mpeg4 package (see Codec private data). A project with a Makefile builds its module with make module, which writes wgpud/build/<project>/module.wasm; a project without one packs an empty module, as a decoder without a parser does. Each Makefile includes wgpud/rules.mk, which pins the toolchain, FFmpeg, dcadec, and libaom, and holds the rules to build an FFmpeg audio module, a libdcadec module, libaom, and a Rust module. wgpud/include/audio_decoder_interface.h declares the audio decoder interface for C modules.

Run from the engine root:

make -C wgpud sources                     # Fetch the pinned FFmpeg, dcadec, and libaom, once, for the audio and AV1 projects
npm run decoders                          # Build every project into bin/wgpud/<project>.wasm
npm run decoders -- hevc vc1              # Build some projects
npm run decoders -- --check               # Rebuild every project into a scratch folder and compare it with bin/wgpud
make -C wgpud source-archives             # Archive the corresponding source of the LGPL projects
make -C wgpud clean                       # Remove the decoder projects' build trees
npm run pinned-releases -- --force        # Replace local builds with the pinned release again

scripts/decoder-packages.mjs iterates over the projects without naming any. It builds every project’s module at once, since each build spends most of its time in one serial chain such as a configure script’s compiler probes, and prefixes each output line with the project’s name. Then, in project order, it packs each module, the decoder source, and the project’s vectors file with tools/decoder-package-format.mjs, and loads the result with the engine’s own loader, as any host would, before it writes the package. A failed build or load check fails the run after the other packages are written. tools/pack-decoder-package.mjs packs one package by hand from the same parts.

How the build works

• Pinned sources. FFmpeg, dcadec, and libaom are git submodules in the engine’s vendor/, pinned to exact commits. They are shallow and marked update = none, so a recursive clone of the engine, or of a host that embeds it, skips them until make -C wgpud sources fetches them. It fetches only each recorded commit, at depth 1, because git submodule update --depth 1 would first clone the tip of the default branch.
• Fresh source trees. An FFmpeg project extracts a fresh copy of the pinned tree into wgpud/build/<project>/source/ with git archive, then configures and builds it there with only that project’s decoders and FFmpeg’s portable C. Building outside a git checkout keeps FFmpeg’s embedded version string and source paths stable. The AV1 project extracts and builds libaom the same way, after applying its patches (see The AV1 parser).
• Configuration checks. The build fails unless configure enabled exactly the whitelisted decoders and kept the LGPL scope: no GPL, nonfree, or version 3 components.
• A fixed environment, as for the engine’s modules, and Rust panic paths mapped to fixed names.
• Audio modules link as standalone reactors with -sIMPORTED_MEMORY, so each imports its memory and exports version 1 of the audio decoder interface.

Self-test vectors

scripts/codec_vector_assets/generate_decoder_package_vectors.py reads the vectors each decoder.json lists and writes wgpud/<project>/<project>.vectors, the set the package carries in its vectors section (see Self-test vectors). A project whose streams no script elsewhere writes keeps them in its own vectors/, from a generator of its own: generate_mpeg2_decoder_vectors.py, generate_jpeg2000_decoder_vectors.py, generate_av1_decoder_vectors.py, and generate_mpeg4_decoder_vectors.py, and for the wma project, whose audio package carries no vectors section, generate_wma_decoder_vectors.py, which writes its media and FFmpeg’s known answers for its tests. The files are committed, byte for byte (the decoders repository’s .gitattributes marks /*/*.vectors and /*/vectors/** -text), and --check fails unless they match what the generators write. scripts/generate_all_codec_vector_assets.py runs them with the other generators, a project’s own media first.

The HEVC parser

wgpud/hevc/parser/ is our crate: the CPU half of the HEVC package, whose WGSL back end is wgpud/hevc/hevc.wgsl. It parses each packet into one GPU record: the picture’s parameters and slices, and per CTB its SAO, its prediction units with their motion, its transform jobs and coefficients, its deblocking edges, and its PCM samples. The WGSL expands the record and reconstructs and filters the picture on the GPU (see Decisions). It implements version 2 of the engine’s parser interface: plan, parse, and finish, with decode running all three on one instance. Every packet is one picture, so plan_next and decode_next report no more pictures, and a picture reports no visible size, repeat counts, or declared maximum of its own. A 4:0:0 picture decodes into the 4:2:0 format of its depth with neutral chroma, so the header declares monochrome: yes.

hevc.wgsl declares the record’s layout and the error codes once, as const RECORD_<NAME>: u32 and const ERROR_<NAME>: u32 lines. build.rs reads them into Rust constants and fails unless the header’s //! error: lines name exactly the error codes the WGSL declares, so the module and the WGSL of one package always agree. A failed check reports one of the decoder’s error codes and its failing value, as a failure on the GPU does.

After a change, from the engine root:

npm run decoders -- hevc
cd wgpud/hevc/parser && cargo test --locked
npx vitest --watch=false wgpud/hevc

The JPEG 2000 parser

wgpud/jpeg2000/parser/ is our crate: the CPU half of the JPEG 2000 package, whose WGSL back end is wgpud/jpeg2000/jpeg2000.wgsl. It reads a JP2 file, a jp2c box, or a bare codestream, picks the pixel format as FFmpeg does, runs tier-2, and writes one GPU record per picture: the code blocks with their bytes, terminations, and dequantization, the wavelet tables, and the output planes. One parse unit is one tile, so the parse workers split a picture by its tiles. The WGSL decodes each code block in one 32-lane workgroup, runs the inverse wavelet, and writes the output planes. An XYZ picture, which FFmpeg picks for the Digital Cinema profiles, is laid out as the 12-bit X’, Y’, and Z’ codes FFmpeg’s converter reads, at 4:4:4, and the record’s output transform makes the output stage convert them as libswscale’s xyz12Torgb48 does, its gamma tables and fixed-point matrix included, then write the R’G’B’ as limited-range BT.709 Y’CbCr in I444P12. Its planes therefore equal no FFmpeg decode, so decoder.json lists no XYZ vector; FFmpeg’s rgb48le decode of the Matroska stream in vectors/ is the conversion’s reference. As for HEVC, the WGSL declares the record’s layout and the error codes once, and build.rs generates the Rust constants from them and fails unless the two agree, and every packet is one picture of version 2 of the parser interface. The parser bounds what a crafted codestream can make it allocate and parse, and fails such a stream as resourceLimit. It fails greyscale pictures as colourSpace, so the header declares no monochrome, and eligibility never offers the package a 4:0:0 stream.

After a change, from the engine root:

npm run decoders -- jpeg2000
cd wgpud/jpeg2000/parser && cargo test --locked
npx vitest --watch=false wgpud/jpeg2000

The MPEG-2 parser

wgpud/mpeg2/parser/ is our crate: the CPU half of the MPEG-2 package, whose WGSL wgpud/mpeg2/mpeg2.wgsl decodes everything, its headers included. The parser reads only what the engine needs before a picture decodes: the size and chroma format of each picture’s sequence. It reads them as the WGSL’s header stage does, from the sequence headers and sequence extensions before a packet’s first picture start code, and ignores a damaged sequence header where the WGSL and FFmpeg ignore it. configure reads the codec private data the same way and reports its sequence’s size as the largest picture. Every packet is one picture of version 2 of the parser interface, without parse units, and its record is the packet, byte for byte, so the WGSL decodes, conceals, and reports fields as it did without a parser. A picture reports its sequence’s size as its visible size and its largest picture, and its sequence’s chroma format at 8 bits as its format, so a larger sequence, or one of another chroma format, moves the engine to a session of its own (see Stream changes). A sequence its pool holds, such as a smaller one, decodes in the session it is in: the WGSL drops its references and starts the new geometry, as FFmpeg reinitializes, and each frame presents at its own picture’s visible size. The parser decides no output events, so frames present in timestamp order as a decoder without a parser’s do, and reports no picture fields, so the WGSL’s own report stands. MPEG-1 has no sequence extension, so its sequences are 4:2:0. The WGSL declares the error codes, and build.rs generates the Rust constants from them and fails unless the header’s //! error: lines name exactly those; the parser’s only failure is recordInvalid, a call the engine never makes.

After a change, from the engine root:

npm run decoders -- mpeg2
cd wgpud/mpeg2/parser && cargo test --locked
npx vitest --watch=false wgpud/mpeg2

The VC-1 parser

wgpud/vc1/parser/ is our crate: the CPU half of the VC-1 package, whose WGSL back end is wgpud/vc1/vc1.wgsl. It parses each packet’s picture as FFmpeg’s vc1 decoder does: the segments and their escapes, entry points, the frame, field, and slice headers, bitplanes, the intensity compensation lookups, and the macroblock layer, with motion vector prediction, coefficients, and intra DC and AC prediction, scan, and dequantization. configure takes the codec private data the engine passes for vc1 and wmv3 (see Codec private data) and reports the Advanced profile’s MAX_CODED_WIDTH and MAX_CODED_HEIGHT, or the Simple and Main profiles’ coded size, as the largest picture. One parse unit is one slice or one field, so the parse workers split a picture by its slices and fields; a picture without slices parses on one worker. Each picture’s record holds its header, each field’s parameter block, the lookup sets those read, and per macroblock its flags, transform types, motion compensation operations, and dequantized coefficients at their raster positions. expandRecord writes them into the buffers the reconstruction, overlap smoothing, and loop filter passes read, one workgroup per macroblock. An IntraX8 picture decodes entirely in the parser, its prediction, transform, and loop filter included, and its record carries the picture’s samples, which copyPicture writes. A B picture’s direct mode reads the next anchor’s motion, which only that anchor’s parse yields, so its plan reports blocked until every earlier anchor finished, then carries that motion. Output events follow FFmpeg’s: a B or BI picture, and any picture of a low-delay stream, presents at once, and an anchor presents the anchor before it, so the header declares reorder: 1. An entry point that changes the coded size drops the anchor waiting to present and every reference, as FFmpeg’s decoder does when it reinitializes, and the next record tells the WGSL to release its anchors. A picture reports its coded size as its visible size, and parser_picture_fields reports its fields from FCM, TFF, and RFF. As for HEVC, the WGSL declares the record’s layout and the error codes once, and build.rs generates the Rust constants from them and fails unless the two agree, and every packet is one picture of version 2 of the parser interface. The damage of a packet, pictureHeader, bitplane, startCodes, macroblockLayer, bitsExhausted, and pictureTooShort (a field picture of one macroblock row, FFmpeg’s “image too short”), resyncs. The configuration, the sequence header, an unsupported feature, an invalid record, and pictureSize, a size beyond the pool or the package’s 16384 macroblocks, end the attempt, since the pictures after it would fail the same way and a resync would hold the last good frame for good.

After a change, from the engine root:

npm run decoders -- vc1
cd wgpud/vc1/parser && cargo test --locked
npx vitest --watch=false wgpud/vc1

The MPEG-4 parser

wgpud/mpeg4/parser/ is our crate: the CPU half of the package that decodes MPEG-4 Part 2, H.263, and MS-MPEG4 v1 to v3, whose WGSL back end is wgpud/mpeg4/mpeg4.wgsl. It is a port of FFmpeg’s decoders: mpeg4videodec.c for MPEG-4 Part 2 (mpeg4/), ituh263dec.c for H.263 and the H.263+ annexes FFmpeg decodes (h263.rs), msmpeg4dec.c for MS-MPEG4 (msmpeg4.rs), the slice loop and resync of h263dec.c (h263dec.rs), the macroblock state and dequantization of mpegvideo.c (mpv.rs), and the operation geometry of mpegvideo_motion.c (motion.rs). configure reads the BITMAPINFOHEADER the engine passes for every codec of the package (see Codec private data), picks the codec from its FourCC as FFmpeg’s riff.c classes it, reads an MPEG-4 Part 2 track’s VOL from the extradata after it, and reports the coded size as the largest picture. FFmpeg’s encoder detection runs on the FourCC and the user data as in FFmpeg, and sets the same bug workarounds and IDCT, the Xvid IDCT included. One parse unit is one picture, so the parse workers decode pictures at once, each from a fresh copy of the stream’s tables, as FFmpeg’s decoder holds no state between pictures besides the headers and the anchors. A DivX packed packet’s second VOP, which the fork’s AVI demuxer unpacks but a Matroska track can still carry, plans as a second picture of the same packet. FFmpeg’s decoder keeps that VOP and decodes it with the next packet instead, so in Matroska the two pictures share one packet’s time, and a stream’s last such VOP, which FFmpeg never decodes, presents. Each picture’s record holds per macroblock its flags, the H.263 loop filter’s quantizers, the motion operations FFmpeg would run (half-pel, quarter-pel, one-point and general global motion compensation, and overlapped block motion compensation, each with its source position, fraction, and rounding), and its dequantized IDCT blocks, dense or as sparse entries. The WGSL runs a picture’s operations in order, then the simple or Xvid IDCT, then the H.263 loop filter over all horizontal and then all vertical edges. A B-VOP’s direct mode reads the next anchor’s motion, which only that anchor’s parse yields, so its plan reports blocked until every earlier anchor finished, then carries that motion. Output events follow FFmpeg’s: a B picture, and any picture of a low-delay stream, presents at once, and an anchor presents the anchor before it, so the header declares reorder: 1. An H.263 picture of another size starts new anchors, as FFmpeg reinitializes, and reports its size as its visible size and the largest so far; parser_picture_fields reports an MPEG-4 Part 2 VOP’s interlacing and top field first. As for HEVC, the WGSL declares the record’s layout and the error codes once, and build.rs generates the Rust constants from them and fails unless the two agree, and every packet is one picture of version 2 of the parser interface. The damage of a packet, sequenceHeader, pictureHeader, macroblockLayer, and bitsExhausted, resyncs; the configuration, an unsupported feature, pictureSize, and an invalid record end the attempt. The parser’s reference is FFmpeg’s bit-exact, single-threaded decode (see decisions).

After a change, from the engine root:

npm run decoders -- mpeg4
cd wgpud/mpeg4/parser && cargo test --locked
npx vitest --watch=false wgpud/mpeg4

The AV1 parser

wgpud/av1/ decodes AV1 with libaom’s decoder as its parser and WGSL for everything after the syntax. libaom v3.15.2 is the git submodule vendor/aom, pinned in wgpud/rules.mk (AOM_COMMIT), shallow and marked update = none, as FFmpeg and dcadec are. The build extracts the pinned tree with git archive, applies the patches in wgpud/av1/parser/patches/ in order with patch --fuzz=0, and builds the decoder alone with libaom’s CMake and Emscripten’s toolchain file: generic C, one thread, and WebAssembly exception handling for libaom’s longjmp error path (-sSUPPORT_LONGJMP=wasm), so the module needs no JavaScript glue. The patches turn the decoder into a parser:

• 0001-parse-only-decoding.patch parses every tile with libaom’s parse-only flag, so the decoder reads modes, motion, palettes, transform sizes, coefficients, CDEF indices, and restoration units, and runs no prediction, reconstruction, or filter; hooks hand each transform block, block, frame, and missing reference to the bridge.
• 0002-intra-edge-parameters.patch exports the edge sample counts and the edge filter type that intra prediction derives.
• 0003-cfl-store-extent.patch keeps the extent of the chroma-from-luma buffer the decoder would have stored, and exports CfL’s alpha.
• 0004-filter-level.patch exports a block’s deblocking level.

libaom’s frame buffers point at an address no module memory maps, so the parser holds no samples, and a stray sample access traps. libaom refuses a frame buffer past an eighth of AOM_MAX_ALLOCABLE_MEMORY, 2 GiB on 32-bit targets, which a level 6 4:4:4 frame with a scaled reference’s borders passes, so the build raises it to 4 GiB less 64 KiB; the 1 GiB instance memory still bounds every allocation that maps memory.

parser/av1_parser.c implements version 2 of the parser interface. plan parses a temporal unit’s frames, one picture per frame, hidden frames and show_existing_frame included, and plan_next hands out the rest. The unit’s last shown frame presents and the others drop, as libaom and FFmpeg’s libdav1d output one frame per temporal unit at operating point 0. A picture reports its visible size, the upscaled width and the frame height, its intra wave pairs, its output format, and its sequence header’s maximum frame size, and configure reports the configured sequence header’s maximum. So a picture of a new sequence that changes the bit depth or the chroma format, or raises the maximum past the pool, moves the engine to a new session, from that sequence’s key frame on. After a start or a reset, temporal units before the first key frame make no picture. parser/av1_record.c writes each frame’s GPU record: the blocks with their motion, warps, transform partitions, and palettes; each superblock’s intra jobs per plane in decode order, with their edges; transform jobs with libaom’s dequantized coefficients, dense or sparse, those with a 64-point side last; and the restoration units, CDEF indices, deblocking levels, references with their scale factors, global motion, and film grain. av1.wgsl declares the record’s layout and the error codes once, as const RECORD_<NAME>: u32 and const ERROR_<NAME>: u32 lines, and parser/record_layout.mjs turns them into the bridge’s C header, and fails unless the //! error: lines name exactly the error codes the WGSL declares and the //! packet: line is the bound the RECORD_PACKET_ constants describe. That bound is a record’s worst case for its pool, between 64 MiB and the engine’s 512 MiB: per 4x4 luma unit the words of a 4x4 block and its side data, and per 4x4 unit of each plane an intra job, a transform job, and 16 coefficients of the bit depth plus 8 bits, libaom’s dequantized range. The parser judges each record against the bound of its frame’s upscaled size rounded up to 64, which never passes its session’s, and fails it as recordSize beyond. The package takes pictures up to AV1’s level 6.x, 16384x8704 and 35,651,584 samples, 8K included; its record fields hold mode info coordinates up to 16384 samples.

decoder.json lists av1.wgsl first, with the header, which declares id: av1, the record layout, the frame store, and the entry points, then the four family files: wgsl/transform.wgsl (inverse transforms, film grain, and the output copy), wgsl/intra.wgsl, wgsl/inter.wgsl, and wgsl/filters.wgsl (deblocking, CDEF, super-resolution, and loop restoration). Each record runs validation, inter prediction per 16x16 tile, the transforms in two passes, the transforms with a 64-point side in a kernel of their own, an intra wavefront over superblocks for intra, inter-intra, and IntraBC blocks, deblocking, CDEF, super-resolution, loop restoration, and the output copy with film grain. References stay in the package’s own frame store, ten 16-bit frame buffers the parser assigns, so the header declares references: 0, and only shown frames reach the frame pool. The store spans two buffers, frames with frame buffers 0 to 4 and framesHigh with the rest, so a device whose largest binding holds half of it decodes: 1.07 GB at 8192x4352 4:2:0 becomes two of 535 MB, and the 2.14 GB of 4:4:4 two of 1.07 GB. A 4:0:0 stream decodes into the 4:2:0 format of its depth with neutral chroma, so the header declares monochrome: yes.

After a change, from the engine root:

npm run decoders -- av1      # About 3 minutes
npx vitest --watch=false wgpud/av1

The WMA bridge

wgpud/wma/ffmpeg_wma_bridge.c puts FFmpeg’s five WMA decoders behind the audio decoder interface. Every one of them needs the track’s WAVEFORMATEX, so a decoder opens only when audio_decoder_configure reads it, and refuses packets as unconfigured until then. It reads the header as FFmpeg’s ff_get_wav_header does: the sample rate, channels, bit rate, block alignment, bits per sample, a WAVE_FORMAT_EXTENSIBLE header’s channel mask when it describes that many channels, and the bytes after the header as extradata. A header shorter than 18 bytes or than its cbSize, an extensible one without its 22 bytes, or one without channels or a sample rate fails as waveFormat; a format tag, or an extensible SubFormat’s, of another codec as formatTag; and a header the decoder refuses as open. Each configure after the first opens a fresh decoder. The decoders hold frames back until the end of the stream, WMA 1 and 2 their last overlap and WMA Lossless the frames it buffered, so the module exports the interface’s optional audio_decoder_drain, which its Makefile names as the second argument of link_ffmpeg_audio_module. A drain before a configure fails as unconfigured, a second drain outputs nothing, and audio_decoder_clear starts a new stream after one. Blocks are planar float: WMA Lossless’s 16-bit and 24-bit integers are scaled by the exact power of two of full scale, and WMA Voice’s packed mono float is copied. A layout without a channel mask reports the mask of FFmpeg’s default layout of its channel count. FFmpeg’s C decode, -cpuflags 0, is the reference, and the module decodes every vector to the same bytes, as WMA decoder vectors describes.

After a change, from the engine root:

npm run decoders -- wma
npx vitest --watch=false wgpud/wma

The TrueHD bridge

wgpud/truehd/ffmpeg_truehd_bridge.c puts FFmpeg’s MLP and TrueHD decoders behind the audio decoder interface, and adds Dolby Atmos object rendering for TrueHD. FFmpeg decodes every packet, and its decode is the output unless the render target needs objects, as the renderChannelMask and renderBinaural options of the audio decoder interface set it: a stream FFmpeg reports as Dolby TrueHD + Dolby Atmos, with binaural rendering on, or with a target mask that has a speaker FFmpeg’s output layout lacks. FFmpeg stops at the 8-channel presentation: it decodes at most three substreams, holds 8 channels, and refuses the 0x31EC restart sync word of the fourth substream, which carries the 16-channel presentation. So MLP, TrueHD without Atmos, and an Atmos stream at a target its 7.1 or 5.1 presentation covers, such as 5.1, 7.1, or stereo without binaural, output FFmpeg’s decode unchanged, and report objectCount 0.

When the target needs objects, three sources of ours take over, packet by packet:

• truehd_presentation.c decodes one presentation of a TrueHD stream, here the 16-channel one, to 24-bit samples in the presentation’s channel order. It is a port of FFmpeg’s mlpdec.c, mlpdsp.c, mlp.c, and mlp_parse.c (major sync, restart headers, decoding parameters, prediction filters, Huffman residuals, primitive matrices and their noise, output shifts, and channel assignment), generalized from 8 channels and 3 decoded substreams to 16 channels and 4. It adds what FFmpeg lacks: the 0x31EC substream’s matrix syntax (a coefficient mask and shift per matrix, bypassed LSBs of up to 3 bits, and coefficient deltas that interpolate across each access unit), the substreams each presentation is built from (the 16-channel one is substream 3 together with the lower substreams that extended_substream_info names), the extra channel meaning of the major sync, and the extra data after the substreams, whose Evolution frame carries object audio metadata as payload 11 in the EMDF payload syntax. Those layouts are documented only by the truehd crate of truehdd (Apache-2.0), which we read to learn them; the code is our own, and the file’s header cites both. It also checks each restart header’s lossless check against the samples it decoded.
• truehd_objects.c hands the presentation to the renderer: its channels are the metadata’s objects in payload order, beds first, and a presentation without metadata is the channel bed its major sync’s 16-channel channel assignment names (L R, C, LFE, Ls Rs, Lb Rb, Tfl Tfr, Tsl Tsr, Tbl Tbr, Lw Rw, LFE2). Each access unit’s payloads are parsed by atmos_parse_oamd and queued at their Evolution sample offset before the access unit renders.
• The renderer in wgpud/atmos/ renders the bed and the dynamic objects into the target, or binaural stereo.

A packet renders only when every access unit in it decoded and rendered; otherwise it outputs FFmpeg’s decode, so rendering starts at the first packet after a major sync and the first object metadata, and a switch back to a covered target returns to FFmpeg’s decode at once. A packet whose access units could hold more than 16,384 frames, the engine’s bound on a block, plays FFmpeg’s decode too. Any packet the object decoder does not read, because FFmpeg dropped it or the target needed no objects, resets that decoder, so it resumes at the next major sync rather than with stale filter and matrix state. A rendered packet is one block at the packet’s timestamp, with the target’s mask and objectCount, the scene’s dynamic objects; FFmpeg’s frames of that packet are discarded. Binaural rendering delays its output by 128 frames, so the module exports audio_decoder_drain, which renders that tail after the last packet; a stock decode drains nothing. audio_decoder_clear resets both decoders and the renderer.

FFmpeg also refuses a TrueHD stream whose one substream carries more than two channels, such as 5.1 coded without a separate 2-channel substream: its read_major_sync names the channels of a single substream only for a stereo or mono stream, so every restart header fails its channel assignment (Assignment of matrix channel 0 to invalid output channel -22) and no access unit decodes, which also leaves the server unable to transcode the track. truehd_substream.c decodes such a stream with truehd_presentation.c, whose presentation 0 is that substream, and lays its channels out as FFmpeg lays out a stream it decodes: the major sync’s 8- or 6-channel arrangement with the decoded channel count names them in TrueHD order, and the block takes them in WAVE order under that arrangement’s WAVE mask, or in TrueHD order under mask 0 when a channel has no WAVE name. Each packet’s first major sync decides which decode plays until the next one, and the decode that takes over starts at that major sync, so FFmpeg never sees such a stream and a stream that changes back plays FFmpeg’s decode again. A packet whose access units do not all decode is dropped, and its block is one block at the packet’s timestamp, with bitsPerSample 24 and the pcmFingerprint of FFmpeg’s packing.

truehd_presentation.c decodes every presentation of the eight TrueHD streams among the truehd crate’s test assets and of the engine’s three TrueHD vectors bit for bit as truehdd (commit 77ddc52) does, the three 16-channel presentations included, and presentations 0 to 2 bit for bit as FFmpeg does wherever FFmpeg’s downmix option can select them, with every lossless check passing. To repeat that, build truehdd natively, decode with truehdd decode --presentation <n>, and compare its 24-bit samples with a native build of truehd_presentation.c; neither the oracle nor the harness enters the repository. The wgpud/truehd/vectors/ streams copied from the truehd crate’s test assets carry that crate’s Apache-2.0 license and PROVENANCE.txt. single-substream-5.1.thd, ours, is 5.1 in one substream, which FFmpeg’s TrueHD encoder writes only after the three changes PROVENANCE.txt gives; the package decodes it to the encoded samples exactly, which its own lossless checks confirm, and stock FFmpeg refuses every access unit of it.

After a change, from the engine root:

npm run decoders -- truehd
npx vitest --watch=false wgpud/truehd

The E-AC-3 bridge

wgpud/eac3/ffmpeg_eac3_bridge.c puts FFmpeg’s AC-3 and E-AC-3 decoders behind the audio decoder interface, and adds Dolby Atmos object rendering for E-AC-3 with joint object coding (JOC, ETSI TS 103 420). FFmpeg decodes every packet, and its decode is the output unless the render target needs objects: binaural rendering on, or a target mask with a speaker the decoded core lacks, such as 7.1 or any height layout from a 5.1 core. A stream without JOC, AC-3, and a JOC stream at a target its core covers, such as 5.1 or stereo without binaural, output FFmpeg’s decode unchanged and report objectCount 0.

When the target needs objects, our sources take over, frame by frame, on FFmpeg’s decoded channels:

• eac3_emdf.c finds the EMDF container (ETSI TS 102 366 Annex H) in the access unit’s last syncframe, as TS 103 420 clause 8.2 places it, and copies out its OAMD (payload 11) and JOC (payload 14) payloads. The container is in the frame-end auxiliary data, which auxdatal locates, or in an audio block’s skip field, as Dolby’s encoder writes it. Reaching a skip field exactly takes the audio block syntax up to it, so the search takes an EMDF syncword whose preceding ten bits read as a set skiple and a skipl that holds the container, and whose container parses completely within it. FFmpeg’s avpriv_ac3_parse_header splits a packet into access units as FFmpeg’s decoder takes them.
• eac3_joc.c parses the JOC payload (clause 6.2) and decodes it: the Huffman trees of Annex A.1, differential decoding, dequantization, the parameter band mapping of table 54, interpolation between data points, and the upmix of the 5 or 7 downmix channels to up to 16 objects in the filter bank domain.
• eac3_qmf.c is the 64-band complex filter bank of clause 7, computed through a 128-point FFT; analysis and synthesis delay the objects by 577 samples. eac3_joc_tables.c holds the Huffman trees and the 640-coefficient window from the specification’s companion archive ts_103420v010201p0.zip.
• The renderer in wgpud/atmos/ renders the objects with the OAMD: each LFE bed object takes the core’s LFE, and the other scene objects, beds then dynamic objects, take the JOC objects in order.

Object blocks keep their packets’ timestamps exactly: the module holds each block until the frames after it fill the filter bank delay and the binaural latency, so the first packet after a start, a clear, or a switch from FFmpeg’s decode outputs no block, and audio_decoder_drain outputs the last one. A switch back to a covered target outputs the held block and FFmpeg’s block of the same packet, the later frames of a packet keep their access units’ timestamps, and a change of speaker layout or binaural rendering finishes the held blocks in the old target and restarts the pipeline. Objects start only at a frame whose OAMD scene the renderer takes; a scene it refuses, such as one with intermediate spatial format objects, plays FFmpeg’s decode. A damaged JOC payload keeps the last matrices, and a damaged OAMD payload the objects’ properties. objectCount is the scene’s dynamic objects that have a JOC object.

The JOC decoder rebuilds the same objects as the JOC decoder of OpenJOC (Apache-2.0, commit e3e330a), run natively on the same FFmpeg downmix and payloads: within 6e-8 of full scale (129 dB or more per object) on Dolby’s sample_eac3joc.ec3 from androidx/media, and within 4e-10 on OpenJOC’s synthetic fixtures; both extract the same payload bytes from all 64 access units of the Dolby stream. Neither the oracle nor the temporary harness enters the repository. The wgpud/eac3/vectors/ media, and how to repeat their generation, are in E-AC-3 Atmos vectors.

After a change, from the engine root:

npm run decoders -- eac3
npx vitest --watch=false wgpud/eac3

The object audio sources

wgpud/atmos/ holds our own object audio code, which the eac3 and truehd modules compile with their bridges; it has no decoder.json, so it is no project and builds no package. atmos.h is the interface those bridges code against: atmos_parse_oamd decodes one object audio metadata (OAMD) payload, and an AtmosRenderer renders a scene’s object planes with that metadata into a target layout. A package’s Makefile passes $(ATMOS_MODULE_SOURCES) from wgpud/rules.mk, the four sources and their include flag, as the third argument of link_ffmpeg_audio_module, whose first argument lists the project’s own sources. The code depends on no other Atmos library; ETSI TS 103 420 V1.2.1 and Recommendations ITU-R BS.2051 and BS.2127 are its references.

• atmos_oamd.c decodes an object_audio_metadata_payload as TS 103 420 clause 5 defines it: the program assignment, whose bed objects come first and are named by FFmpeg’s channel bits (the LFE of a dynamic-object program is a bed object), then ISF objects, then dynamic objects; and the object element’s updates, each with its start (sample_offset plus 32 times block_offset_factor) and ramp duration, and per object its activity, gain, priority, room-anchored position (differential coding, extended precision, and a position outside the room projected onto its walls), zones, elevation, size, screen factors, and channel lock. Other elements, such as trim, are skipped by their declared size. A payload holds at most 32 objects and 8 updates; every read is bounded, and a truncated payload, an element that overruns its size, a version other than 0, counts that do not add up, or a reserved ISF index returns an error code. Where the clause’s pseudo-code and its tables disagree on which bit of a flag array is which, which happens only for obj_render_info[], the tables win: element [k] of an array is the k-th bit the payload carries, as real streams confirm for content_description[] and the bed assignments.
• atmos_renderer.c renders to every layout the engine qualifies: stereo, 5.1 (back or side), 7.1, 5.1.2, 5.1.4, 7.1.2, and 7.1.4, in WAVE channel order. Speakers stand on the room’s walls as TS 103 420 Table B.10 places them, a 5.1 layout’s surround pair in the back corners. The panner is separable and room-centric: an object pans between the floor and the heights, then between the rows of speakers in front of and behind it, then between the speakers left and right of it in each row, each step with the sine and cosine law, so its gains’ squares sum to 1 and an object at a speaker plays from that speaker alone. An object with a size averages that power over a 3 by 3 by 3 grid across its extent; zone constraints and elevation take the excluded speakers (TS 103 420 Annex A.2) out of the panner, falling back to all of them when none remain; channel lock plays an object from its nearest speaker within 0.4 in ADM units. A bed channel plays on its own speaker, or folds as the engine folds layouts: a side or back channel onto the other of its side’s pair, a top front or top back channel onto the floor channel under it (folding on from there), a wide channel onto its side’s front, and LFE2 onto the LFE; a top side channel, which the engine’s layouts lack, splits over its side’s top front and top back at sqrt(1/2) each, or plays from the top front of a layout with only that pair, or else from the side under it. Every bed channel that lands whole on a speaker, its own included, takes 1/sqrt(n) of the n channels landing there, as the engine’s layout routing does, so a 5.1 surround with Ls, Lb, and Tsl takes each at sqrt(1/3) and one alone keeps unit gain; a split channel takes sqrt(1/2) on each of its two. Stereo takes that fold into 7.1 and mixes it with the engine’s default Lo/Ro matrix: fronts at 1, center, sides, and backs at sqrt(1/2), the LFE dropped. ISF objects are parsed but not rendered: a scene with them fails as unsupported. Updates queue at their start sample, at most 32 at once; an update’s properties ramp linearly from the ones the object has when it starts, gain vectors computed every 32 samples and the gains moving linearly in between, so they reach the update’s values at its start plus its ramp duration whatever the render calls’ lengths. A target changes between render calls: objects keep their properties and ramps and move to the new layout’s gains at once. Nothing allocates after atmos_renderer_create.
• atmos_binaural.c renders binaural stereo at 48 kHz: the scene renders to a virtual 7.1.4 layout, and each of its 11 speakers is convolved with its pair of head-related impulse responses by uniformly partitioned overlap-save convolution, 128-sample partitions in 256-point transforms, with two real signals sharing each complex transform, so a block of 128 samples costs six forward transforms, one inverse, and the products of two partitions. The output is 128 samples late, which atmos_renderer_latency reports; the LFE joins both ears at 0.5 without HRIRs, delayed the same.
• atmos_hrir.c holds the 11 virtual speakers’ responses, 160 taps per ear at 48 kHz, which node scripts/generate-atmos-hrir.mjs [diffuse.zip] writes. The source is the diffuse-field equalized set of Bill Gardner and Keith Martin, “HRTF Measurements of a KEMAR Dummy-Head Microphone”, MIT Media Lab, 1994 (diffuse.zip from https://sound.media.mit.edu/resources/KEMAR.html, SHA-256 a48e73137390626ef1e45d1de6a4d98530c6aa570e71b40f7c4442241ed5fa22, which the script checks). The directions follow BS.2051’s 4+7+0 layout: the floor speakers at azimuths 0, 30, 90, and 135 at elevation 0, the heights at azimuths 45 and 135 at KEMAR’s 40 degree elevation, the left speakers taking their mirror images with the ears swapped, as the set’s symmetric responses intend. The script resamples each 128-tap 44.1 kHz response with a Kaiser-windowed sinc, fades its last 16 taps, and scales the set to a mean energy of 0.5 per ear, a stereo pan’s level for one channel.

A test module of these sources and test/atmos_test_exports.c, which adds exports to build scenes and read parsed metadata, backs test/Atmos.test.ts. The tests parse hand-built payloads and real ones from TrueHD and E-AC-3 streams, pan objects onto every layout’s speakers, check power, fold-down, zones, ramps, and retargeting, and compare the binaural output with a direct convolution by the table’s taps. The parser decodes the object metadata of the truehd crate’s Atmos test assets as truehdd does, positions within 3.2e-7 in ADM units.

After a change, from the engine root:

make -C wgpud/atmos module                # Builds wgpud/build/atmos/module.wasm, the tests' module
npx vitest --watch=false wgpud/atmos
npm run decoders -- eac3 truehd            # The packages that compile these sources

Licenses

• The engine’s modules, the bridges, the object audio sources in wgpud/atmos/, the HEVC, JPEG 2000, MPEG-2, and VC-1 parsers, the decoder sources, and the libdovi crate are MIT, like the rest of the engine. The vendored dolby_vision crate is MIT too and keeps its own LICENSE.
• FFmpeg and libdcadec are LGPL-2.1-or-later, and only the eac3, truehd, wma, and dts packages contain them. wgpud/licenses/ holds their license texts, FFmpeg-COPYING.LGPLv2.1 and libdcadec-COPYING.LGPLv2.1. wgpud/truehd/truehd_presentation.c and its header, a port of FFmpeg’s MLP decoder, are LGPL-2.1-or-later too, and so are truehd_substream.c and its header, which build on it. The eac3 package’s JOC sources (eac3_emdf.c, eac3_joc.c, eac3_qmf.c, eac3_joc_tables.c) are our own code, MIT; the tables are facts of ETSI TS 103 420.
• libaom is BSD-2-Clause with the Alliance for Open Media Patent License 1.0, and only the av1 package contains it. wgpud/licenses/ holds its LICENSE as libaom-LICENSE.
• The HRIR table in wgpud/atmos/atmos_hrir.c, which the eac3 and truehd packages contain, comes from the MIT KEMAR measurements, “Copyright 1994 by the MIT Media Laboratory”, “provided free with no restrictions on use, provided the authors are cited when the data is used in any research or commercial application”; the file’s header cites them, as whoever distributes those packages must.
• The engine ships no decoder package; the decoders repository’s releases do. Whoever distributes the LGPL packages must offer their corresponding source: make -C wgpud source-archives writes the exact FFmpeg and dcadec trees, as ffmpeg-<commit>.tar.gz and dcadec-<commit>.tar.gz, and the decoders repository at its checked-out commit, as webgpu-player-decoders-<revision>.tar.gz, to wgpud/build/source-archives/, and every release carries the three.

Layout

Codec vectors

A codec vector is a known input with a known answer. Qualification vectors ship to the browser, where the capability probes check native decoders against them before a route is advertised. A decoder package’s self-test vectors ship inside the package, and run only when a host verifies it (see Decoder packages). Test vectors only feed tests and generators.

The generators are Python 3.10 or later, in scripts/codec_vector_assets/, with their tests in scripts/codec_vector_assets/test/. They write the committed bin/codec_vector_assets/, except the local playback media generators, which write the ignored bin/playback_smoke_media/. A generate_ script makes its output from nothing but the repository and FFmpeg, while a create_ script derives it from media you supply. Hand-made vectors live in src/capability/vectors/.

Generate or check everything

From the engine root:

python scripts/generate_all_codec_vector_assets.py            # Writes bin/codec_vector_assets/
python scripts/generate_all_codec_vector_assets.py --check    # Fails unless every committed asset matches
python -m unittest discover -s scripts/codec_vector_assets/test -p "*_test.py"

generate_all_codec_vector_assets.py runs every generator of committed assets in order, and passes --regenerate-sources to the TrueHD generator so its source streams are re-encoded and compared too. It continues past a failure and exits with status 1 if any generator failed. It does not run the local playback media generators.

bin/codec_vector_assets/ is the one committed folder in bin/, so every change to an asset shows in review. .gitattributes stores it without line-ending conversion, so the committed bytes are the generated bytes.

Every committed asset reproduces with FFmpeg and FFprobe 2026-03-01-git-862338fe31-full_build-www.gyan.dev on PATH. The HEVC range-extension, AV1 raw-format, H.264 raw-format, AV1 Dolby Vision, HDR10+ AV1, HDR10+ VP9, and WMA decoder generators refuse any other build.

Committed assets

Two helpers serve them: generated_output.py (shared write, check, and install helpers) and engine_layout.py (the engine’s folders, read from tools/constants.json through tools/constants.py). avi_rewriting.py rewrites an FFmpeg AVI into the OpenDML layout FFmpeg’s muxer writes past 1 GiB, with a smaller RIFF list limit, and into DivX’s packed bitstream. The TrueHD generator computes the TypeScript import paths it embeds from the same constants, so moving a folder only needs a rerun.

The AV1 Dolby Vision and both HDR10+ generators take the error they raise, the pinned FFmpeg check, JSON comparison, and bit packing from media_tools.py. Both HDR10+ generators take the HDR10+ frame model, its ITU-T T.35 serialization, and its known answers for expectations.json and FFprobe from HDR10_plus_metadata.py.

The truehd decoder project’s tests import the TrueHD module as #codec_vector_assets/*, and scripts/library-assets.mjs serves the range extension, AV1 raw-format, and H.264 raw-format streams.

The generators work in one of three modes:

• Deterministic generators rewrite their output, and with --check fail when a committed file differs from a fresh run: the TrueHD module, the 7.1 downmix reference, the MPEG-2 decoder streams, and the decoder projects’ vectors files.
• Encoder-dependent generators compare a fresh encode with the committed bytes: the JPEG 2000, AV1, and MPEG-4 decoder media and the TrueHD source streams under --regenerate-sources. Another encoder build writes different bytes, so a differing encode never replaces a committed asset, while a missing one is installed. With --check, a missing asset fails as well.
• The HEVC range-extension, AV1 raw-format, H.264 raw-format, AV1 Dolby Vision, HDR10+ AV1, HDR10+ VP9, and WMA decoder generators refuse any FFmpeg build but the pinned one. Before writing each encode the HEVC generator checks the access-unit lengths, the picture types, the profile and constraint flags of the VPS and SPS, and the decoded fingerprints. The AV1 raw-format generator runs the checks listed in AV1 raw-format vectors, and the H.264 raw-format generator those in H.264 raw-format vectors. The AV1 Dolby Vision generator runs the checks listed in AV1 Dolby Vision vectors, the HDR10+ AV1 generator those in HDR10+ AV1 vectors, and the HDR10+ VP9 generator those in HDR10+ VP9 vectors. The WMA decoder generator installs its media as the encoder-dependent generators do and rewrites its known answers from the committed media.

HEVC range-extension vectors

These Annex B files are deterministic WebCodecs capability probes. They are 192x192, because common NVIDIA HEVC decoders reject smaller coded sizes. They prove only that the exact codec configuration decodes, copies to the expected WebCodecs raw format, and produces the expected fingerprints. They prove nothing about performance, resolution or level limits, seeking, or transport.

Toolchain: FFmpeg 2026-03-01-git-862338fe31-full_build-www.gyan.dev, libavcodec 62.24.100, libx265 4.1+225-1b48507eb.

The source is FFmpeg testsrc2 at 192x192 and one frame per second. Each of the nine vectors is one IDR access unit followed by one P access unit, which qualifies general inter decode for the exact tuple.

Every vector uses these x265 settings, serial because there is no thread pool and one frame thread:

-preset fast -crf 32
-x265-params info=0:pools=none:frame-threads=1:wpp=0:log-level=error:keyint=30:min-keyint=30:scenecut=0:bframes=0:repeat-headers=1

info=0 prevents the x265 user-data SEI, so nothing is stripped after encoding. For example:

ffmpeg -f lavfi -i "testsrc2=size=192x192:rate=1:duration=2" \
  -frames:v 2 -pix_fmt yuv422p10le -c:v libx265 \
  -profile:v main422-10 -preset fast -crf 32 \
  -x265-params "info=0:pools=none:frame-threads=1:wpp=0:log-level=error:keyint=30:min-keyint=30:scenecut=0:bframes=0:repeat-headers=1" \
  -f hevc -y main422-10.hevc

After each encode the generator rewrites the profile, tier, and level fields of the VPS and SPS, removing and reinserting the emulation-prevention bytes around the rewritten RBSP:

• Level: every vector signals Level 3.1 (general_level_idc 93). x265 gets no level-idc, because with CRF it would enforce the level through VBV, which it reports as non-deterministic. Without it x265 signals Level 1 and the slices are byte-identical, so the generator writes 93 instead.
• Profile: x265 does not emit general profile IDC 4 for 4:2:0 at 8 or 10 bits, so for rext420-8 and rext420-10 the generator starts from conforming Main or Main 10 syntax and rewrites the fields to profile IDC 4, compatibility flag 4, and the general constraints 9F.88 or 9D.88.

For every vector the generator verifies the FFprobe access-unit lengths and I/P types, the VPS and SPS constraint flags, and raw plane fingerprints computed as in production. Without --check, it replaces each committed file once its checks pass. --inspect prints the measured evidence without comparing or writing.

test/capability/HEVCRangeExtensionCapabilities.test.ts verifies the VPS constraint bytes, the intra and one-picture flags, the access-unit lengths, and the I/P slice sequence. Fingerprints use the production 32-bit FNV-1a sampler in capability/CustomDecodeCapabilities.ts: the plane dimensions, then a 64 by 36 uniform grid of samples over each output plane.

Re-encoding with another toolchain may not be byte-identical. Do not update an access-unit length, a constraint string, or a fingerprint until the static vector tests and the decoded raw output have been inspected together.

AV1 raw-format vectors

These low-overhead OBU streams (AV1 Section 5) qualify the raw-plane route for the AV1 formats outside Main 10, which the raw AV1 HDR probe qualifies. Like the HEVC range-extension vectors they are 192x192, fetched as library assets, and prove only that the exact codec configuration decodes, copies to the expected WebCodecs raw format, and produces the expected fingerprints. scripts/library-assets.mjs serves each av1-raw-format/<variant>.obu as webgpu-player/av1-raw/<variant>.bin.

Toolchain: FFmpeg 2026-03-01-git-862338fe31-full_build-www.gyan.dev, libavcodec 62.24.100, libaom 3.13.1-271-g85d21cdb24, libdav1d 1.5.3-18-gdaef396.

The source is FFmpeg testsrc2 at 192x192 and one frame per second. Each of the eight vectors is one key temporal unit followed by one inter temporal unit, back to back, and only the key unit carries the sequence header.

Every vector uses these libaom settings, serial because there is one thread, no row multithreading, and one tile:

-usage good -cpu-used 6 -crf 40 -b:v 0 -g 30 -lag-in-frames 0 -threads 1 -row-mt 0 -tiles 1x1

Good-quality usage clamps its speed to 6, so a higher -cpu-used writes the same bytes. No lookahead keeps each temporal unit to one shown frame, with no hidden alternate reference. For example:

ffmpeg -f lavfi -i "testsrc2=size=192x192:rate=1:duration=2" \
  -frames:v 2 -map_metadata -1 -fflags +bitexact -flags:v +bitexact \
  -pix_fmt yuv422p10le -c:v libaom-av1 \
  -usage good -cpu-used 6 -crf 40 -b:v 0 -g 30 -lag-in-frames 0 \
  -threads 1 -row-mt 0 -tiles 1x1 \
  -f obu -y professional422-10.obu

Before writing each vector the generator checks:

• the FFprobe temporal-unit lengths, one OBU demuxer packet each;
• the I/P picture types of libdav1d’s decode;
• the sequence header’s seq_profile, BitDepth, mono_chrome 0, and subsampling;
• raw plane fingerprints of libdav1d’s decode, computed as in production.

Without --check, it replaces each committed file once its checks pass. --inspect prints the measured evidence without comparing or writing.

test/capability/AV1RawFormatCapabilities.test.ts reads the committed bytes and verifies the temporal-unit lengths, the key then inter frame types, and the sequence header’s profile, depth, and subsampling. The fingerprints use the same production sampler as the HEVC range-extension vectors. Re-encoding with another toolchain may not be byte-identical, so update a length or a fingerprint only after inspecting the static vector tests and the decoded raw output together.

H.264 raw-format vectors

These Annex B streams qualify the raw-plane route for H.264 beyond 8-bit 4:2:0, the profiles beyond High. Like the AV1 raw-format vectors they are 192x192, fetched as library assets, and prove only that the exact codec configuration decodes, copies to the expected WebCodecs raw format, and produces the expected fingerprints. scripts/library-assets.mjs serves each h264-raw-format/<variant>.h264 as webgpu-player/h264-raw/<variant>.bin.

Toolchain: FFmpeg 2026-03-01-git-862338fe31-full_build-www.gyan.dev, libavcodec 62.24.100, libx264 core 165 r3223 0480cb0, whose version the generator reads from the user data SEI of an unfiltered encode.

The source is FFmpeg testsrc2 at 192x192 and one frame per second. Each of the five vectors is an IDR access unit, a P access unit, and a B access unit, back to back in decode order, and only the IDR access unit carries the SPS and PPS. The B picture presents between the other two, so its frame outputs before the P frame’s, and a probe that expected decode order would fail. The capability definitions stamp each access unit with its presentation time: 0 s, 2 s, and 1 s.

Every vector uses these libx264 settings, serial because there is one thread and no sliced threads, with one B-frame, no adaptive B-frame placement, and no pyramid:

-preset medium -crf 30 -threads 1
-x264-params keyint=30:min-keyint=30:scenecut=0:bframes=1:b-adapt=0:b-pyramid=none:threads=1:sliced-threads=0
-bsf:v filter_units=remove_types=6

The bitstream filter drops every SEI NAL unit, x264’s version and option string among them. For example:

ffmpeg -f lavfi -i "testsrc2=size=192x192:rate=1:duration=3" \
  -frames:v 3 -map_metadata -1 -fflags +bitexact -flags:v +bitexact \
  -pix_fmt yuv422p10le -c:v libx264 -profile:v high422 \
  -preset medium -crf 30 -threads 1 \
  -x264-params "keyint=30:min-keyint=30:scenecut=0:bframes=1:b-adapt=0:b-pyramid=none:threads=1:sliced-threads=0" \
  -bsf:v filter_units=remove_types=6 -f h264 -y high422-10.h264

Before writing each vector the generator checks:

• the FFprobe access-unit lengths, one packet of FFmpeg’s H.264 demuxer each;
• the slice types I, P, B in decode order, from each access unit’s first slice header, and the picture types I, B, P of FFmpeg’s decode in presentation order;
• the SPS’s profile_idc, chroma_format_idc, and both bit depths, and the codec string, which is the SPS’s profile, constraint flags, and level;
• raw plane fingerprints of FFmpeg’s decode in presentation order, computed as in production.

FFmpeg’s H.264 decoder is bit-exact, and Chrome’s software H.264 decoder is FFmpeg’s, so Chrome decodes each vector to the pinned fingerprints. Without --check, it replaces each committed file once its checks pass. --inspect prints the measured evidence without comparing or writing.

test/capability/H264RawFormatCapabilities.test.ts reads the committed bytes and verifies the access-unit lengths, the NAL unit types, the slice types, the SPS’s profile, chroma format, and bit depths, and the codec string. Re-encoding with another toolchain may not be byte-identical, so update a length or a fingerprint only after inspecting the static vector tests and the decoded raw output together.

AV1 decoder vectors

These are the av1 decoder project’s self-test media, one Matroska stream per output format. wgpud/av1/decoder.json lists them with the decoder options -c:v libdav1d, so the package’s known answers are libdav1d’s decode, film grain included. libaom’s decoder decodes every frame of every stream to the same bytes as libdav1d. generate_av1_decoder_vectors.py --print-declarations prints that list.

FFmpeg’s libaom-av1 encoder encodes each stream in one thread at speed 0 and CRF 50, and the generator runs the nine encodes at once. Each stream is 13 frames at 24 frames per second with a key frame every 12, so it holds two key frames. A 19-frame lookahead makes four hidden alternate references, three or four of which show_existing_frame shows.

The source is a lavfi canvas of 256x192 that rotates, zooms, and pans as a whole, from the identity at the first frame, which libaom codes with global and local warped motion:

• a frozen 64x64 Game of Life pattern of two colours at (0, 0), copied to (128, 64) and (128, 128), where IntraBC may copy it;
• a second pattern at (64, 0), for palette;
• two testsrc2 pictures, whose moving parts give OBMC, compound, and inter-intra prediction;
• smooth planes that geq draws at (192, 64), drifting each frame, for 64-point transforms. lavfi’s gradients source draws random points whatever its seed, so it is not used.

libaom lets an IntraBC block copy only from more than four 64x64 superblocks back in its tile, in raster order. With 64x64 superblocks the tiles are 3+1 columns and 2+1 rows of superblocks, so the copy at (128, 64) and the pattern at (0, 0) share a tile three superblocks wide. With 128x128 superblocks there is one tile, and the copy at (128, 128) is one superblock row below the pattern.

Every encode also asks for screen content tools (tune-content=screen), segmentation (-aq-mode 1), delta quantizer and loop filter levels, and quantization matrices. It turns off the temporal filtering of key frames, which blurs the copies IntraBC needs.

A schedule is a libaom film grain table the generator writes beside the encode, one entry per range of frames:

• schedule A: autoregression lag 0 with chroma scaled from luma, then no grain, then lag 1 without overlap, then lag 3;
• schedule B: lag 2 with chroma scaled from luma and without overlap, then lag 3, then no grain, then lag 1.

The table has bare line feeds, because libaom’s reader stops at a carriage return without an error and the encode then has no grain. libaom’s test tables carry no lag 0 or 1, but they clip to the restricted range, which a table cannot ask for.

The streams hold no super-resolution, reference scaling, dual filters, lossless frames, intra-only frames, or switch frames, so a self-test does not cover them. The pinned FFmpeg cannot ask its libaom for super-resolution or a frame size change, because that libaom takes no superres-mode or resize-mode key-value option, and libaom’s speed features disable dual filters at every speed.

MPEG-4 decoder media

These are the mpeg4 decoder project’s media: its self-test stream, and a stream of each codec in each container the engine routes it from, which the route tests play. wgpud/mpeg4/decoder.json lists the self-test stream with the decoder options -threads 1 -flags +bitexact, FFmpeg’s C code in one thread, and with codecPrivate bitmap-info-header, the codec private data the engine passes the package.

The self-test stream is 200x120 from libxvid, with global motion compensation, quarter-sample motion, and two B-VOPs between anchors. Its source is testsrc2 at twice the size, turned and zoomed into its center a little more each frame, so libxvid codes S-VOPs among its P-VOPs. libxvid writes a raw stream, from which the generator drops the N-VOPs libxvid writes for its delayed frames, as a native Matroska muxer drops them. FFmpeg then remuxes it into Matroska with the presentation timestamps it generates, since FFmpeg’s libxvid output in Matroska stamps each P-VOP before the B-VOPs shown ahead of it.

The other streams are 16 frames with a key frame every 8, the MPEG-4 Part 2 and MS-MPEG4 ones at 176x144. Every encoder runs in one thread, since the MPEG-4 Part 2 encoder’s slice threads change its bitstream with the thread count. FFmpeg’s Matroska muxer writes MPEG-4 Part 2 only as a native track, so the generator rewrites its file into mpeg4-b-vfw.mkv. The rewrite sets the codec ID, a CodecPrivate of the BITMAPINFOHEADER FFmpeg’s muxers write followed by the VOL, and block timestamps one frame apart in decode order, as an AVI stamps its frames. It drops the seek head and the cues, whose offsets it moves.

Each medium must probe as its codec, FourCC, and size, show its picture types, and decode one frame per packet in the reference decode without an error. The self-test stream must also present its frames in the order of their timestamps.

WMA decoder vectors

These are the wma decoder project’s media and FFmpeg’s known answers for them, which wgpud/wma/test/WMAPackage.test.ts decodes through the package and the engine’s WAVEFORMATEX reader, and two media with video for the engine’s decode worker tests, which play only sources with video. The generator writes them with the pinned FFmpeg, libavcodec 62.24.100.