Residual
A small video codec, broken on purpose, for Resolume
What it does
An FFGL effect for Resolume Arena and Avenue. Datamosh is a video decoder doing its job perfectly well with the wrong information. A P-frame is a field of motion vectors plus a residual, the correction, applied to the last frame the decoder reconstructed.
So Residual is the codec: a hierarchical block-matching motion estimator, closed-loop prediction from the decoder's own reconstructed frame, a quantised 8×8 DCT residual in YCoCg-R, and a GOP with scene-cut detection. Its controls each break one piece of it, so every datamosh look is a named failure of a named stage.
Drop I takes away the I-frame at a scene cut, so the decoder carries on predicting the new scene from the old picture: the classic mosh. Residual Gain at zero is a decoder that never corrects, so whatever was on screen flows wherever the vectors say. Vector Hold reuses one frame's vectors for up to sixty frames, and the pixels keep travelling, which is the bloom. Vector Scale moves everything too far. Q quantises harder, into blocks and a closed loop that slowly loses the picture. Refresh is one I-frame now, the clean-up button, and Drop I can also be armed by an audio onset.
Everything the decoder keeps is an integer texture read with texelFetch, with no filtering anywhere in the loop, which is what lets the harness make bitwise claims about it: at Q 0 the codec is lossless.
- A real hybrid codec loop: motion search, closed-loop prediction, quantised 8×8 DCT residual, GOP
- Drop I (Next, All or On Onset) for the classic mosh at a cut
- Residual Gain, Vector Hold and Vector Scale for pure motion and the bloom
- Block sizes 8, 16 and 32, search range to ±32, optional half-pel
- Scene-cut detection and a Refresh button that forces one I-frame
- Lossless at Q 0, in reversible YCoCg-R
- Show Vectors draws the motion field over the picture
- Universal macOS bundle and a Windows x64 DLL
Where it stands
v0.1.0, and early days. It hasn't been loaded into Resolume on macOS yet. On Windows it has: a build loads, registers and renders in Resolume Arena 7.27.1 on software rendering (no GPU), with every parameter as declared, but the motion-search and scene-cut controls couldn't be shown moving there, because the host check's picture is a still with no motion and no cut. The codec is measured by an offline harness driving the real plugin class at two rasters, mostly bit for bit: with the I-frame dropped and the residual at zero, the frame after a cut is the previous decoded frame moved by the estimator's own vectors with 0 of 2,592,000 samples different; at Q 0 the codec is lossless; a moved texture returns exactly its offset from every interior block; an I-frame depends on its source alone; and closed-loop drift grows by one code a frame. It catches seven models broken on purpose and a one-character sabotage of its own shader, and all 14 sweepable controls change the picture. Render cost on macOS at the defaults is 5.1 ms a frame at 720p, 6.1 at 1080p and 14.0 at 4K. The motion search is the expensive part, and 8-pixel blocks at ±32 take about 50 ms, which isn't a setting for a show. Chroma is coded at full resolution rather than 4:2:0, so the chroma-block look of a real mosh isn't here yet. There's a user guide and a browser demo, which is a port rather than the plugin, but no presets and no OpenFX port. The macOS builds are signed and notarised. Try it in your own rig before you trust it in a show.
Watch it running
Nothing loads from YouTube until you press play.Watch it there instead