Residual
Datamosh, built as a real codec, for Resolume
What it does
An FFGL effect for Resolume Arena and Avenue. Datamosh is not a glitch. It is a video decoder doing exactly its job with the wrong information. A P-frame is a field of motion vectors plus a residual, the correction, applied to the last frame the decoder reconstructed.
So Residual is the codec: a hierarchical block-matching motion estimator, closed-loop prediction from the decoder's own reconstructed frame, a quantised 8×8 DCT residual in YCoCg-R, and a GOP with scene-cut detection. Then it has controls that break exactly one piece of it at a time, so every datamosh look is a named failure of a named stage.
Drop I takes away the I-frame at a scene cut, so the decoder carries on predicting the new scene from the old picture: the classic mosh. Residual Gain at zero is a decoder that never corrects, so whatever was on screen flows wherever the vectors say. Vector Hold reuses one frame's vectors for up to sixty frames, and the pixels keep travelling: the bloom. Vector Scale moves everything too far. Q quantises harder, into blocks and a closed loop that slowly loses the picture. Refresh is one I-frame now, the clean-up button, and Drop I can also be armed by an audio onset.
Everything the decoder keeps is an integer texture read with texelFetch, with no filtering anywhere in the loop, which is what lets the harness make bitwise claims about it: at Q 0 the codec is lossless.
- A real hybrid codec loop: motion search, closed-loop prediction, quantised 8×8 DCT residual, GOP
- Drop I (Next, All or On Onset) for the classic mosh at a cut
- Residual Gain, Vector Hold and Vector Scale for pure motion and the bloom
- Block sizes 8, 16 and 32, search range to ±32, optional half-pel
- Scene-cut detection and a Refresh button that forces one I-frame
- Lossless at Q 0, in reversible YCoCg-R
- Show Vectors draws the motion field over the picture
- Universal macOS bundle and a Windows x64 DLL
Where it stands
v0.1.0, and honestly early. The codec is measured, not asserted, and most of it bitwise, by an offline harness that drives the real plugin class headlessly at two rasters: with the I-frame dropped and the residual at zero, the frame after a hard cut is the previous decoded frame block-copied by the estimator's own vectors, 0 of 2,592,000 samples different across six cases; at Q 0 the codec is lossless, 0 of 49,766,400 samples different; a translated texture returns exactly its (dx, dy) from all 29,961 interior blocks over 63 cases; an I-frame is bitwise a function of its source alone; closed-loop drift grows by exactly one code a frame for 31 frames; and seven negative controls and a one-character mutation of the shipped shader are caught. All 14 sweepable controls change the picture. Render cost on macOS at the defaults is 5.1 ms/frame at 720p, 6.1 at 1080p and 14.0 at 4K; the motion search is the cost, and 8-pixel blocks at ±32 run at about 50 ms, which is not a setting for a show. It has not yet been loaded into Resolume on macOS. On Windows it has: a build of this source loads, registers and renders in Resolume Arena 7.27.1 on software rendering (win-lab, no GPU), with every parameter as declared; the motion-search and scene-cut controls could not be shown moving there, because the gate's picture is a still with no motion and no cut. Chroma is coded at full resolution, not 4:2:0, so the chroma-block look of a real mosh is not here yet. There is a user guide and a browser demo, which is a port rather than the plugin; no presets and no OpenFX port. The macOS builds are signed and notarised. Check it in your own rig before trusting it in a show.
Watch it running
Nothing loads from YouTube until you press play.Watch it there instead