Macroblock
Audio-reactive chroma subsampling, taken too far, for Resolume
What it does
Video has never stored colour at full resolution. A codec splits the picture into luma (brightness, where all the detail lives) and two chroma channels, then samples the chroma on a coarser grid. H.264 and HEVC take one colour sample per 2×2 block of pixels, DV took one per four pixels across, and a broadcast link takes one per two. You've never noticed, because at that scale it's very hard to see. That's how the codecs get away with it.
Macroblock lets you pick the grid. It's the same arithmetic a codec does, with the block size on a slider that runs from the two-pixel pair all the way to one colour for the entire canvas, at which point the picture keeps every bit of its detail and turns into a monochrome of its own average colour. That's worth getting your head round before you touch anything: subsampling chroma doesn't soften a picture. Nothing moves and nothing blurs. The only thing that changes is how often the colour gets measured.
Which is why there's a second, separate lattice for luma, doing something no sampling format has ever offered: brightness in blocks, with every colour edge left exactly where it was. It looks like a mosaic that's somehow hung on to its outlines.
In Resolume the grid follows the music. Resolume hands a plugin a 64-bin spectrum each frame, and Macroblock splits it into three bands plus full range and normalises each against its own recent peak, so the same patch works on a quiet stem and on a mastered track. There are three rules for what that does to the lattice: follow the level, latch a new size on each onset and let it decay, or gate hard above a threshold. Chroma and luma each get their own band and their own signed amount, so the audio can sharpen the colour on a hit as easily as coarsen it.
- An arbitrary chroma lattice: absolute in pixels below 64 of them, so a format preset is that format at 720p, 1080p and 4K alike, and relative to the canvas above, so the end of the travel is one colour for the whole frame at any raster
- An independent luma lattice — the inverse of every codec, and a thing no sampling format does
- Ten presets, four of them real formats: 4:2:2, 4:2:0, 4:1:1 (DV) and 4:1:0 (Video CD)
- Rec. 709, 601, 2020 and YCoCg matrices, averaging in gamma or in linear light
- Box mean or point sampling; blocky or smooth reconstruction; centred or co-sited siting
- Audio-reactive in Resolume: Follow, Step and Gate over four bands, with attack, release, sensitivity and hold
- The same effect as an OpenFX plugin for Resolve, Nuke, Natron and Vegas, rendering from the same code
- Costs the same at every block size — the frame is read exactly once whatever the lattice is set to
Where it stands
Built, measured and released as v0.1.4. v0.1.1 only changed the Windows installer, which no longer touches the system PATH (the earlier one could overwrite it); v0.1.2 keeps a factory preset selected when the host restates its own values, and gives the effect a proper description in the field Resolume shows; v0.1.3 reports its name as SW Macroblock, so the Stoatworks plugins sit together in Resolume's browser; v0.1.4 is maintenance. The sampling lattice is proved to tile the frame across 154 combinations of block size and raster, including sizes that fall between pixels, which is where two real off-by-one bugs turned up, and every block's value is checked against a box mean worked out a third way. The GPU shader and the OpenFX CPU renderer are compared pixel for pixel across eleven configurations, so Resolume and Resolve agreeing is something I've measured. At zero the effect is bit-identical to its input, and all 24 controls are measured to change the picture. What hasn't happened: it's never been run inside Resolume or in any OpenFX host. The audio side has had real music through it (an audio file analysed into a spectrum and fed to the plugin through the same call Resolume makes), but what Resolume's 64 bins mean in hertz is an assumption only a session in Arena can settle.
Watch it running
Nothing loads from YouTube until you press play.Watch it there instead