Macroblock
Audio-reactive chroma subsampling for Resolume
What it does
Video has never stored colour at full resolution. A codec splits the picture into luma — brightness, where all the detail is — and two chroma channels, then samples the chroma on a coarser grid: H.264 and HEVC take one colour sample per 2×2 block of pixels, DV took one per four pixels across, a broadcast link takes one per two. You have never noticed, because at that scale it is genuinely hard to see.
Macroblock makes that grid arbitrary. The same arithmetic a codec performs, with the block size on a slider that runs from the two-pixel pair all the way to a single colour for the entire canvas — at which point the picture keeps every bit of its detail and becomes a monochrome of its own average colour. That is the part worth understanding before touching anything: subsampling chroma does not soften a picture. Nothing moves, nothing blurs; only how often the colour is measured changes.
Which is why there is a second, independent lattice for luma, doing the thing no sampling format has ever offered: brightness in blocks with every colour edge left exactly where it was. It reads as a mosaic that has somehow kept its outlines.
In Resolume the grid follows the music. Resolume hands a plugin a 64-bin spectrum each frame; Macroblock splits it into three bands plus full range, normalises each against its own recent peak so the same patch works on a quiet stem and a mastered track, and offers three rules for what that does to the lattice — follow the level, latch a new size on each detected onset and decay, or gate hard above a threshold. Chroma and luma get their own band and their own signed amount, so audio can sharpen the colour on a hit as easily as coarsen it.
- An arbitrary chroma lattice: absolute in pixels below 64 of them, so a format preset is that format at 720p, 1080p and 4K alike, and relative to the canvas above, so the end of the travel is one colour for the whole frame at any raster
- An independent luma lattice — the inverse of every codec, and a thing no sampling format does
- Ten presets, four of them real formats: 4:2:2, 4:2:0, 4:1:1 (DV) and 4:1:0 (Video CD)
- Rec. 709, 601, 2020 and YCoCg matrices, averaging in gamma or in linear light
- Box mean or point sampling; blocky or smooth reconstruction; centred or co-sited siting
- Audio-reactive in Resolume: Follow, Step and Gate over four bands, with attack, release, sensitivity and hold
- The same effect as an OpenFX plugin for Resolve, Nuke, Natron and Vegas, rendering from the same code
- Costs the same at every block size — the frame is read exactly once whatever the lattice is set to
Where it stands
Built, measured and released as v0.1.1; the point release changed nothing but the Windows installer, which no longer touches the system PATH — the earlier one could overwrite it. The sampling lattice is proved to partition the frame across 154 combinations of block size and raster — including sizes that fall between pixels, which is where two real off-by-one bugs were caught — and every block's value is checked against a box mean computed a third way. The GPU shader and the OpenFX CPU renderer are compared pixel for pixel across eleven configurations, so 'Resolume and Resolve agree' is a measurement rather than an intention. At zero the effect is bit-identical to its input, not merely close, and all 24 controls are measured to change the picture. What has NOT happened: it has never been run inside Resolume or in any OpenFX host. The audio side has been driven by real music — an audio file analysed into a spectrum and fed to the plugin through the same call Resolume makes — but what Resolume's 64 bins mean in hertz is an assumption only a session in Arena can settle.
Watch it running
Nothing loads from YouTube until you press play.Watch it there instead