Vocoder
A vocoder with your picture as the carrier, for Resolume
What it does
An FFGL effect for Resolume Arena and Avenue. A channel vocoder splits the carrier into frequency bands and sets each band's gain from the matching band of the modulator. That's the whole machine, and the rest is what you feed it. So feed it a picture: the carrier is the frame, its frequency is spatial frequency, and the modulator is the audio.
The frame is split into a Laplacian pyramid (eight octave bands, from one-pixel detail up to 128-pixel shapes, plus a residual), each band is multiplied by a gain, and the bands are added back together. With every gain at 1× the input comes back exactly. The reconstruction is arranged so that the thing being expanded is a difference that's identically zero, so the output is the input to the bit. Everything else in the plugin stands on that null, which is how you know the bands are doing what they say.
You get three things out of the same machine. With nothing routed it's a graphic EQ for spatial frequency: a boost at the 4 px band makes the picture ring at 4 px, a cut removes detail at that scale and leaves every other scale alone, and a tilt across the bank sharpens at one end and softens at the other. It's nine independent faders across scale, which is a different animal from a blur or a sharpen filter. With audio routed it's a vocoder: bass pumps the large shapes and treble sparkles the fine detail (or the other way round, with a switch), and the audio gains multiply the slider gains, so a band you've cut stays cut whatever the music does. And turning the Residual down leaves a new kind of edge picture, the picture as its detail bands with its DC and coarsest shapes taken out.
The band sliders run 0 to 4×, linear, with 1× a quarter of the way up. That's unusual, and it's on purpose. A graphic EQ's fader is symmetric in dB, but a vocoder band's gain is a multiplier that spends most of its life somewhere between off and as loud as the carrier, so this gives the cut side a usable length and leaves the boost side room to ring. A band is named after its level's sampling pitch, and its power peaks a bit lower (an octave band of a Burt–Adelson pyramid is centred well below its level's Nyquist), so the 4 px band rings with a period of about 21 px. The harness measures that rather than assuming it.
Sidechain Mode picks one envelope per band, which is the vocoder, or one envelope for the lot driving the Tilt, which makes it a dynamics processor across scale. Floor decides what a silent band does: at 0 quiet bands go dark, and at 1 the audio only ever adds.
- An eight-band Laplacian pyramid plus residual — 1 px detail to 128 px shapes
- Exact reconstruction at unity: the output is the input to the bit, which is the null every other claim rests on
- A graphic EQ for spatial frequency with nothing routed, and a vocoder with audio routed
- Band gains 0 to 4× with unity at a quarter, so cuts have length and boosts have room to ring
- ±6 dB tilt across the bank — sharpen at one end, soften at the other
- Direct or Reverse audio mapping: bass to coarse shapes, or bass to fine detail
- Vocoder or Dynamics sidechain — one envelope per band, or one envelope driving the tilt
- Residual at zero for a detail-band edge picture that is not an edge detector
- RGB or Luma carrier, so the input's colour can be left alone
- Universal macOS bundle and a Windows x64 DLL
Where it stands
v0.1.0, and early days. The central claim is numerical, so tools/verify.sh measures it against a fresh universal Release build: reconstruction is exact, with a maximum difference of 0 at 720p, 1080p and 4K (5.96e-08 through the Luma carrier's Y/Cb/Cr round trip); the eight band-cuts partition the picture; the shipped shaders agree with an independent CPU implementation of Burt–Adelson over all eight bands; the bands peak at 2.0 px up to 677.1 px, doubling as an octave bank must; all 16 audio-band-to-picture-band mapping cases are exact; and the envelope followers hold their attack and release at three settings. All 19 swept controls change the picture, with 2 skipped and the reasons given. It costs 0.77 ms at 720p, 0.66 at 1080p and 1.89 at 4K on macOS, and 1080p being quicker than 720p isn't a typo: there are 33 passes for eight levels, most of them tiny, so at small sizes the per-pass overhead wins. On 21 September 2026 the x64 DLL was registered, loaded and instantiated in Resolume Arena 7.27.1 on a Windows VM, listed as SW Vocoder under idstring VC01 among 112 video effects, with its shaders compiling, the inspector drawn and a clean diag log. That VM has no GPU (OpenGL came from Mesa llvmpipe), so it has never run on a GPU in Resolume, has never been instantiated in Arena on macOS, and nothing was timed on Windows. No real audio reached it in Arena, so what Resolume's 64 FFT bins actually mean is still an assumption rather than a measurement, and that's the biggest open question in the repo. There's been no long session, composition save and reload or preset recall in the host, and there's no OpenFX port, no user guide and no factory presets. The browser demo is live, but it's only the spatial-frequency EQ side: the audio group is left out because a browser has nothing like Resolume's FFT parameter. Try it in your own rig before you trust it in a show.
Watch it running
Nothing loads from YouTube until you press play.Watch it there instead