Live audio and AudioReactive

Contract for the 360 ART CENTER MVP. Schemas: schemas/api/1.0/audio-reactive.schema.json, gateway.schema.json (resolumeConfig, resolumeStatus), asset.schema.json (texture, textureSource). Operations: openapi/paths/audio-reactive.yaml. Topics: events/topics.yaml.

Rates, intervals and defaults here are limits of the interface. They say nothing about measured latency or throughput.

Chain and layers

Link	Layer	Component
Capture and analysis	L1	anAntaDigital; frames over ananta.digital.v1
Mapping, smoothing, scenes	L3	media server, AudioReactive service
Resolume driver (OSC, REST, WebSocket)	L3	media server, gateway resolume
Frames from Resolume, warp, blend, output	L1	freedome-engined, stream asset

The topics below are for monitoring, such as meters in Manager. The live path from anAntaDigital to the media server is the L2 gRPC stream: AudioReactive must not depend on the public event bus, and analysisHz does not limit it.

Analysis

PUT /venues/{venueId}/audio/analysis sets the inputs. Each input has an id that scenes refer to, such as vocal.

Each frame carries, for one input:

Field	Range	Meaning
rms, peak	0..1	linear amplitude relative to full scale, clamped, after envelope.attackMs / decayMs (defaults 8 ms and 120 ms)
rmsDb, peakDb	dBFS	the same before normalisation
bands.subBass	0..1	30–150 Hz
bands.mid	0..1	150–800 Hz
bands.highMid	0..1	800–4000 Hz
bands.treble	0..1	4000–16000 Hz
centroid, centroidHz	0..1, Hz	spectral centroid, log scale over centroidRange
onset	0..1	attack strength in this frame, 0 = none
gated	boolean	below gateDb; normalised values are 0
captureUs	µs	start of the audio block on the analysis node’s monotonic clock

bands.* are linear amplitude too, so a voice at -24 dBFS reads 0.06. Scenes usually need range.inMax or a curve to use the full output range. captureUs is there to measure each link. It is comparable only between timestamps taken on the same node.

Topic	Kind	Class
venues/{venueId}/audio/analysis	audioAnalysis.state	S
venues/{venueId}/audio/analysis/{inputId}	audio.analysis	T: default 15 Hz, up to 60 Hz with subscribe option analysisHz; not kept in the ring buffer
venues/{venueId}/audio/onsets/{inputId}	audio.onset	E: one per attack, spaced by onset.refractoryMs

Scenes

A scene is a document: a list of mappings from one input metric to one target. PUT /venues/{venueId}/audio/reactive/scenes/{sceneId} stores it (If-None-Match: * to create, If-Match to replace, as for audio presets). PATCH /venues/{venueId}/audio/reactive selects it with activeSceneId. Example scenes are in examples/api/1.0/valid/audio-reactive@scene--*.json: vocal-aura, percussion-wave, combined-space.

Order of processing for a continuous mapping:

1. Take the metric from the newest frame of source.input.
2. Map range.inMin..inMax to 0..1 and clamp.
3. Apply curve to t in 0..1 (linear by default, table below).
4. Map 0..1 to range.outMin..outMax. outMin above outMax inverts.
5. Apply smoothing (attackMs when rising, decayMs when falling).
6. Blend towards outMin by the state’s intensity: 1 = the full value, 0 = outMin.
7. Send to target.

curve	Value	At 0.5
linear	t	0.5
exp	t²	0.25
log	log10(1 + 9t)	0.74
s-curve	3t² − 2t³ (smoothstep)	0.5

All four map 0 to 0 and 1 to 1. A server that does not know a curve value refuses the scene with 422.

A trigger mapping sends one message with target.value (default 1) when the metric rises through threshold, and then stays silent for cooldownMs. intensity does not apply to it.

The server refuses with 422: inMax not above inMin, duplicate mapping ids, a target kind it does not know. Selecting a scene is also refused with 422 when its gateway or input does not exist, or when it does not fit the gateway (see Resolume below). Storing such a scene succeeds, because the gateway may be configured later. If an input stops at run time, the status becomes degraded and its mappings hold their last value.

Resolume gateway

Gateway id: resolume, direction: outbound, protocol: resolume, managed with getGateway, patchGateway and testGateway. config is resolumeConfig; live state is the read-only member resolume. patchGateway cannot see the protocol in its schema, so the server validates config and answers 422.

gateways:manage is an admin scope. Operators read the same status object as member resolume of getAudioReactive and event audioReactive.state (playback:read). It is absent when no gateway resolume is configured, and it is read-only.

OSC over UDP has no acknowledgement. resolume.osc.state: up means the driver is sending, not that Resolume received it. rest and ws report real connections.

Values. Resolume normalises OSC floats: 0..1 covers the whole range of the parameter, whatever its unit. For clip or transform scale, 0..1 is 0–1000 %, so 100–200 % is 0.1..0.2. On a resolume target a continuous mapping must use type f with outMin and outMax in 0..1; selecting a scene that breaks this is 422, not a silent clamp. The ranges 0..2 and 0..5 are not wire values. The scene addresses of effects are examples: confirm them on the real composition.

Driver settings may be fixed by the server’s configuration: patchGateway then answers 422 for config.

Ports. Resolume uses OSC 7000 and web 8080 by default, the same as the media server. On one machine move the media server’s ports in its configuration, so that Resolume keeps its own defaults.

Texture sources

Frames from another application reach the engine as an asset of type: stream. src selects the transport:

Transport	src	Platform
Spout	spout://<sender>	Windows
Syphon	syphon://<application>/<server>	macOS
DMA-BUF	dmabuf://<pipewire-node>	Linux
Capture card	v4l2:///dev/<device>	Linux
NDI	ndi://<source>	spare path

Names are percent-encoded. POST /assets with url registers the asset, and fallbackUrl names a spare source. The server fills the read-only texture member from src. The engine receives src unchanged as ContentRef.uri, with AssetRef.type = "stream". Play it like any other asset; warp, blend and output are unchanged.

GET /venues/{venueId}/texture-sources lists the senders the engines can see (engine RPC Inputs.ListInputs), and events textureSource.state on venues/{venueId}/texture-sources/{sourceId} report changes. engine.capabilities.textureInputs lists what an engine can receive (ListInputsResponse.kinds). A transport the engine lacks: the asset is ready but the player reports player.error.