Skip to content

Release 8 — Audio-Reactive & Live Scripts

Theme: Releases 3–7 delivered a scalable, networked LED controller. Release 8 makes it react — to sound from a built-in mic or a WLED-sync peer, and to on-device .sc scripts that users can edit from the UI without reflashing. Plus the first IR/PIR/button/I2C sensor inputs.

Release Overview

Gap Addressed by
No audio input; no FFT Sprints 1–2
Audio-reactive effects can't run without local mic Sprint 3 (WLED-sync remote audio)
No audio-reactive effect set Sprints 4–5
IR remote / PIR motion / I2C sensors have no host in IoModule Sprints 6–7
Effects can only be C++ compiled-in; no runtime scripting Sprints 8–10 (Live Scripts)

Sprint 1 — IO Audio: I2S + PDM Mic

Scope

Goal: extend IoModule with audio pin types; init I2S or PDM at boot.

Part A — Audio pin types. audio_i2s_sd, audio_i2s_ws, audio_i2s_sck, audio_pdm_sd, audio_pdm_ws. Auto-detect PDM when only SD+WS are assigned (no SCK).

Part B — AudioInputModule. Claims audio pins; configures I2S peripheral; provides a getSampleBlock(uint16_t* out, size_t n) API. Default: 16 kHz mono, block size 1024.

Part C — Supported mics. INMP441 (I2S standard), onboard PDM mic on Dig-Next-2, line-in via the MHC P4 switch position. All appear identical to downstream modules.

Definition of Done:

  • INMP441 on an ESP32-S3 DevKit produces non-zero samples
  • PDM auto-detection verified: configuring only SD+WS switches to PDM mode
  • Block-level SNR documented per mic type

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 2 — FFT Pipeline: bands[16] + Volume + Magnitude

Scope

Goal: turn audio samples into the shared global audio fields that effects read.

Part A — FftModule. Child of AudioInputModule. Runs FFT (arduinoFFT or ESP-DSP for S3/P4) per block. Exposes:

  • bands[16] — log-spaced frequency bins
  • volume — RMS of the block
  • magnitude — peak amplitude

Updated at the block rate (~62 Hz at 16 kHz / 256 samples).

Part B — Smoothing + AGC. Per-band smoothing (exponential average) + optional AGC to normalize across quiet-vs-loud rooms. Controls: smoothing, agc_enabled, agc_target.

Part C — Shared global struct. AudioState struct (1 KB) lives in Scheduler; FftModule writes, all effects read. No mutex — single-writer / multi-reader with atomic snapshot swap.

Definition of Done:

  • Clapping produces visible peaks in bands[3..8] (mid frequencies)
  • AGC normalizes 40 dB dynamic range within 5 s
  • Published AudioState accessible from an effect via scheduler.audio()

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 3 — WLED-Sync UDP Audio Input

Scope

Goal: receive audio fields from a peer (WLED device with audio module, or another projectMM) — no local mic required.

Part A — WledSyncInputModule. Listens on the WLED audio-sync UDP port; packs received bands[16] / volume / magnitude into the same AudioState struct.

Part B — Source selection. LightsControlModule gains an audio_source dropdown: local_mic, wled_sync, none. Only one source active at a time.

Part C — Resilience. If WLED-sync packets stop arriving, fade fields to zero over 500 ms; no "frozen loud" state.

Definition of Done:

  • WLED audio peer → projectMM receives and reacts
  • projectMM with local mic → another projectMM with audio_source=wled_sync reacts identically
  • Packet-loss test (inject 50% drop) still produces visually continuous reaction

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 4 — Audio Effects: GEQ + Frequency Matrix

Scope

Goal: the two canonical audio visualizations.

Part A — GeqEffect. 16 vertical bars; each bar height = bands[i]. Classic graphic EQ. 2D-native; on 1D layouts it collapses to a rolling spectrum.

Part B — FrequencyMatrixEffect. WLED-ported. Each column = one frequency band; each row = time; scrolling matrix.

Definition of Done:

  • GeqEffect on 16×16 panel shows 16 bars moving with audio
  • FrequencyMatrixEffect visibly responds to treble vs bass
  • Both work with local mic and wled_sync sources interchangeably

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 5 — Audio Effects: Noise Meter, Waverly, Blurz, PopCorn

Scope

Goal: four more WLED-ported audio effects across a range of styles.

Part A — NoiseMeterEffect. 2D noise field with intensity driven by volume; scrolls faster with louder audio.

Part B — WaverlyEffect. Horizontal waves modulated by bass; color mapped to bands[0..3].

Part C — BlurzEffect. Gaussian-blurred highlights on detected beats; uses magnitude threshold for beat detection.

Part D — PopCornEffect. Particles pop up from the bottom on beats; fall back under gravity. Builds on R5's ParticlesEffect.

Definition of Done:

  • All 4 registered; each reacts visibly to audio
  • Render determinism test stubbed: given a canned AudioState snapshot, same pixels emitted
  • Combined footprint: ≤ 15 KB flash for all 4

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 6 — IO Sensors: IR, PIR, Button/Toggle

Scope

Goal: basic inputs that affect the light state.

Part A — IrReceiverModule. Claims an ir_in pin; decodes NEC / RC5 / Sony. Exposes last_code (hex string) + a mapping table to control actions.

Part B — PirMotionModule. Claims a pir pin. Triggers on → true on motion, on → false after configurable timeout.

Part C — Button expansion. IoModule's button pin type gets variants: button_momentary, button_toggle, button_lighton (toggles lights-control on/off directly). Active-low with 20 ms debounce.

Definition of Done:

  • IR remote button mapped to brightness++ updates brightness on each press
  • PIR triggers lights on 30-sec timeout within 1 second of motion
  • Three button variants all work from hardware side + unit-tested

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 7 — IO: I2C Bus + Energy Monitoring ADC

Scope

Goal: the I2C infrastructure (for future IMUs, temperature sensors) and the energy-monitoring ADCs MoonLight exposes.

Part A — I2cBusModule. Claims SDA/SCL. Initializes bus at 100/400 kHz. Scan-on-demand returns address + (if known) device name table.

Part B — ADC pin types. adc_voltage, adc_current, adc_battery. EnergyMonitorModule polls these and exposes to SystemStatusModule for display.

Part C — Scan UI. I2C bus module has a "Scan" button in the UI; results appear as a read-only display control listing each address found.

Definition of Done:

  • I2C scan finds a known bench sensor (e.g. BME280 at 0x76)
  • Voltage + current + battery ADC reading displayed in the UI
  • Pin-claim test: attempting to assign SDA+SCL on a bus-inactive board succeeds; attempting the same pins to a second I2C module fails

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 8 — Live Scripts: ESPLiveScript Integration

Scope

Goal: ship the on-device script compiler + a runtime that can execute a single .sc file.

Part A — ESPLiveScript vendored. Commit-pinned copy of the ESPLiveScript repo in lib/. PSRAM required → ESP32-S3 / ESP32-P4 only; PC build stubs the compile step.

Part B — LiveScriptModule. Reads a .sc file from LittleFS, compiles, runs the compiled function as a FreeRTOS task (8 KB stack, priority 3). Compilation itself runs on a dedicated 8 KB compile task.

Part C — File association by prefix. - E_*.sc → effect (called every tick via a VirtualLayer) - L_*.sc → layout (called once to build mapping) - P_*.sc → palette (called once to populate CRGBPalette)

Definition of Done:

  • A hand-written E_test.sc compiles, runs, outputs expected pixels
  • Compile error reported in the UI (not just serial) with line number
  • Runtime crash inside a script doesn't take down the main runtime — script task is killed cleanly

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 9 — Live Script API Bindings

Scope

Goal: the ~40 C++ helpers MoonLight exposes to scripts.

Part A — Generic / math bindings. millis, delay, random8, sqrt, sin, cos, atan2, sin8, cos8, inoise8, beatsin8.

Part B — LED bindings. setRGB(i, c), setRGBXY(x,y,c), setHSV(x,y,h,s,v), fadeToBlackBy(amount), ColorFromPalette(pal, idx), drawLine3D(a,b,c), blur2d(amount).

Part C — Layout bindings. addLight(x,y,z), nextPin(), addControl(name, type, default) (for Live-Script-declared controls).

Part D — Documentation. docs/developer-guide/live-scripts.md with one worked example per bindings group.

Definition of Done:

  • All ~40 bindings callable from a script; each covered in a smoke test
  • A non-trivial effect (E_wave.sc) renders correctly using palette + setHSV + noise
  • Gotchas from MoonLight docs (8-bit int, no uint16_t cast) flagged in the doc

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Sprint 10 — Live Scripts: Concurrency + Safety

Scope

Goal: up to 4 concurrent scripts with sync semaphores + watchdog protection.

Part A — Concurrency cap. Max 4 running scripts. LiveScriptModule counts active scripts; 5th attempt fails with a clear error.

Part B — Sync semaphore. WaitAnimationSync counting semaphore (capacity 4) with 1-sec timeout per cycle — prevents a stuck script from blocking others forever.

Part C — Watchdog. Each script task gets a watchdog registration; if a script loop takes > 200 ms, it's killed and the user sees the error.

Part D — UI editor. Simple editor in the frontend for .sc files: read, edit, save (triggers recompile), error overlay.

Definition of Done:

  • 4 scripts running in parallel, all producing independent layer output
  • Intentionally-slow script triggers watchdog within 250 ms, main runtime unaffected
  • Edit-in-UI cycle: edit → save → recompile → new behavior visible within 2 s

Result

To be completed after implementation.

Retrospective

To be completed after implementation.

Release 8 Backlog

  • Live Script debugger. Set breakpoints, step, inspect — heavyweight; pick up when demand is real.
  • Audio on classic ESP32. Memory-tight; defer until use case is proven.
  • Line-in audio processing pipeline. Higher dynamic range than mic. Pick up when bench use-case lands.
  • IMU-based effects (via I2C). Deferred to R11 alongside other frontier items.
  • Temperature sensors. Pick up alongside the first real thermal-aware effect or protective shutdown.
  • Beat-detection improvements. Current implementation is threshold on magnitude; BPM-tracking is richer but more complex.