Skip to content

Developer Guide — Architecture

This document is the single home for how projectMM is structured, why each design decision was made, and how to extend or modify the system. Separate files for architecture, design, and implementation no longer exist — they are redirect stubs pointing here. The API contract (REST + WebSocket) lives in api.md. Build, deploy, and CI instructions are in deploy.md.

System Overview

+--------------------------------------------------------------+
|                         Frontend                             |
|  (HTML/CSS/JS, renders JSON UI descriptions from Modules)    |
+----------------------------^---------------------------------+
                             | HTTP / WebSocket
+----------------------------v---------------------------------+
|                      Control Interfaces                      |
|           REST  |  MQTT  |  WebSocket  |  (future)           |
+----------------------------^---------------------------------+
                             |
+----------------------------v---------------------------------+
|                      Module Runtime (Core)                   |
|                                                              |
|   Scheduler  |  ModuleManager  |  TypeRegistry               |
|                                                              |
|                        Hot Path                              |
|   [ Module A .loop() ] -> [ Module B .loop() ] -> [ I/O ]    |
+----------------------------^---------------------------------+
                             |
+----------------------------v---------------------------------+
|                    Platform Abstraction                      |
|  GPIO  |  Net (Art-NET / DDP / E1.31)  |  FS  |  Light drivers  |
+----+-----------------------+-----------------------+---------+
     |                       |                       |
+----v------+         +------v-----+         +-------v------+
|  ESP32    |         |    rPi     |         |     PC       |
+-----------+         +------------+         +--------------+

Strict domain-agnostic rule. The core knows nothing about LEDs, audio, or sensors. Domain knowledge — including types like RGB and Channel — lives in domain-specific Modules (for lights, in Layer modules), not in src/core/.

Project Layout

src/
  core/        # Scheduler, Module base class, platform-agnostic runtime only
  modules/     # All Modules — both system and application
    layer/     # Lights-domain: EffectsLayer, DriverLayer, RGB, Channel
    ...        # effects, drivers, previewers, future audio/sensor domains
  platform/    # Platform abstraction layer (PAL)

Why src/core/ is domain-agnostic. In early development, lights types (RGB, Channel) lived in src/core/ as the shortest path to a working double buffer. They were moved to src/modules/layer/ once the rule was established. A future audio or sensor domain adds its own subdirectory under src/modules/ (e.g. src/modules/audio/) — the runtime in src/core/ does not change.

Module Runtime (Core)

The core is small, portable C/C++. It owns:

  • The Scheduler — walks a registered list of Modules each tick, calls setup() once on load, loop() repeatedly while active, and teardown() on unload. Records per-Module timings. Knows nothing about JSON, persistence, or module kinds. → Scheduler test coverage
  • The ModuleManager — a StatefulModule that owns the persistent application configuration (state/modulemanager.json), instantiates application modules from it via a type registry, injects their controls, wires their data-flow inputs, and calls scheduler.add() on each during its own setup().
  • The TypeRegistry — maps type-name strings to zero-argument factory functions. Modules self-register via the REGISTER_MODULE(TypeName) macro in src/modules/ModuleRegistrations.cpp, which is compiled directly into each executable (not into the core static library) to prevent the linker from dead-stripping the static initializers.

Module kinds and the ModuleManager

Modules come in two lifecycle kinds — a scheduling and configuration property, not a subclass:

  • System modules are loaded automatically after initial flash: AP/WiFi bring-up, system status, firmware update, the ModuleManager itself. Always present; a freshly-flashed device is usable with only system modules.
  • Application modules are the user-configurable set: light effects, Layers, drivers, sensors. Controlled by a persistent configuration document.

ModuleManager three-pass setup():

  1. Create — parse state/modulemanager.json; for each entry call TypeRegistry::create(type) then setProps(props). The base class stashes all prop values into pendingProps_.
  2. Wire — resolve "inputs" references by id; call setInput(key, sourceModule) for each connection.
  3. State + register — load state/<id>.json if present (calls loadState(), which merges saved values into pendingProps_), then scheduler.add() (which calls runSetup() → setup() → each addControl() drains and applies the matching pending value → pendingProps_ cleared).

teardown() serialises each module's state to state/<id>.json (via saveState(), which iterates all registered ControlDescriptors automatically).

Config file format (state/modulemanager.json): a JSON object with a "modules" array. Each entry has "id", "type", optional "props", optional "inputs" (data-flow wiring by module id), optional "parent_id", and optional "core". Array order is the scheduler execution order. The file is written automatically on first structural change and on shutdown.

{"modules": [
  {"id": "producer1", "type": "EffectsLayer",    "core": 1, "props": {"width": 16, "height": 16}},
  {"id": "sine1",     "type": "SineEffectModule", "core": 1, "parent_id": "producer1",
   "props": {"frequency": 1.0}, "inputs": {"layer": "producer1"}},
  {"id": "consumer1", "type": "DriverLayer",    "core": 1, "props": {"width": 16, "height": 16},
   "inputs": {"source": "producer1"}},
  {"id": "preview1",  "type": "PreviewModule",    "core": 1, "parent_id": "consumer1",
   "inputs": {"source": "consumer1"}}
]}

Why Scheduler and ModuleManager are kept separate. The Scheduler stays a minimal execution engine that knows nothing about JSON, persistence, or module kinds. The ModuleManager is a Module like any other and uses the Scheduler's public add() / remove() API. This layering keeps the Scheduler trivially testable and makes "modules managing modules" a module-level concern. Whether to later consolidate is deferred until real workloads give us a reason.

Auto-setup guard — decision (R4 Sprint 9)

After the three-pass setup() completes, ModuleManager checks whether any EffectsLayer or DriverLayer was loaded. If neither is found (e.g. first boot on a clean device, or a corrupted state file), it calls instantiateDefaultPipeline_() to create a usable starting configuration.

Guard logic (ModuleManager::setup()): 

if (!inhibitState_) {          // tests set inhibitState_=true to skip this
    if (!hasDriver && !hasEffects)
        instantiateDefaultPipeline_();
}

Decision (Sprint 9): the guard is correct and unchanged by Sprint 2 dual-core work. The inhibitState_ flag is set only in tests (via disableStatePersistence()); production paths always run the guard. The guard was verified to still fire correctly in both "empty config" and "pipeline present" cases by tests in tests/test_module_manager.cpp (the auto-setup tests added in Sprint 4).

Dynamic Module Management

At runtime, modules can be added and removed without reflashing.

Ownership model: ModuleManager always owns all modules via unique_ptr. Runtime addition and removal go through ModuleManager — no module creates another module by directly calling new. Parent modules request child creation/destruction through the ModuleManager API.

state/modulemanager.json is the persistence layer. When a module is added or removed at runtime, the change is written immediately. The file is the on-disk view of the running configuration, not a read-once startup script.

Module Lifecycle

We adopt the Arduino-style lifecycle because it is simple, widely understood, and maps cleanly to both MCU and desktop runtimes.

Phase Called Purpose
construction Once, by the ModuleManager via TypeRegistry Default-constructible only — no parameters
control injection Once, after construction, before setup() ModuleManager loads persisted control values (or defaults)
wiring Once, after control injection, before setup() ModuleManager wires data-flow inputs (Layer references, etc.)
setup() Once, when the Module is loaded Allocate buffers, register UI description, subscribe to inputs, acquire resources
loop() Repeatedly, as long as the Module is active Do the work — produce output, read sensors, drive pixels
teardown() Once, when the Module is unloaded Release every resource acquired during setup()

Invariant: after teardown(), the Module has zero live allocations and zero registered callbacks. A Module that leaks on teardown is a bug, not a trade-off.

Why this lifecycle: - setup() / loop() is a mental model the target audience already knows (Arduino, PlatformIO, ESP-IDF's app_main pattern). - Explicit teardown() is the lesson learned from WLED-MM and MoonLight, where dynamic module removal was painful because cleanup was an afterthought. - A uniform lifecycle means the scheduler is trivially testable: create, setup, loop-N-times, teardown, assert no residue. - Hot-reload: Modules are loaded and unloaded at runtime without restarting. Memory is claimed in setup() and freed in teardown(), meaning only active Modules consume memory — a significant advantage on constrained devices. - Default-constructible is a hard rule: constructor arguments cannot be the source of runtime-configurable values, because at runtime those values come from a JSON config file and the UI, not source code.

Dispatch naming: loop vs runLoop — decision (R4 Sprint 9)

The Module base class uses a two-layer naming convention that was audited and confirmed in Sprint 9:

Layer Names Called by Implemented by
Implementation setup(), loop(), teardown(), loop20ms(), loop1s() Never directly — only through the dispatch layer Module authors (subclasses)
Dispatch runSetup(), runLoop(), runTeardown(), runLoop20ms(), runLoop1s() Scheduler StatefulModule (recurses into children); base default just forwards

Decision: keep as-is. The naming is consistent across Module, StatefulModule, and Scheduler. The run* prefix signals "this is a dispatcher, not the work itself". No renames are needed.

Naming the Unit: "Module"

This document uses Module. MoonLight had two distinct concepts — Modules (menu-accessible functionality) and Nodes (composable units executed on the hot path). projectMM unifies them under a single concept. The distinction between hot-path Modules and background Modules is a scheduling property, not a naming difference.

Module Controls and Data Flow

A Module's runtime-configurable state has two sources, kept distinct:

  • Controls are values a user can change — through the UI, REST, MQTT, or any other control surface. They have defaults, are persisted by the ModuleManager, survive restarts, and are described in the Module's JSON schema so the frontend can render them.
  • Data-flow inputs are values a Module receives from another Module — typically through a Layer. They are not user-editable; they are wired at load time.

Why the split matters. Mixing the two leads to the constructor-argument trap: "just pass it in." Constructor arguments cannot be the source of either kind of value, because at boot the runtime does not yet know which Modules will exist. Both controls and data-flow inputs must be settable on an already-constructed object. That is why Modules are default-constructible.

Lifecycle. The ModuleManager constructs each Module, then injects controls (from the persisted config or defaults), then wires data-flow inputs (resolving Layer references), then calls setup(). By the time setup() runs, every input the Module declared is in place.

Declaration. Controls and data-flow inputs are declared through two mechanisms on StatefulModule (src/core/StatefulModule.h):

Method Called by Module author overrides? Purpose
setProps(JsonObjectConst) ModuleManager, before setup() No — base class handles it Stashes all key-value pairs into pendingProps_; addControl() applies them at registration time
setInput(key, Module*) ModuleManager, before setup() Yes, to store the pointer Wire a named data-flow input (e.g. "layer" → EffectsLayer*); carries a Module*, not a scalar — cannot go through addControl()
loadState(JsonObjectConst) ModuleManager, before setup() No — base class handles it Merges persisted values into pendingProps_ (overwriting stashed props); addControl() applies them
saveState(JsonObject) ModuleManager, on teardown() No — base class handles it Iterates all registered ControlDescriptors and writes current field values automatically

The rule that replaces all three prop/state overrides: register a control before you use its field in setup().

// The new paradigm — no setProps / loadState / saveState overrides needed
void setup() override {
    addControl(speed_, "speed", "slider", 0, 255);  // pending prop/state applied here
    buildTable_();                                   // speed_ is now correct
}

addControl() drains the matching pending value (from props or saved state) into the backing field at registration time. After runSetup() completes, pendingProps_ is cleared to release memory.

Modules that have array-valued props (e.g. "start": [x, y, z]) or side-effect logic on load may still override setProps or loadState — they call StatefulModule::setProps(props) for the generic path and handle the special case themselves. This is the exception, not the rule.

Access. Inside loop(), controls are read through plain class fields (e.g. frequency_), not through JSON lookups. This keeps the hot path free of allocation and map traversal.

Control Declaration and Direct-pointer Access

Adapted from MoonLight's addControl / updateControl pattern, projectMM uses direct-pointer binding to eliminate per-tick JSON lookups:

  • addControl(variable, name, type, min, max) — called in setup(). Registers the control in the module's JSON schema and stores the memory address of the class variable alongside the control descriptor.
  • setControl(key, value) — called by the ModuleManager when a control change arrives. Looks up the stored pointer by key, type-dispatches, and writes the new value directly into the class field. Optionally calls onUpdate(key).
  • virtual onUpdate(const char* key) — override to react when a specific control changes (e.g. recompute a derived lookup table after frequency is updated). Default is a no-op.

Why direct-pointer binding: - loop() reads a plain float frequency_ — zero JSON overhead per tick. - Control metadata (label, min, max, type) lives only in the schema; it is not duplicated alongside the field. - saveState / loadState iterate the registered control descriptors automatically, so modules do not manage serialisation by hand.

Debounce and write coalescing. Slider drags produce many rapid setControl calls. Writing a JSON file per call would wear flash. The debounce timer lives in ModuleManager — after each setControl the ModuleManager starts (or resets) a short timer; the filesystem flush fires only when the timer expires without a new call.

defVal and the reset-to-default icon — decision (R4 Sprint 9)

Each control descriptor stores a defVal field captured by addControl() in setup(). This value is the C++ field initializer value at the moment setup() is called, which is before loadState() runs.

Decision: defVal always equals the C++ initializer, not the saved state. This means the ↺ button in the UI resets to the compiled-in default, not to the last-saved value. Rationale: the last-saved value is what the user chose; the default is what the module author specified. Resetting to "last saved" would be no-op for any slider that was already saved.

Invariant: defVal is set once by addControl() and never changes. setControl() and loadState() update the live field value and the value field in the schema, but never defVal. This invariant is verified by tests in tests/test_stateful_module.cpp (R4S9/A cases).

StatefulModule virtual interface — audit (Sprint 5, updated Sprint 7)

Every virtual on StatefulModule was audited against concrete module implementations to identify dead weight. Updated in Sprint 7 to reflect the new pendingProps_ paradigm:

Virtual Default Module authors override? Notes
setProps() stash into pendingProps_ No (exception: array-valued props or dimension side effects) Base stashes; addControl() applies. No concrete module needs this override.
setInput() no-op Yes — to store a Module* pointer Carries a module reference, not a scalar; cannot go through addControl(). Every wired module overrides this.
loadState() / saveState() stash / auto-iterate controls No (exception: load-time side effects) Base handles both directions automatically. No concrete module needs these overrides.
onUpdate() no-op Targeted use GridLayout (live resize), EffectsLayer (recompute dims)
onSizeChanged() no-op EffectsLayer (reallocates on driver resize) Fired by DriverLayer after it finalises its own buffer size
onChildrenReady() no-op DriverLayer (allocates after layout extent is known) Correct hook — called by runSetup() after all children finish
fillSystemJson() no-op SystemStatusModule only Contributes metrics to /api/system
isPermanent() false SystemStatusModule, ModuleManager Used by REST to block deletion
snapshot() false DriverLayer (binary pixel frame for WebSocket) Essential for WebGL preview
healthReport() empty SineEffect, PreviewModule, DriverLayer, WifiSta, WifiAp Drives automated testing
tags() "" none Reserved for add-module picker UI — keep, no overrides yet
dim() DIM_ANY none Reserved for dimension filtering in add-module UI — keep, no overrides yet

Sprint 7 conclusion: setProps, loadState, and saveState no longer need to be overridden by any concrete module. The base class handles all three for all modules via pendingProps_ and ControlDescriptor iteration. setInput remains the only lifecycle hook that module authors routinely override — it wires a Module* reference that cannot go through the scalar addControl() path.

Module Footprint and Scalability

The system is designed to host many modules simultaneously — a production light rig may have 20–100 active modules. Footprint decisions made at module count = 7 must still be correct at module count = 100.

Design targets per module (ESP32 classic as the tightest platform):

Resource Target Notes
Static sizeof ≤ 64 B (base) + owned buffers Base struct cost after lazy controls_ allocation.
controls_ side-table n_controls × 28 B (32-bit), allocated once in setup() Allocated lazily in steps of 4. A module with 2 controls costs 4 × 28 B = 112 B.
Flash per module ≤ 2 KB typical Effect module: ~1–2 KB. System module: ~1–3 KB.
Hot-path allocation 0 B Strictly enforced.

Rules:

  1. Start small, stay small. Every new module is measured at implementation time via classSize() (static) and heapSize() (dynamic). SineEffectModule (28 B) and BrightnessModifierModule (24 B) are the reference points.
  2. Lazy allocation over embedded arrays. Fixed embedded arrays in structs are forbidden unless the size is strictly 1. The controls_ refactor (Release 2 Sprint 1c) is the canonical example: 16-slot embedded array → heap pointer grown on demand.
  3. Control count is a footprint cost. Each addControl() allocates one ControlDescriptor (28 B on 32-bit). Keep effect and modifier modules to ≤ 4 controls where possible.
  4. String buffers belong in PROGMEM or are shared. Use const char* pointing to a string literal for labels and category strings.
  5. No hot-path allocations, ever. loop() runs up to 50+ Hz. Allocate everything in setup(); free in teardown().
  6. Footprint is measured, not estimated. Every sprint that adds or changes modules records the footprint impact in the sprint's Result section. CI enforces this via the ESP32 footprint report on every PR.

Scaling checkpoints:

Module count Action
> 10 Audit cumulative controls_ heap via printSizes()
> 20 Profile WebSocket push cost — getModulesJson() iterates all modules
> 50 Consider paging or lazy-loading in the frontend

Module Hierarchy and Master-Detail

Modules are not all equal peers. Some modules own others: a EffectsLayer owns its effect modules, a DriverLayer owns its drivers, a NetworkManager owns transport modules. This is the master-detail pattern carried forward from MoonLight.

Design decisions:

  • ModuleManager is the sole owner. Even child modules live in ModuleManager's owned_ list. A parent module never holds a unique_ptr to a child — it holds the child's id string and requests lifecycle operations through the ModuleManager API.
  • Parent modules request, ModuleManager executes. A parent calls mm.addModule(type, id, props) to create a child and mm.removeModule(id) to destroy it.
  • Permanence is per-module, not per-category. SystemStatusModule is permanently required on every platform. WiFiModule is only relevant on hardware that has WiFi. A module declares its own permanent flag.
  • Child lifecycle follows parent lifecycle. When a parent module is torn down, it removes its children from ModuleManager before its own teardown completes.

Illustrative hierarchy:

SystemStatusModule          (permanent on all platforms)
NetworkModule       (present only on WiFi-capable hardware)
  └─ WifiStaModule        (child: station connection)
  └─ WifiApModule         (child: access point)
EffectsLayer "producer1" (parent, application)
  └─ SineEffectModule "sine1"
DriverLayer "consumer1" (parent, application)
  └─ PreviewModule "preview1"

Layers — EffectsLayer and DriverLayer

A Layer is a Module whose job is to own a channels array and mediate bulk data between other Modules. Effects never hold pointers to each other and never allocate their own pixel buffers; they write into, or read from, a Layer.

  • EffectsLayer — the destination for an effect's output. Owns the pixel buffer. Multiple EffectsLayers can coexist.
  • DriverLayer — the source for a driver or preview module. Owns its own channels array and, during its loop(), blends the channels of all active EffectsLayers into it.

Why blending lives on the DriverLayer. It keeps the Scheduler agnostic (no special "blend step" phase) and co-locates the blending logic with the layer that owns the output buffer.

Domain scope. Layers are a lights-domain concept. RGB and Channel live in the Layer module's files, not in src/core/. A future audio or sensor domain introduces its own equivalent.

Double-buffer layout. The buffer shared between producers and consumers is dynamically sized at startup, based on the active configuration. Dynamic sizing lets small devices keep as much heap free as possible.

Concurrency Model

projectMM adopts a producer/consumer model for the hot path:

  • Producers (effect Modules) run their loop() on one core/thread and write into a shared output buffer.
  • Consumers (driver Modules) flush that buffer to real-world I/O on another core/thread.
  • A double buffer between producers and consumers lets both sides run in parallel without blocking.

Multi-core Strategy: MoonLight model vs. advanced model

MoonLight's approach (proven, simpler): all producer Modules run on Core 1; all consumer Modules run on Core 0. The atomic Channel* hand-off between EffectsLayer and DriverLayer provides the synchronisation point.

Advanced model (our target): per-Module "core" assignment in state/modulemanager.json — any Module can be pinned to any core. The Scheduler already stores a cores_ vector alongside modules_; dispatch via xTaskCreatePinnedToCore is the next step.

Current state (Release 2, Sprint 4): "core" is parsed and stored but not yet dispatched — all Modules run on Core 1. See Release 2 backlog.

On PC and rPi, std::thread with one thread per "core" group replaces xTaskCreatePinnedToCore.

ProducerModule / ConsumerModule

Decision history: Sprint 3b dropped ProducerModule and ConsumerModule as inheritance layers for the LED pipeline. Sprint 5 confirmed that decision. Sprint 9 reconfirmed it. Sprint R4S11 added them as opt-in base classes for new domain-specific consumers.

Why EffectsLayer and DriverLayer do NOT extend them

  • EffectsLayer owns a double-buffered Channel object, not a simple void* buffer. Wrapping it behind bufferPtr() would lose the typed atomic handoff semantics that make dual-core safe.
  • DriverLayer has up to 8 EffectsLayer* sources, not a single producer_. The ConsumerModule single-producer contract does not fit.
  • No existing test or runtime behaviour changes.

What they are (R4S11)

Thin base classes in src/core/ for new domain-specific modules — primarily library consumers such as FastLED-MM.

// src/core/ProducerModule.h
class ProducerModule : public StatefulModule {
public:
    void declareBuffer(void* buf, size_t len, size_t elemSize);
    void*  bufferPtr()      const;   // pointer to pixel data
    size_t bufferLen()      const;   // number of elements
    size_t bufferElemSize() const;   // bytes per element
    size_t bufferBytes()    const;   // total bytes
};

// src/core/ConsumerModule.h
class ConsumerModule : public StatefulModule {
public:
    void setInput(const char* key, Module* m) override;  // key "producer"
protected:
    ProducerModule* producer_ = nullptr;
};

A FastLED-MM effect subclasses ProducerModule and calls declareBuffer(flm_leds, N, sizeof(CRGB)) in setup(). Its driver subclasses ConsumerModule and reads producer_->bufferPtr() (or the shared array directly).

Boundary rule

src/core/ is the stable public API. No LED-specific types (RGB, Channel, EffectsLayer) belong there. ProducerModule and ConsumerModule use void* intentionally to stay domain-agnostic.

Module Ordering and Connections

Modules execute in a defined order each tick. The initial model is a sequential ordered list — the same approach used in MoonLight. Each Module may additionally declare successors (Modules that must run after it) and/or predecessors. The scheduler uses these to refine the sequential order into a valid execution sequence.

Inter-module Communication

Modules exchange two kinds of data:

  • Hot-path bulk data (pixel buffers, audio frames) — passed through Layer modules (see above).
  • Generic typed values (control parameters, shared scalars) — passed through a typed key/value table with fixed-size slots. Small and predictable enough for the hot path; no dynamic allocation on the hot path itself.

State Persistence

Module state must survive restarts on all platforms and must be observable by other parts of the system.

  • Filesystem: LittleFS on ESP32; standard filesystem paths on rPi and PC. One JSON document per Module at state/<id>.json.
  • Ownership. Each Module owns its state blob. The runtime (via ModuleManager) loads the blob before setup(), persists changes to disk, and notifies subscribers. Modules never touch the filesystem directly for their own state.
  • State change notification: a lightweight publish/subscribe mechanism propagates changes from any source (UI, REST, MQTT, another Module) to all subscribers. The prior implementation in MoonLight relied on ESP32-sveltekit's StatefulService — correct but resource-heavy. A lighter-weight replacement is an open design task.
  • Write coalescing. Rapid mutations (slider drags) produce one filesystem write after the debounce timer expires, not one per mutation.

Hot Path

The hot path is the sequence executed every scheduler tick:

  1. Run loop() for each active Module, in a defined order.
  2. Flush output drivers (push pixel buffers to LED strips or network light controllers).

The hot path is measured and budgeted. A Module that blows its budget is a bug, not a feature request.

Tick rate and budget

Default policy: run as fast as possible. The working target is roughly 20 ms per frame (50 Hz) as the threshold at which lights look smooth to the eye. There is no enforced per-Module time slice.

Measurement. Each Module's loop() is timed with esp_timer_get_time() on ESP-IDF and std::chrono on PC. The scheduler accumulates min/max/average per Module per second and publishes these as part of each Module's state JSON.

Frontend

JSON-driven UI

Each Module publishes a JSON document describing its controls. The frontend renders that document into a UI. No Module ships hand-written HTML.

Why JSON-driven: - One frontend, many Modules. Adding a Module does not require frontend code. - Same frontend, many clients. The same JSON can drive a web UI, a mobile UI, or a CLI. - Platform independence. A Module's UI works the same on ESP32, rPi, or PC. - Small footprint. One generic interpreter beats N hand-coded forms for flash usage on MCU.

Schema v0.1. Module identity fields: id, name, category ("effect", "modifier", "layout", "driver", "system"). Control types: slider (min/max/default), toggle, select. The schema is versioned from day one (schema: "0.1") — renaming a control is a breaking change; adding a new field is not.

Frontend Technology

Decision (confirmed, Release 1 Sprint 9): plain HTML/CSS/JS, no framework, no build toolchain dependency.

Rationale: - Frameworks (React, Svelte, Vue) solve the problem of synchronizing state with a hand-written DOM. If the DOM is generated from JSON, most of that problem evaporates. - MoonLight's experience: the ESP32-sveltekit dependency — not Svelte itself — consumed most of a classic ESP32's flash and a significant share of heap. The framework stack as a whole was the problem, not the reactive model. - A framework-free frontend is easier to serve from an MCU, easier to bundle into an installer on PC, and easier for a contributor to modify without learning a build toolchain.

Serving model. The frontend (HTML/CSS/JS) is compiled into the binary on all platforms rather than served from the filesystem. A build step converts src/frontend/ into a C++ header (src/frontend/frontend_bundle.h) containing the content as a string literal or PROGMEM array. The HTTP server handler returns this in-memory bundle on requests to /.

Why embedded, not filesystem: - On ESP32, flash (PROGMEM) is the natural home for static assets — LittleFS is limited and may be absent on some variants. - On PC/rPi, embedding avoids a runtime file path dependency and makes the binary self-contained. - One consistent approach across all platforms minimises PAL surface.

WebSocket Bandwidth Management

The server pushes state and pixel frames continuously (~50 Hz). When the browser tab is in the background, frames pile up and waste bandwidth on both sides.

Decision: the frontend pauses processing when the page is hidden and resumes when it becomes visible again, using the Page Visibility API. The server does not need to know about client visibility — the client discards frames while hidden. On Safari, the pageshow event handles the back-forward cache (bfcache) case where pages are restored without re-firing DOMContentLoaded.

A 25-second heartbeat ping keeps the socket alive and prevents idle-timeout disconnections on mobile carriers.

Pixel Data Transport

For bulk pixel data (DriverLayer → frontend WebGL previewer), the runtime sends binary WebSocket frames:

Offset Size Content
0 1 byte Frame type (0x01 = pixel frame)
1 2 bytes Width (little-endian uint16)
3 2 bytes Height (little-endian uint16)
5 width × height × 3 bytes RGB pixels, row-major

At 16×16 = 256 pixels the frame is 773 bytes. Rationale: JSON encoding of the same data is 3–5× larger on the wire and requires parsing before upload to a WebGL texture. A typed binary frame maps directly to a Uint8Array → gl.texImage2D call in JS with no intermediate conversion.

HTTP Server and WebSocket

Library choice is per-platform:

ESP32: ESPAsyncWebServer — callback-based, handles concurrent connections without blocking the hot path, proven in the MoonLight ecosystem.

PC / rPi: cpp-httplib — a single-header C++ HTTP/WebSocket library chosen over Crow for these reasons:

Criterion cpp-httplib Crow
Distribution Single header Multi-file source tree
WebSocket Built-in since v0.14 Requires separate library
Callback style Callback-based — close to ESPAsyncWebServer Macro-based routing
Build deps None Boost (optional but common)

Principle: matching callback styles across platforms means HTTP handler code is portable — only the server bootstrap differs behind the PAL.

Platform Abstraction Layer

The PAL mediates two independent axes:

  • Framework axis (ESP32 only): ESP-IDF vs Arduino-on-ESP-IDF.
  • Platform axis: ESP32 vs Raspberry Pi vs PC.

These are kept decoupled so that swapping the ESP32 framework does not touch rPi or PC code, and vice versa.

Minimal API surface (provisional): - GPIO read/write - High-resolution timers - Filesystem (read/write/list, LittleFS on ESP32, POSIX elsewhere) - Network sockets and discovery - Thread/task creation and pinning - Light output drivers (GPIO-based and network: Art-NET, DDP, E1.31)

Testing benefit. The PAL doubles as a testing seam. Platform-specific calls through the PAL can be faked in tests on PC, independent of ESP32/rPi hardware.

ESP-IDF vs Arduino Framework

Status: open. Favor ESP-IDF, particularly for newer boards (ESP32-P4) where ESP-IDF exposes hardware capabilities not available through the Arduino layer. The key enabler is the PAL — keeping Modules portable so the underlying framework can change without rewriting Module code. The decision rests primarily on whether the light driver story works without the Arduino framework (FastLED compatibility is the biggest question).

Third-party Libraries

Library Used for Notes
FastLED LED / light driving Arduino-oriented; see ESP-IDF question above
ArduinoJson JSON parsing and serialization Works across Arduino and ESP-IDF; low footprint
ESPAsyncWebServer HTTP/WebSocket server on ESP32 Well-known but has a history of fragility; alternatives worth evaluating
cpp-httplib HTTP/WebSocket server on PC/rPi Single-header; no build deps
doctest Unit/integration tests Single-header; runs on PC, rPi, and ESP32

Principle: any library that pulls a large flash or heap cost must earn its place by measurement, not by familiarity.

Platform Targets

Platform Role Notes
PC (Linux/macOS/Windows) Primary development and iteration target Fast build/test loop; no flashing
Raspberry Pi Production target when ESP32 CPU is insufficient Same binary shape as PC where possible
ESP32 classic (D0WDQ5) Tightest embedded target — budget baseline 320 KB heap, 1.3 MB app flash. Every feature is costed against this.
ESP32-S3 (N16R8) Primary production embedded target PSRAM support, USB-OTG, faster CPU.
ESP32-P4 High-performance embedded target Dual-core Xtensa LX9 @ 400 MHz, hardware JPEG/H264. Requires ESP-IDF 5.x.

Classic ESP32 as the budget floor. Every new feature is measured against the classic ESP32's budget. A feature that fits on classic fits everywhere.

Control Interfaces

External tools control a projectMM instance through:

  • REST for request/response control and introspection. See api.md for the full reference.
  • MQTT for pub/sub, including inter-instance coordination. Planned.
  • WebSocket for the frontend's live state stream.

Modules do not talk to REST or MQTT directly. They publish JSON state and accept JSON control messages through the runtime. This keeps transport and business logic separable.

State Files

State files live at state/<id>.json alongside the running binary (or in LittleFS root on ESP32). The state/ directory is created automatically on first write. state/modulemanager.json is the persistent configuration listing which modules exist.

Inter-instance Synchronization (Supersync)

Architectural placeholder — detailed design deferred.

Multiple projectMM instances can discover each other and coordinate at several levels:

  • Mirror — one instance passively follows another's output.
  • Control — one instance issues control changes to others.
  • Group control — one instance controls a group as a unit.
  • Shared processing — instances divide work across a synchronized clock domain.

Open questions: discovery (mDNS, MQTT broker, explicit config?), clock sync (PTP, NTP?), failure handling, security.

Integration with Home Automation Ecosystems

projectMM is a runtime, not a home-automation hub. Integration happens at the Control Interfaces layer:

Ecosystem Integration path Status
Home Assistant MQTT (native HA discovery topics) and/or REST endpoints Target — first integration
Matter A Matter bridge Module exposing projectMM Modules as Matter endpoints Exploratory
Zigbee Via MQTT only (companion bridge e.g. Zigbee2MQTT) Via MQTT only
Art-NET / DDP / E1.31 First-class light output protocols via driver Modules Core scope

Principle: protocols live in Modules or in Control Interfaces, never in Module business logic.

Project Primary domain Overlap with projectMM Where they differ
projectMM Light control (entry), generic loop-driven processes — Cross-platform peer targets from day one; runtime is domain-agnostic
Tasmota Home automation firmware JSON / MQTT control surface MCU-only; firmware-per-device model
WLED / WLED-MM LED effects on ESP32 Effect Modules, JSON UI LED-specific; MCU-only
MoonLight LED effects + Nodes runtime Node/Module concept, schema-driven UI — direct predecessor MCU-only; bound to ESP32-sveltekit
ESPHome Home automation firmware Device-side modularity YAML-generated build, not a runtime Module system

The gap we address: a runtime whose Module shape is generic enough to cover lights, audio, and sensors, with PC, rPi, and ESP32 as peer targets from commit #1.

Anti-Debt as a Design Constraint

Design decisions are evaluated not only on what they enable but on what they prevent:

  • No implicit conventions. If a contributor needs to know something, it is in a document or in a comment on the thing itself.
  • No dead code paths. Features that "might be useful later" are not added until needed.
  • No frameworks we do not control the budget of. A dependency that dictates our flash/heap ceiling is rejected unless we have an exit strategy.
  • No monolithic internals. Each component is replaceable in isolation.

Open Architectural Questions

  • Richer module dependency graph. The successors/predecessors table covers the minimum viable case. Whether we need a full dependency graph with implicit data-flow ordering is an open question.
  • Hot path budget enforcement. Log-only is the default. Whether to add skip-on-overrun, soft caps, or hard caps depends on whether real workloads produce misbehaving Modules we cannot fix at the source.
  • State pub/sub implementation. Parked until we hit real constraints — see Backlog.
  • ESP-IDF vs Arduino — see above.
  • Simulator coverage — how far can Wokwi/WASM take us?
  • Hardware-in-the-loop — what does an affordable, reliable setup look like?