Skip to content

Release 2 — Observable, Hierarchical, and Network-Ready

Theme: Release 1 proved the concept. Release 2 makes it operable — observable at runtime, manageable through a module hierarchy, configurable over the network without reflashing, and polished enough to use daily.

Release Overview

Gap from Release 1 Addressed by
No runtime visibility into a running system Sprints 1–2 (SystemStatusModule, log levels, live control values)
Tests stopped at the unit-test binary Sprints 3–4 (in-process HTTP/WS test harness, JSON reporter, CI gates)
No way to add/remove modules without reflashing Sprint 5 (REST add/remove, TypeRegistry, config round-trip)
Flat module list; no parent-child relationship Sprints 6–7 (parent-child data model, parent-driven lifecycle, UI tree)
WiFi credentials baked into firmware Sprint 8 (NetworkModule hierarchy, credentials from UI, sensitive controls)
Functional but unpolished UI; grayscale 1D effects Sprint 9 (progress bars, nav, status bar, RGB 2D effects, enabled toggle)
Overlapping doc structure; no end-user path Sprint 10 (doc restructure: user-guide, developer-guide, architecture merge)

Backlog items closed: health check shared scratch buffer, dirty-flag debounce, live control values on page load, KvStore spinlock for dual-core safety.

Sprint 1 — Runtime Observability

Goal: make a running system legible — heap usage, fps, uptime, and structured log output for automated testing.

  • SystemStatusModule samples heap, fps, uptime, and platform info each loop(), exposing all as display controls live-updated via WebSocket. Replaces ad-hoc MemoryStats calls in main.cpp.
  • display uiType: read-only green text span; generic frontend renderer; no per-module HTML.
  • heapSize() virtual on StatefulModule — modules that allocate in setup() override it; printSizes() shows both static (classSize()) and dynamic columns.
  • Log levels (SETUP, HEALTH, TIMING, STATE, DEBUG); --count N mode defaults to HEALTH+TIMING only, making CI output immediately parseable.
  • Health-check shared scratch buffer: char[64] moved from each module into a single Scheduler-owned healthScratch_ — ~64 B saved per module.
Metric Value
Tests 99 total, +14 new, 323 assertions
PC fps ~266 K (5 000 ticks)
ESP32 fps ~139
ESP32 RAM 11.3% (+64 B)
ESP32 Flash 51.9% (+2 708 B)

Retrospective: display type required ~15 frontend lines. The shared scratch buffer eliminates per-module heap for health reporting with no API change. healthReport(char*, size_t) (output buffer) was the right signature over returning const char*.

Sprint 2 — State, Controls, and Persistence

Goal: prove the live-value path works, tighten the debounce window, and validate the control schema.

  • GET /api/modules reads live in-memory field values via pointer dereference — confirmed correct by a new test (mutate, read back, assert value without file round-trip).
  • Debounce raised to 2 000 ms; setDebounceMs(0) seam for tests (synchronous flush, no sleep_for).
  • validateControls test: all slider controls have min < max — confirmed consistently followed even before the test existed.
  • pioarduino (ESP-IDF 5.x) adopted as baseline — required __has_include guards for PSRAM detection and explicit esp_chip_info.h include.
Metric Value
Tests 102 total, +3 new, 356 assertions
PC fps ~175 K
ESP32 fps ~145
ESP32 RAM 11.3% (+8 B)
ESP32 Flash 52.2% (+160 B)

Note: pioarduino increased ESP32 flash from 52.2% to 82.2% (+394 KB framework overhead) — application code unchanged.

Sprint 3 — Integration and Runtime Testing

Goal: extend the test suite past the process boundary — start a server, send HTTP and WebSocket requests, assert correctness.

  • In-process TestServer (not subprocess + libcurl): HttpServer + WsServer + Scheduler started in background threads inside the tests binary. Simpler, faster, directly observable — no subprocess lifetime races.
  • JsonReporter: custom doctest reporter writing build-logs/test-results.json on every run (source of truth for CI and test dashboards).
  • WsTestClient: minimal RFC 6455 client (POSIX sockets, select() timeout) — skips binary frames on readText(), skips text frames on readBinary(), decoupling tests from broadcast interleaving.
  • Slider hang root cause (Part D): document.activeElement !== input is unreliable during pointer-drag on macOS/Chrome. Fixed with dragTs map — suppress WS updates for 1 s after last input event.
  • WS cold-start root cause (Part E): on ESP32, ws.begin() was called after server.begin(), leaving a window where HTTP accepted connections before the WS endpoint existed. Fixed by swapping order. Client-side backoff (1 s → 2 s → 4 s → 5 s) adds defence in depth.
  • Thread safety fix: WsServer and HttpServer background threads were detached — caused heap corruption SIGABRT when they accessed this after destruction. Fixed by storing joinable std::thread members and joining in destructors. Rule: any background thread holding this must be joined before the owning object is destroyed.
Metric Value
Tests 113 total, +11 new, 431 assertions
PC fps ~287 K (5 000 ticks)
ESP32 fps ~143 steady
ESP32 RAM 14.2% (+0 B)
ESP32 Flash 82.3% (+960 B)

Sprint 4 — Multi-core Execution

Goal: concurrency hardening, binary WS frame regression test, CI failure gate, and platform_version display.

  • "core" field parsed from modules.json and stored in Scheduler::cores_ — infrastructure for FreeRTOS xTaskCreatePinnedToCore dispatch (deferred; stored but not yet dispatched).
  • KvStore spinlock: std::atomic_flag RAII SpinLock protecting all set* calls; get* unsynchronised (safe for single-reader hot path).
  • ModuleManager mutex: std::mutex controlMutex_ guards setModuleControl / getStateJson / getModulesJson; hot path (loop()) does not hold the lock. dirty_ / dirtyAtMs_ changed to std::atomic.
  • platform_version display control in SystemStatusModule: "arduino-esp32 MAJOR.MINOR.PATCH (env)" on ESP32, "PC (PC)" on PC. Use ESP_ARDUINO_VERSION_MAJOR/MINOR/PATCH — not ARDUINO_ESP32_RELEASE (undefined in pioarduino).
  • CI gate: --no-header -m fail added to doctest invocation; failing test now fails the CI job.
  • readBinary() timeout bug: duration_cast<seconds> truncates sub-second remainders to zero. Fixed with ceiling millisecond division: (remMs + 999) / 1000. Rule: when tracking a deadline across iterations, keep milliseconds internally and convert with ceiling.
  • addFilter("reporters", ...) replace-not-append: two calls replaces instead of appending. Fixed with "console,json" as a single comma-separated call.
Metric Value
Tests 114 total, +1 new, 443 assertions
PC fps ~131 K (mutex contention in WS broadcast path — not a production bottleneck)
ESP32 fps ~260
ESP32 RAM 14.2% (+24 B)
ESP32 Flash 82.4% (+1 688 B)

Sprint 5 — Dynamic Module Management

Goal: add and remove modules at runtime from the UI without reflashing.

  • ModuleManager becomes a StatefulModule subclass (isPermanent() = true; module_count display control; appears first in /api/modules). Config moves from modules.json to state/modulemanager.json — consistent with every other module. Fallback to modules.json on first boot.
  • addModule(type, id, props) — three-pass setup (create → wire inputs → load state → add to Scheduler); appends to state/modulemanager.json.
  • removeModule(id) — teardown, save state, remove from Scheduler and owned_, remove from state/modulemanager.json; rejects isPermanent() modules with 403.
  • POST /api/modules and DELETE /api/modules/{id} REST endpoints.
  • Thread safety: schedulerMutex_ (outer) + controlMutex_ (inner); HTTP handler blocks for at most one tick duration. Lock ordering must be followed everywhere — violation will deadlock.
Metric Value
Tests 121 total, +7 new, 467 assertions
PC fps ~450 K
ESP32 fps ~264
ESP32 RAM 14.2% (+24 B)
ESP32 Flash 83.2% (+9 900 B)

Retrospective: "Everything is a module" paid off — ModuleManager appearing in the REST schema and WebSocket broadcast required zero special-casing. The modules.json → state/modulemanager.json migration was transparent: first boot seeds the new file on teardown.

Sprint 6 — Parent-Child Module Hierarchy

Goal: modules can contain other modules; the UI renders the tree; REST is hierarchy-aware.

  • parent_id field in ModuleManager::Entry (empty = root); serialised to state/modulemanager.json and returned by GET /api/modules.
  • POST /api/modules accepts optional "parent_id". DELETE /api/modules/{id} rejects with 409 if the module has children.
  • Frontend: buildTree() / buildCard() recursive rendering — child cards inside parent cards, up to three levels. Add-child button and per-card delete button.
  • GET /api/types returns all registered type names from TypeRegistry.
  • getTypesJson dangling-pointer bug: passing t.c_str() from a local std::vector<std::string> into ArduinoJson stores a reference into a destroyed object. Fixed by passing t (std::string) so ArduinoJson copies into its pool. ArduinoJson add(const char*) stores a reference — never pass a pointer to a temporary.
Metric Value
Tests 127 total, +6 new, 491 assertions
PC fps ~138 K
ESP32 fps — (not flashed)

Sprint 7 — Parent-Driven Loop, Child Lifecycle, and WebSocket Reliability

Goal: make the hierarchy live — parent drives children's loop(); WebSocket is reliable across page refreshes and tab focus/blur.

  • StatefulModule gains children_ (lazy grow-by-4 array, same pattern as controls_) and addChild(Module*).
  • runSetup() delegates to children parent-first; runTeardown() delegates children-first (deepest first). Children release resource references before parent frees underlying resources.
  • ModuleManager 5-pass instantiateFromArray: Pass 4 wires addChild, Pass 5 registers only root modules with the Scheduler. Children are driven by their parent; Scheduler timing table shows roots only.
  • Runtime addModule/removeModule updated for the child path.
  • WS alternating-F5 bug: ESPAsyncWebServer did not clean up stale WS client slots on disconnect. Fixed by calling cleanupClients(0) immediately in WS_EVT_DISCONNECT.
  • visibilitychange listener: when tab is hidden, pauses DOM updates (accepts messages, skips rendering); on show, reconnects if needed and calls loadModules().
  • timingFor(const Module*) pointer-based overload replaces index-based lookup — children return nullptr, making the root/child distinction explicit in the API.
Metric Value
Tests 133 total, +6 new, 533 assertions
PC fps ~391 K
ESP32 fps ~170
ESP32 RAM 14.2% (+8 B)
ESP32 Flash 84.2% (+13 684 B)

Retrospective: "Universal child delegation" (all StatefulModule types, not just layer types) required zero changes to existing concrete modules. The teardown ordering constraint (children-first) was the natural correct choice once framed. Lazy children_ allocation mirrors controls_ — childless modules pay 6 bytes and zero heap.

Sprint 8 — Network

Goal: full network stack as a first-class module hierarchy; WiFi credentials configurable from the UI without reflashing.

NetworkModule       ← device name, MAC, owns children
  ├─ WifiStaModule  ← STA: connects to router; non-blocking; reconnects on credential change
  ├─ WifiApModule   ← AP:  creates hotspot for bootstrap (open by default); simultaneous STA+AP
  └─ EthernetModule ← placeholder ("unsupported" on classic ESP32)
  • NetworkModule parent: device_name (editable, persisted, default "MM" + last4(MAC)), mac_address (read-only).
  • WifiStaModule: ssid, password (sensitive), ip_address (display), signal_dbm (slider). Non-blocking connect via pendingConnect_ flag so HTTP/WS stay alive during credential entry. onUpdate("ssid"/"password") triggers immediate reconnect.
  • WifiApModule: reads device_name from parent via setInput("network").
  • Sensitive control flag (sensitive: true): renders as <input type="password">; excluded from WebSocket state pushes; excluded from getModulesJson() response values; written to state files.
  • WiFi.macAddress() returns zeros before WiFi connect. Fixed with esp_efuse_mac_get_default() (reads burned-in eFuse MAC immediately at any point in firmware execution).
  • EditStr control type added to StatefulModule for string fields.
Metric Value
Tests 161 total, +28 new, 587 assertions
PC fps ~278 K
ESP32 fps ~278
ESP32 RAM 14.2% (−40 B)
ESP32 Flash 85.1% (+11 624 B)

Sprint 9 — UI Polish + Network UX

Goal: noticeable UI polish and complete network configuration flow.

  • progress control type: <progress> bar with raw value alongside; used in SystemStatusModule for RAM/FS/Flash/PSRAM.
  • button control type: <button> POSTs {value: true} to trigger one-shot actions. SystemStatusModule reboot button calls ESP.restart() on ESP32.
  • String display controls now render as text, not NaN (frontend detects non-numeric value).
  • uint8_t control overload (CtrlType::Uint8): avoids float↔int conversion in hot path; SineEffectModule and BrightnessModifierModule migrated. KvStore brightness normalised to 0–1: amplitude_ / 255.0f.
  • Fixed status bar: device name (from WS state), WS indicator, reconnect button, hamburger nav. Selected module persisted via localStorage. Device name also set as browser tab document.title.
  • Safari WS reliability: keep socket alive on tab hide (was closing and reopening); pageshow listener for bfcache restoration (does not fire DOMContentLoaded); 25-second heartbeat ping.
  • Vivid RGB 2D effects: three-channel sine at 120° phase offsets; true 2D formula (R varies with x, G with y, B with diagonal) — each pixel unique from its 2D position.
  • Per-module enabled toggle: added in runSetup() after setup() (so clearControls() doesn't wipe it); runLoop() skips loop() when false; saveBaseState/loadBaseState persist it. Zero changes to any concrete module.
  • Integration test state contamination fixed: disableStatePersistence() on ModuleManager routes all state I/O to /dev/null in tests.
Metric Value
Tests 176 total, +15 new, 611 assertions
PC fps ~258 K (5 000 ticks)
ESP32 fps ~92
ESP32 RAM 40.6% (131.9 KB / 324.6 KB)
ESP32 Flash 85.8% (1 124 KB / 1 280 KB)

Sprint 10 — Documentation Restructure

Goal: one concept, one home; end-user path that does not require reading architecture docs.

Previous structure had five overlapping top-level files (architecture.md, design.md, implementation.md, build.md, standards-and-guidelines.md) with no user-facing content.

New structure: 

user-guide/          getting-started, ui walkthrough, module overview
developer-guide/     architecture (merged arch+design+impl concepts), api, build, standards, add-a-module
development/         process, backlog, release history (unchanged)
modules/             per-module detail pages (unchanged)

design.md and implementation.md retired as standalone files — content re-homed into developer-guide/architecture.md and developer-guide/api.md. All existing anchors preserved via stub files.

Metric Value
Tests 176 total, +0 (docs-only sprint)
mkdocs build 1 known INFO line (LICENSE); zero anchor warnings

Result: 13 new/updated doc pages; all 13 DoD items complete; mkdocs build clean.