Developer Guide — Deploy¶

How to build, run, flash, and test projectMM on PC and ESP32. Architecture and design rationale is in architecture.md; the development history is in ../development/.

Overview¶

All build, flash, test, and summary steps live in deploy/ as cross-platform Python scripts. No shell scripts, no Unix-only tools — the same commands work on macOS, Linux, and Windows.

deploy/
  _lib.py          — shared helpers: device selection, pio_paths(), load/save devicelist
  all.py           — full pipeline: build + unittest + flash + run + livetest + summarise
  build.py         — CMake (PC) or PlatformIO (ESP32) build
  unittest.py      — run doctest suite (PC only)
  flash.py         — flash firmware to ESP32 devices
  flashfs.py       — build and flash LittleFS filesystem
  run.py           — verify devices are responding (/api/system or serial)
  livetest.py      — run live REST tests against all matched devices
  live_suite.py    — REST test library and standalone test runner
  summarise.py     — parse all logs → docs/status/deploy-summary.md
  devicelist.py    — scan USB ports and update devicelist.json
  wifi.py          — manage WiFi credentials in data/state/sta1.json

  build/           — build outputs (gitignored)
    pc/            — CMake build tree and binaries
    esp32/<env>/   — PlatformIO artifacts per environment
  flash/           — per-device flash logs
  run/             — per-device run logs
  live/            — per-device live test logs and JSON results
  test/            — unit test log and results JSON

Requirements¶

PC (macOS / Linux / Windows)¶

CMake ≥ 3.16
A C++17-capable compiler (Clang, GCC, or MSVC)
Python 3.10+ (for deploy/ scripts)
Git (used by CMake to download doctest at configure time)

ESP32¶

PlatformIO Core CLI — self-managed penv at ~/.platformio/penv/; requires Python 3.10–3.13 (see below)
USB cable to flash

First-time setup¶

Git hooks are installed automatically on the first build:

PC: cmake -B deploy/build/pc (the configure step) installs the hooks.
ESP32: platformio run installs the hooks via scripts/install_hooks.py.

To install hooks without building (e.g. documentation-only work):

git config core.hooksPath .githooks

Device registry — `deploy/devicelist.json`¶

deploy/devicelist.json (gitignored) is a JSON array listing all known devices. A template is committed; copy it to get started:

cp deploy/devicelist.json.template deploy/devicelist.json

Key fields per entry:

Field	Meaning
`type`	`"pc"` or `"esp32"`
`device_name`	Human label (e.g. `"MM-70BC"`)
`ip`	WiFi/network IP (ESP32); `"localhost"` for PC
`port`	Serial port path (ESP32 only)
`mac`	MAC address (ESP32 only)
`env`	PlatformIO environment name (ESP32 only)
`test`	`true` = include in default `all.py` runs
`firmware`, `last_seen`	Updated automatically by `devicelist.py`

Scan USB serial ports and update the registry automatically:

python3 deploy/devicelist.py             # probe all ports, update last_seen
python3 deploy/devicelist.py --reset     # DTR-reset each device to capture hello line
python3 deploy/devicelist.py --timeout 10

Boards running projectMM are identified by the projectMM hello device=MM-XXXX serial line and promoted to firmware: "projectmm". Unknown boards are recorded as inventory.

Device selection¶

All deploy scripts accept -<field> <value> filters to select a subset of devices from devicelist.json:

python3 deploy/flash.py -type esp32              # all test:true ESP32 devices
python3 deploy/livetest.py -ip 192.168.1.234     # single device by IP
python3 deploy/livetest.py -env esp32s3_n16r8    # all test:true devices of this env
python3 deploy/run.py -device_name MM-70BC       # single device by name

Multiple flags AND together. No flags = all test:true devices.

Pipeline scripts¶

Script	What it does	Output
`deploy/build.py -target pc`	CMake configure + build	`deploy/build/pc/`
`deploy/build.py -target <env>`	PlatformIO build	`deploy/build/esp32/<env>/`
`deploy/unittest.py`	Run doctest suite	`deploy/test/`
`deploy/flash.py [-<field> <value>]`	Flash firmware to ESP32(s)	`deploy/flash/`
`deploy/flashfs.py [--wifi]`	Build + flash LittleFS FS	`deploy/flash/`
`deploy/run.py [-<field> <value>]`	Verify devices responding	`deploy/run/`
`deploy/livetest.py [-<field> <value>]`	Live REST tests (starts PC server automatically)	`deploy/live/`
`deploy/summarise.py`	Parse all logs → markdown table	`docs/status/deploy-summary.md`
`deploy/all.py`	Full pipeline in one command	all of the above

PC workflow¶

Build¶

python3 deploy/build.py -target pc

Runs CMake configure + build. Output: deploy/build/pc/projectMM (or .exe on Windows). Log: deploy/build/pc/build.log.

Run unit tests¶

python3 deploy/unittest.py

Runs deploy/build/pc/tests/tests, streams output to stdout and deploy/test/run-tests.log, and generates docs/status/test-results.md from the JSON results.

Run the PC server manually¶

deploy/build/pc/projectMM --count 999999999   # run until Ctrl-C
deploy/build/pc/projectMM --count 2000        # run ~2 s then exit cleanly
deploy/build/pc/projectMM --verbose           # lifecycle + health logging

Always run from repo root so the server finds its state files in state/.

Verify the PC server¶

python3 deploy/run.py -type pc

Starts the binary, hits /api/system, stops it. Log: deploy/run/run-pc.log.

Run live tests against PC¶

python3 deploy/livetest.py -type pc

Starts the binary, waits for it to be ready, runs deploy/live_suite.py, stops the binary. Log: deploy/live/live-pc.log.

Full PC pipeline¶

python3 deploy/all.py

ESP32 workflow¶

Build¶

python3 deploy/build.py -target esp32dev
python3 deploy/build.py -target esp32s3_n16r8

Calls PlatformIO, copies artifacts to deploy/build/esp32/<env>/, and writes flash_args.json (chip, offsets, baud). Log: deploy/build/esp32/<env>/build.log.

Flash firmware¶

python3 deploy/flash.py                          # all test:true ESP32 devices
python3 deploy/flash.py -env esp32s3_n16r8       # all devices of this env
python3 deploy/flash.py -ip 192.168.1.234        # single device by IP
python3 deploy/flash.py -device_name MM-70BC     # single device by name

Reads flash_args.json produced by build.py, calls esptool directly with flash-mode keep and flash-freq keep (safe for qio_opi boards). Log: deploy/flash/flash-<env>-MM-<last4>.log.

Flash LittleFS filesystem¶

python3 deploy/flashfs.py                        # all test:true ESP32 devices
python3 deploy/flashfs.py --wifi                 # verify WiFi creds in data/state/sta1.json first

Builds the LittleFS image via PlatformIO (once per env), then flashes it alongside firmware in a single esptool pass. Log: deploy/flash/flashfs-<env>-MM-<last4>.log.

Warning: flashing LittleFS erases all device state (module list, saved controls).

WiFi credentials:

python3 deploy/wifi.py --ssid <ssid> --password <pw>   # writes data/state/sta1.json
python3 deploy/flashfs.py --wifi                        # verifies creds, then flashes

data/state/sta1.json is gitignored — credentials stay local. Without it, credentials can be entered at runtime via the web UI (connect to the AP MM-XXXX, open http://4.3.2.1).

Verify ESP32 devices¶

python3 deploy/run.py                            # all test:true devices
python3 deploy/run.py -env esp32dev              # single env

Prefers HTTP /api/system if ip is set in devicelist; falls back to serial (pyserial). Log: deploy/run/run-<env>-MM-<last4>.log.

Run live tests against ESP32¶

python3 deploy/livetest.py -type esp32           # all test:true ESP32 devices
python3 deploy/livetest.py -ip 192.168.1.234     # single device

Requires the device to be running (no flash step). Log: deploy/live/live-<env>-MM-<last4>.log.

Resetting stale LittleFS state¶

Flashing new firmware does not clear the LittleFS partition. If the running binary loads stale state from a previous firmware version, use flashfs.py to wipe and re-flash:

python3 deploy/flashfs.py

Then re-add modules via the UI or REST API.

PlatformIO IDE index rebuilding warning¶

Running build.py for multiple environments back-to-back triggers the PlatformIO VS Code extension's file watcher on .pio/libdeps/. Fix by excluding .pio/ from VS Code's file watcher in .vscode/settings.json:

{
    "files.watcherExclude": {
        "**/.pio/**": true
    }
}

Full pipeline¶

python3 deploy/all.py

Runs every step against all test:true devices in devicelist.json:

build.py -target pc
unittest.py
build.py -target <env> (once per unique ESP32 env)
flash.py (all test:true ESP32 devices)
(waits for ESP32 devices to boot)
run.py
livetest.py
summarise.py → writes docs/status/deploy-summary.md

Each step runs regardless of prior failures so the full summary is always produced.

Live system tests¶

deploy/live_suite.py runs REST API integration tests against a running projectMM server — PC or ESP32 — with no USB cable required. Tests set up a known module pipeline, exercise it, and verify observable state via GET /api/system.

Basic usage¶

# Against the local PC server (must already be running)
python3 deploy/live_suite.py

# Against an ESP32 over WiFi
python3 deploy/live_suite.py --host 192.168.1.234

# Run specific tests
python3 deploy/live_suite.py --tests test0,test1

# Custom settle time (seconds to wait after each module add)
python3 deploy/live_suite.py --settle 1.5

# Custom output file
python3 deploy/live_suite.py --out deploy/live/live-results-esp32dev.json

Options¶

Flag	Default	Meaning
`--host`	`localhost`	Target host (IP or hostname)
`--port`	`80`	HTTP port
`--tests`	all	Comma-separated list of test names to run
`--settle`	`0.5`	Seconds to wait after each setup step before asserting
`--out`	(none)	Path to write JSON results

Infrastructure protection¶

delete_all_modules() skips modules whose type is in INFRA_TYPES:

INFRA_TYPES = {
    "NetworkModule", "SystemStatusModule", "WifiStaModule", "WifiApModule",
}

These modules are never deleted by the test suite — deleting them would leave an ESP32 unreachable after a reboot.

Test descriptions¶

Test	Description
`test0`	Ensures all infrastructure modules are present; adds any that are missing. Runs first so subsequent tests can rely on SystemStatusModule health data.
`test1`	Deletes all non-infra modules, then builds a fresh pipeline: DriverLayer + GridLayout + EffectsLayer + RipplesEffectModule. Verifies modules are present and RipplesEffect is producing non-zero checksums.
`test2`	Adds a second EffectsLayer with LinesEffectModule. Verifies both effects run simultaneously and the driver blends them.
`test3`	Resizes GridLayout (depth 1→5, width 10→20) and verifies the pipeline resizes without crashing. Checksums must change as the pixel space grows.

Adding a new test¶

Define a function in deploy/live_suite.py:

def test4_my_test(client: Client, r: R, settle_s: float):
    """Short description shown in summary output."""
    result = client.add_module("MyModule", "mymod1")
    r.check(result.get("ok"), "MyModule added")
    time.sleep(settle_s)
    mods = client.modules()
    r.check("mymod1" in mods, "mymod1 present")

Register it in main():

runner.register("test4", test4_my_test)

Tests run in registration order. The first line of the docstring becomes the description in JSON results.

Log files¶

All logs live under deploy/ (gitignored by default, staged by pre-commit hook for committed runs):

Path	Contents
`deploy/build/pc/build.log`	CMake configure + build output
`deploy/build/esp32/<env>/build.log`	PlatformIO compile output per environment
`deploy/test/run-tests.log`	doctest output
`deploy/test/test-results.json`	Machine-readable doctest results (CI)
`deploy/run/run-pc.log`	PC `/api/system` response
`deploy/run/run-<env>-MM-<last4>.log`	ESP32 `/api/system` response or serial snapshot
`deploy/flash/flash-<env>-MM-<last4>.log`	esptool flash output per device
`deploy/flash/flashfs-<env>-MM-<last4>.log`	esptool LittleFS flash output per device
`deploy/live/live-pc.log`	PC server output + live test log
`deploy/live/live-<env>-MM-<last4>.log`	Live test log per ESP32 device
`deploy/live/live-results-pc.json`	Live test JSON results (PC)
`deploy/live/live-results-<env>-MM-<last4>.json`	Live test JSON results per ESP32 device
`docs/status/test-results.md`	Human-readable test summary (served by mkdocs)
`docs/status/deploy-summary.md`	Markdown table: all devices × all steps (served by mkdocs)

CI¶

GitHub Actions runs on every push to main and every pull request (.github/workflows/ci.yml):

Job	Runner	What it does
PC (macOS)	`macos-latest`	CMake build + doctest suite + run verification
ESP32 (`esp32dev`)	`ubuntu-latest`	PlatformIO compile + footprint report
ESP32 (`esp32s3_n16r8`)	`ubuntu-latest`	PlatformIO compile + footprint report

CI artifacts (uploaded on every run): deploy/ directory containing build logs, test results, and run output.

Footprint report¶

Every CI run appends an ESP32 flash/RAM report to the job summary. To run locally:

python3 scripts/esp32_footprint.py --log deploy/build/esp32/esp32dev/build.log

Budget caps and thresholds are in standards.md — Footprint Budgeting.

→ Memory and Filesystem test coverage — MemoryStats filesystem stats, heap sanity checks, consistent snapshots across multiple calls.

PlatformIO Python version¶

PlatformIO requires Python 3.10–3.13. If your system Python is newer (e.g. Homebrew upgraded to 3.14+), the build will fail.

The deploy/build.py script automatically prepends PlatformIO's own Python to the PATH before invoking pio, so it works correctly regardless of system Python version.

If you need to invoke pio directly from a shell:

PATH=~/.platformio/python3/bin:~/.platformio/penv/bin:$PATH platformio run

Permanent fix — install PlatformIO via pipx with a pinned Python:

brew install pipx && pipx ensurepath
brew install python@3.12
pipx uninstall platformio 2>/dev/null || true
pipx install --python $(which python3.12) platformio

Developer Guide — Deploy¶

Overview¶

Requirements¶

PC (macOS / Linux / Windows)¶

ESP32¶

First-time setup¶

Device registry — deploy/devicelist.json¶

Device selection¶

Pipeline scripts¶

PC workflow¶

Build¶

Run unit tests¶

Run the PC server manually¶

Verify the PC server¶

Run live tests against PC¶

Full PC pipeline¶

ESP32 workflow¶

Build¶

Flash firmware¶

Flash LittleFS filesystem¶

Verify ESP32 devices¶

Run live tests against ESP32¶

Resetting stale LittleFS state¶

PlatformIO IDE index rebuilding warning¶

Full pipeline¶

Live system tests¶

Basic usage¶

Options¶

Infrastructure protection¶

Test descriptions¶

Adding a new test¶

Log files¶

CI¶

Footprint report¶

PlatformIO Python version¶

Device registry — `deploy/devicelist.json`¶