Testing Guide

This page covers the test infrastructure, the four test-complexity levels, and how to write a new test (including how to lock down a specific bug so it can never silently reappear).

---

Does a test already cover my scenario?

Before writing a new test, search the existing ones:

grep -r "BrightnessModifier" tests/

Example: does changing brightness actually affect the raw pixel bytes?

Yes: tests/test_brightness_mod.cpp contains exactly this assertion:

TEST_CASE("BrightnessModifierModule - loop() output scales with brightness from KvStore") {
    KvStore& kv = KvStore::instance();
    kv.clear();

    EffectsLayer layer(4, 4);
    layer.setup();
    BrightnessModifierModule m(&layer, 0.0f);  // frequency=0 → sin(0)=0
    m.setup();

    // brightness=0 → every pixel byte must be 0
    kv.setFloat("brightness", 0.0f);
    m.loop();
    layer.publish();
    Channel* ch = layer.readyChannel();
    bool allZero = true;
    for (uint32_t i = 0; i < 4u * 4u; ++i)
        if (ch->pixels[i].r != 0 || ch->pixels[i].g != 0 || ch->pixels[i].b != 0)
            allZero = false;
    CHECK(allZero);

    // brightness=1.0 → pixels should be ~127 (mid-scale)
    kv.setFloat("brightness", 1.0f);
    m.loop();
    layer.publish();
    ch = layer.readyChannel();
    bool allMid = true;
    for (uint32_t i = 0; i < 4u * 4u; ++i) {
        uint8_t r = ch->pixels[i].r;
        if (r < 120 || r > 135) { allMid = false; break; }
    }
    CHECK(allMid);
    m.teardown(); layer.teardown(); kv.clear();
}

This is a behavioral test: it drives the module through two states and checks actual output byte values, not just that the code compiled.

---

Test infrastructure

Framework: doctest, a single-header C++17 library that compiles and runs on PC, Raspberry Pi, and (with the ESP-IDF target) ESP32.

Location: tests/test_*.cpp, one file per feature area.

Run all unit tests:

python3 deploy/build.py -target pc    # build first (required)
python3 deploy/unittest.py            # run tests; writes test-results.json
# or run the binary directly after a build:
deploy/build/pc/tests/tests --no-header -m fail

Register a new test file in tests/CMakeLists.txt:

target_sources(tests PRIVATE
    ...
    test_my_feature.cpp
)

And add a title entry in deploy/unittest.py:

FILE_TITLES = {
    ...
    "test_my_feature.cpp": "My Feature",
}

---

Test complexity levels

Every test case carries an implicit complexity level. The level is assigned in deploy/unittest.py by keyword matching on the test name. Aim to have most tests at behavioral or integration.

Level	What it verifies	Example
smoke	Code does not crash. Lifecycle (setup/loop/teardown) without an assertion on output.	ArtNetOutModule - lifecycle without crash
format	String or schema shape: field names present, healthReport format, schema keys. No output values checked.	BrightnessModifierModule - setup registers frequency control
behavioral	Actual output values, state transitions, field round-trips, boundary conditions.	BrightnessModifierModule - loop() output scales with brightness from KvStore
integration	Multiple modules wired together, cross-module paths, protocol correctness, HTTP/WS end-to-end.	SineEffectModule publishes brightness, BrightnessMod reads it

A healthy test suite has few smoke and format tests relative to behavioral and integration. If most tests are smoke, the suite will pass even when output is silently wrong.

---

Writing a smoke test

Check that setup() / loop() / teardown() complete without crashing. No assertions on output.

TEST_CASE("MyModule - lifecycle") {
    MyModule m;
    m.setup();
    m.loop();
    m.teardown();
    // No CHECK — pass = no crash.
}

---

Writing a format test

Check schema keys and healthReport structure without verifying values.

TEST_CASE("MyModule - getSchema has expected controls") {
    MyModule m;
    m.setup();

    JsonDocument doc;
    m.getSchema(doc.to<JsonObject>());
    JsonArray ctrls = doc["controls"].as<JsonArray>();

    bool foundSpeed = false;
    for (JsonObject c : ctrls)
        if (strcmp(c["key"] | "", "speed") == 0) foundSpeed = true;
    CHECK(foundSpeed);
    m.teardown();
}

---

Writing a behavioral test

Check that output changes correctly when inputs change. This is the most valuable level: use it for every non-trivial state transition and every bug fix.

Pattern:

1. Wire the module with a controlled input (fixed layer, known KvStore value, known props).
2. Call loop() / publish().
3. Read Channel::pixels and assert on the actual byte values.

TEST_CASE("MyEffect - output is non-zero when enabled") {
    EffectsLayer layer(4, 4);
    layer.setup();

    MyEffect m;
    m.setInput("layer", &layer);
    m.setup();
    m.loop();
    layer.publish();

    Channel* ch = layer.readyChannel();
    REQUIRE(ch != nullptr);
    bool anyNonZero = false;
    for (uint32_t i = 0; i < 4u * 4u; ++i)
        if (ch->pixels[i].r || ch->pixels[i].g || ch->pixels[i].b) anyNonZero = true;
    CHECK(anyNonZero);

    m.teardown();
    layer.teardown();
}

Using KvStore as input (for modifiers that read a shared float):

KvStore& kv = KvStore::instance();
kv.clear();
kv.setFloat("brightness", 0.5f);
// ... call loop(), check pixels ...
kv.clear();  // always restore global state

---

Writing an integration test

Wire multiple modules together and verify the end-to-end path. Use ModuleManager for full pipeline tests; wire manually for fast, focused ones.

Manual pipeline (fast, no file I/O):

TEST_CASE("SineEffect -> BrightnessModifier pipeline") {
    KvStore& kv = KvStore::instance(); kv.clear();

    EffectsLayer p1(4, 4), p2(4, 4);
    p1.setup(); p2.setup();

    SineEffectModule sine;
    sine.setInput("layer", &p1);
    sine.setup();
    sine.setControl("amplitude", 0.0f);  // brightness=0 published to KvStore

    BrightnessModifierModule bmod;
    bmod.setInput("layer", &p2);
    bmod.setup();

    sine.loop(); p1.publish();   // publishes brightness=0 to KvStore
    bmod.loop(); p2.publish();   // reads brightness=0, output should be black

    Channel* ch = p2.readyChannel();
    for (uint32_t i = 0; i < 4u * 4u; ++i)
        CHECK(ch->pixels[i].r == 0);

    sine.teardown(); bmod.teardown();
    p1.teardown(); p2.teardown();
    kv.clear();
}

Via ModuleManager (slower, exercises full lifecycle including state persistence; use for HTTP/WS and config-load tests, see test_http_server.cpp).

---

Locking down a bug with a regression test

When a bug is reported:

1. Write a test that reproduces it (it should fail on the current code).
2. Fix the bug.
3. Verify the test now passes.
4. The test stays in the suite forever, catching any future regression.

Example: user reports "setting brightness to 0 in the UI still shows dim pixels".

TEST_CASE("BrightnessModifier - brightness=0 produces all-black output (regression)") {
    KvStore& kv = KvStore::instance(); kv.clear();
    EffectsLayer layer(4, 4); layer.setup();
    BrightnessModifierModule m; m.setInput("layer", &layer); m.setup();

    kv.setFloat("brightness", 0.0f);
    m.loop(); layer.publish();

    Channel* ch = layer.readyChannel();
    for (uint32_t i = 0; i < 4u * 4u; ++i) {
        CHECK(ch->pixels[i].r == 0);
        CHECK(ch->pixels[i].g == 0);
        CHECK(ch->pixels[i].b == 0);
    }
    m.teardown(); layer.teardown(); kv.clear();
}

Name the test clearly so the intent is obvious in test-results.md. The word regression is optional but useful as a marker.

---

Live tests (hardware-in-the-loop)

Unit tests run on PC. Live tests run against real devices over HTTP and verify end-to-end behavior without touching USB.

Run:

python3 deploy/run.py              # all devices in devicelist.json
python3 deploy/run.py -target pc   # PC only

Live tests are registered in deploy/live_pc.py / live_esp32.py:

runner.register("test6_brightness", test6_brightness_pipeline, level="behavioral")

A live test function makes HTTP calls and asserts on JSON responses:

def test6_brightness_pipeline(device):
    # Set brightness to 0 via REST
    r = requests.post(f"{device.url}/api/control",
                      json={"id": "bmod1", "key": "brightness", "value": 0})
    assert r.status_code == 200
    # Read health report and verify output is dark
    r = requests.get(f"{device.url}/api/test")
    assert r.json()["bmod1"]["checksum"] == 0  # all-black

Live tests run on every push as part of deploy/all.py if hardware is reachable. CI skips them (no hardware present).

---

Electrical validation with a logic analyzer

Live tests assert on REST responses, which proves the application logic is correct but does not verify that the GPIO output actually carries the expected bits. A USB logic analyzer closes that gap by capturing the signal at the pin.

This is currently a manual / exploratory tier. Automating it is tracked in the backlog ("HIL instrument integration").

Reference device: DLA Mini (24 MHz, 8 channels)

A cheap Saleae Logic 8 clone using the Cypress FX2LP chip. Driver: fx2lafw (open source, ships with sigrok). Voltage: 5V tolerant, reads 3.3V ESP32 GPIO as logic high without a level shifter.

What it can capture

Protocol	Bit rate / clock	Verdict at 24 MHz sample rate
WS2812 / WS2811 / NeoPixel	800 kHz (T0H 0.4 µs, T1H 0.8 µs)	Comfortable. ~10 samples per T0H, ~20 per T1H.
APA102 / SK9822	1-30 MHz SPI clock	Borderline above ~5 MHz. Want 4-10x oversampling.
Art-Net / DDP / E1.31	Network protocols	Not applicable. Capture with Wireshark.
Hub75 panel data	5-20 MHz pixel clock, multiple data lines	Too slow. Need a faster analyzer.

The 24 MHz figure is shared bandwidth across active channels. Sustained captures across all 8 channels can overrun the USB 2.0 stream buffer, but 1-2 channels at full rate are fine.

Wiring

1. Power the ESP32 from USB or its own supply. Do not power LEDs from the analyzer.
2. Connect the analyzer GND to the ESP32 GND (required: shared ground).
3. Connect one analyzer channel (e.g. CH0) to the GPIO data pin you want to inspect (the pin you configured the driver module to use).
4. For APA102: also connect CH1 to the clock pin.

Software (sigrok / PulseView)

Free, open-source, has a built-in WS2812 protocol decoder.

# macOS
brew install sigrok-cli pulseview
# Linux
sudo apt install sigrok-cli pulseview

Interactive (PulseView): open PulseView, select the fx2lafw device, set sample rate to 24 MHz, set a rising-edge trigger on CH0, capture for 10 ms, then add the WS2812 protocol decoder on CH0. The decoder shows the per-pixel R/G/B byte stream.

Headless (sigrok-cli): capture and decode in one command, useful for scripted assertions.

sigrok-cli --driver fx2lafw --channels D0 \
  --config samplerate=24m --samples 240000 \
  --output-format binary --output-file capture.bin
sigrok-cli --input-file capture.bin \
  --protocol-decoders ws2811 \
  --protocol-decoder-annotations ws2811=rgb

Future automation path

The intended pytest -m hil tier (see backlog) will:

1. Set a known pixel pattern via REST (POST /api/control with a fixed RGB value).
2. Trigger a sigrok-cli capture for ~50 ms.
3. Decode the WS2812 stream.
4. Assert that the decoded byte sequence matches the expected pattern.

This validates that the rendering pipeline, the driver module, and the physical GPIO output are all correct end-to-end.

---

Quick reference

Task	Command
Build unit tests	python3 deploy/build.py -target pc
Run all unit tests	python3 deploy/unittest.py
Run one test file	deploy/build/pc/tests/tests -tc="BrightnessModifier*"
Add a test file	Add to tests/CMakeLists.txt + deploy/unittest.py FILE_TITLES
Check test classification	See docs/status/test-results.md after running unittest.py
Run live tests	python3 deploy/run.py