Lessons

Hard-won debugging lessons and gotchas, recorded with the code that proved them. The PR-merge carry-forward gate writes new entries here (CLAUDE.md § Lifecycle Events). This is the lesson record: a bug, its root cause, and the fix. Three neighbours hold the other genres, and a lesson belongs in whichever fits:

• A genuine architectural decision (chose approach A over B/C) is an ADR, immutable and kept.
• A durable rule that graduated from a lesson lives in CLAUDE.md or coding-standards.md, not here.
• The forward-looking design intent of a feature is a plan.

Entries run oldest-first by branch. A lesson fully absorbed into a rule doc or the code is pruned (per Mandatory subtraction): the git history is the permanent record, this file is the working narrative on top.

Foundational lessons (early projectMM, pre-branch-log)

The debugging lessons from the first iterations that a present-tense doc can't hold — the architecture-shaping retrospectives they came with have been absorbed into CLAUDE.md and architecture.md:

• Verify a protocol against the spec's test vectors, not internal consistency. A one-character typo in the RFC 6455 WebSocket magic GUID passed every internal check (the SHA-1 was self-consistent) but the browser rejected the handshake — silently. When implementing a wire protocol, assert against the RFC's published test vectors.
• freeHeap() vs freeInternalHeap() on PSRAM devices. freeHeap() returns internal + PSRAM combined, but the HEAP_RESERVE guard must use freeInternalHeap() — stack, HTTP, and WiFi need internal RAM, not PSRAM. Checking the combined figure lets internal RAM exhaust while the reserve check still passes.
• Delete copy/move on any class owning raw memory (Rule of Five). MoonModule and ControlList owned raw pointers but had implicit copy/move — a double-free waiting to happen. A class that owns a raw resource declares (or deletes) all five special members.
• setName must copy, not store a pointer. HTTP module creation stored a pointer to a stack-local name buffer; after the call returned the name was garbage. Owned strings are char[N] + memcpy, never a borrowed pointer.

Lessons from the next-iteration branch (plans 08-12)

The branch covering SystemModule/NetworkModule, persistence, the UI rewrite, the eth-only build, and the side-nav.

• Plan-09's persistence abandonment was a success. The first attempt (~1700 LOC) didn't pay for itself; ~700 LOC of foundations (partition scheme, platform fs API, MoonModule additions) were kept, the rest dropped, and plan-10 succeeded with a smaller control-list-driven design.
• ESP-IDF v6.x removes WiFi via EXCLUDE_COMPONENTS, not Kconfig. CONFIG_ESP_WIFI_ENABLED=n is silently ignored in v6.x. The eth-only profile excludes esp_wifi/wpa_supplicant/esp_coex (NOT esp_phy, the EMAC needs it) plus an MM_NO_WIFI define gating if constexpr branches.
• The render tick collapsed on a blocking 49 KB preview WebSocket write spinning vTaskDelay until lwIP drained. Fix: non-blocking scatter-gather write + downsample to fit the send buffer.
• WiFi UDP is ~4× the Ethernet per-packet cost (WiFi CSMA/CA, retries, rate adaptation, not a code regression): ArtNet at 16K lights is ~7 FPS on WiFi vs ~19 on Ethernet.
• Two no-op wrappers were found and removed by the Reviewer (HttpServerModule::parseJsonString re-namespacing mm::json::*; NetworkModule::rebuildLocalControlsAndPipeline whose name contradicted its body).
• An unescaped control value containing " or \ produced malformed JSON in /api/state and the persisted config; caught by a round-trip test, not the reviewer.
• A mid-implementation design change (password length-only → XOR+base64) left one header's comment stale, it documented a security property the code no longer provided.

Lessons from this branch (plans 13-16)

Plan-13 (nest child cards), plan-14 (replace-type button), plan-15 (stream /api/state), plan-16 (Layouts/Layers/Drivers reshape), plus mid-branch fixes (effect freeze, Int16 zero-corruption, Layouts disable crash, status slot, layers reorg, FilesystemModule + Scheduler split).

• ControlDescriptor.min/max are uint8_t and can't bound wider widths. Applying them to Int16/Uint16 clamps every value into [0..0]; addInt16/addUint16 leave bounds at 0,0 because the slot can't represent the wider range. The load path silently zeroed every Int16 control on every reboot ("Layouts cannot be activated after reboot"); reverted with a comment in Control.h on why bounds stay 0,0.
• Per-tick integer division rounded four effects' animation rate to zero on fast devices. phase += dt * bpm * 256 / 60000 truncates to 0 when dt < 234/bpm ms (desktop dt ≈ 0..1ms; ESP32 at 16K LEDs is fine). Fix: keep the raw dt * bpm numerator in the accumulator, divide at the read site (NoiseEffect's pattern). Now CLAUDE.md § Hard Rules: "effects must run at every grid size and tick rate."
• Layer::onAllocateMemory early-returned on empty layouts without resetting its LUT or buffer; Drivers then reallocated the output buffer to 0 bytes while the stale LUT pointed at 16K destinations, and blendMap dereferenced null. "Nothing to do" branches must still reach a consistent zero state.
• HTML5 dragstart's e.target is always the draggable element, so e.target.closest(".child-class") in dragstart never matches (e.target === card). Use the mousedown target instead; toggle draggable on mousedown, with a touchstart mirror. Shipped silently because the exclusion list happened to cover <input>; surfaced only with a <details>/<summary> control.
• A single warning slot conflates info, degradation, and failure. Three levels (Status/Warning/Error) earn their keep once more than one module produces non-degradation messages; wire format mirrors the C++ enum lowercased ("status"/"warning"/"error"), the field-name/severity status collision documented at the introduction site.
• The lifecycle events are "commit" and "merge"; "push" has no work of its own. A separate Push event for the Reviewer bred the "address-reviewer" noise-commit anti-pattern; the Reviewer at Commit cost 5-7 min per commit. Final shape: reviewer at PR-merge over the whole branch diff, on-demand pre-commit as the safety valve.
• The reviewer agent flags real findings AND wrong ones (~30% valid across three passes here). Wrong ones misread line numbers, repeated an accepted finding, or proposed re-introducing a fixed bug (the c.min/c.max Int16 finding twice). "Skip with one-line reason in the commit body" is the honest response, and the reason becomes the audit trail.
• Top-level system docs are flat (architecture.md, coding-standards.md, building.md, testing.md); per-module specs live under docs/moonmodules/. The earlier architecture.md + architecture-light.md pair was asymmetric and pulled toward more suffixes; merging into one architecture.md with # Core/# Light domain sections matches every well-known project. Subfolders only kick in under moonmodules/ where there's a plural of each kind.
• util/ and modules/ buckets were rejected. A util/ bucket groups by file-shape not concern (each header already names what it does); a modules/ bucket needs a plural of each kind to earn its keep, but the four system services are singletons. The right cleanup was header-only → .h+.cpp splits, done lazily.
• .claude/scheduled_tasks.lock is harness runtime state, not project content. Ignore .claude/*.lock (a broader pattern than per-file); the single-file .claude/settings.local.json ignore was too narrow.

Lessons from this branch (plans 17-23)

The plan-18 branch landed plans 17 + 18 + six unplanned follow-ups (19, 19.1, 20, 21, 22, 23).

• GitHub Pages CDN returns no Access-Control-Allow-Origin, so cross-origin fetches of release-asset .bin files fail, CORS-on-static-files isn't fixable from your side. Plan-18 pivoted to self-hosting the last N releases' binaries on Pages content (same-origin), rather than a third-party CORS proxy.
• Chrome's mixed-content policy blocks an HTTPS Pages page from fetch("http://192.168.1.X/…") even with Access-Control-Allow-Origin: *; the block happens before the request leaves the browser. Plan-20's Diagnose feature moved to the device UI (same-origin, mixed-content moot).
• ESP Web Tools' rich panel ("Visit Device" / "Configure Wi-Fi") is in-browser-session-only, the device URL is browser-side memory, not asked-back from the device. A device-side GET_CURRENT_STATE → URL follow-up (ESPHome pattern) surfaces the data but does NOT change the third-party tool's UI.
• improv_provision returns ERROR_UNABLE_TO_CONNECT when WiFi STA is already connected, by design (protects large installs from a scan-induced ArtNet drop). The browser shows "Unknown error (255)" because Improv's error mapping carries no human reason; document the rejection at every layer the user meets it.
• Per-board build directories land as build/<board>/, not <chip>/build/<board>/. build/esp32-<board>/ keeps every target under one root shared with desktop targets (build/macos/, …); the doubled esp32-<board> prefix is intentional. One root, one cleaner.
• idf.py -B <dir> needs -DSDKCONFIG=<dir>/sdkconfig to isolate per-build-dir sdkconfigs. Without it, idf.py writes <project>/sdkconfig shared across boards, tripping "project sdkconfig was generated for target X, CMakeCache contains Y" on the second board build. Retrofitted to build_esp32/flash_esp32/collect_kpi.
• Adding an Improv child to NetworkModule (plan-21) reverted, then resolved by the Peripheral-role branch. The base lifecycle does propagate every callback to children; a child misses one only when the parent overrides that method and forgets to chain to base. Fix is per-parent (SystemModule chains setup()/loop1s()), not a scheduler refactor. unit_SystemModule pins it; the general rule lives in coding-standards.md § Override-and-chain convention.
• Split platform_esp32.cpp at public-API boundaries, not section banners. Improv + OTA + LittleFS each own private state and talk back only through platform.h, so they split cleanly; Network stayed put because Eth + WiFi + sockets + mDNS share eight file-scope variables.
• Desktop's platform_desktop.cpp is correctly asymmetric with ESP32's, its OTA/Improv/FS sections are 6-line stubs, so per-subsystem files would be all overhead. Symmetry across platforms is a heuristic, not a rule.
• Nightly builds live in their own workflow (nightly.yml tags nightly-YYYY-MM-DD, release.yml builds via tag push): zero duplication of the build matrix. The skip-on-no-change check costs ~2s on quiet days.
• workflow_dispatch reads the workflow YAML from the default branch, not the dispatched branch. Cost a CI cycle on RC2: dispatched against plan-18 with a tag invalid against main's older release.yml. inputs.tag arrives correctly but the consuming logic is whatever main has.
• The reviewer agent's PR-merge job is architectural drift across N commits, not line-level bugs (CodeRabbit's job). Two agents, two scopes.
• A 13-commit branch is the upper end of what one merge should carry, the merge train is heavy and the reviewer's job gets harder as commits stack. Aim for "ship 3-4 plans, merge, start the next branch." This one worked because the plans were mostly independent.
• MoonModule's asymmetric lifecycle propagation was historical, not principled. setup/teardown/onBuildControls/onAllocateMemory propagated to children, but loop/loop20ms/loop1s defaulted to empty no-ops, so every container duplicated a 5-line per-child block. A shared tickChildren helper (gated by !respectsEnabled() || enabled(), per-child timing accumulated) closed it; leaf modules pay one predicted-not-taken branch. Desktop tick stayed 55-160 µs. The parked Plan-21 move became four lines.
• Override-and-chain: option A for loop (parent prepares, then chain so children read fresh state, Drivers::loop runs blendMap before children read outputBuffer_); option B for setup (chain first so children init before the parent depends on them); teardown chains late. The conventions live in coding-standards.md § Override-and-chain convention and architecture.md § Lifecycle propagation to children.
• Control-change reactions are a three-tier split; the coarse-grained rebuild debt is closed. handleSetControl now: (1) always calls MoonModule::onUpdate(controlName); (2) calls scheduler_->buildState() only when controlChangeTriggersBuildState(controlName) is true (default false, overridden on LayoutBase/ModifierBase); (3) the sweep reaches each onBuildState(). Slider drag no longer triggers a tree-wide realloc. Mirrors MoonLight's onUpdate/requestMappings/onSizeChanged split (confirmed against MoonLight source); the verb is "build" not "rebuild" (idempotent), so onAllocateMemory became onBuildState.
• Output correction (brightness/reorder/white) is a per-driver stage shared via the Drivers container. ArtNet was sending raw bytes (no brightness/order/white), a gap. Drivers owns a Correction (256-entry brightness LUT + channel-order table + derive-white flag) and hands each child a const Correction*; brightness applies before white derivation. The field is briLut not gammaLut so gamma folds in as a fill, not a rename. Follows MoonLight's driver-edge per-channel LUTs.
• Three call sites carried their own 50-60-line switch (c.type) over ControlType (HttpServerModule::writeControls, FilesystemModule::writeValue, scenario_runner), which drifted (the scenario runner stopped recognising new types). Extracted to free functions in Control.cpp (writeControlValue/applyControlValue/…); JsonSink gained a fixed-buffer mode so the FS path shares the serializer without a per-value alloc, and an ApplyPolicy param (Strict/Clamp) preserved tolerant load. Codified in coding-standards.md § Per-type behaviour lives with the type.
• Local Improv testing closed a high-friction dev loop. Before, verifying Improv end-to-end meant tag a release → CI → deploy Pages → flash from the live installer, burning a release tag per iteration. preview_installer.py's flash-ready mode stages local build/esp32-*/projectMM.bin under releases/latest/, generates Pages-relative manifests, and serves at localhost:8000 (Web Serial works on localhost without the secure-origin gate). Paired with improv_smoke_test.py (probe + provision + LAN reachability).

Lessons from this branch (Board injection follow-ups)

• src/ui/release-picker.js is now src/ui/install-picker.js (symbol installPicker, C array installPickerJs), renamed once the picker grew from "pick a release" to "pick release + board + firmware + install". A wide but mechanical ~20-file sweep. Recorded so a search for the old name lands here.
• Why the web installer dropped ESP Web Tools for a custom orchestrator. ESP Web Tools 10.x's <esp-web-install-button> held the SerialPort exclusively across flash + provision and fired state-changed inside its dialog's shadow DOM, so post-PROVISIONED board injection was structurally impossible and devices.js's auto-add broke. Owning the SerialPort end-to-end in install-orchestrator.js lets both fixes land in one place, and each future injectable adds one vendor command ID + one dispatcher case.

Lessons from the ESP32-S3 N16R8 (DevKitC) enablement branch

Three non-obvious failures adding native-USB S3 support, all with the shape "symptom looks like X, root cause is elsewhere."

1. USB-Serial-JTAG ≠ UART0 on ESP32-S3. The Dev's USB-C wires through the built-in USB-Serial-JTAG peripheral, not an external bridge to UART0. IDF's secondary console (CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG=y, default for S3) mirrors stdio both paths, so boot logs appear and the developer assumes UART0 works. The Improv-listener-on-UART0 was deaf. Fix: install BOTH drivers and read both transports; #if SOC_USB_SERIAL_JTAG_SUPPORTED keeps classic builds clean. Don't make it compile-time-per-board, the same binary should work with either wiring.
2. CORS preflight is silent on the client side. A cross-origin POST with Content-Type: application/json triggers an OPTIONS preflight; if the device returns 405 to OPTIONS, the browser silently drops the POST, no error, no network line, no console message. Burned a session diagnosing what looked like a fan-out bug. Fix: always implement OPTIONS (204 + Access-Control-Allow-Origin: *, -Methods, -Headers: Content-Type); verify with curl -X OPTIONS … returning 204 not 405.
3. Cached "last applied" TX-power went stale when the WiFi stack restarted. appliedTxPowerSetting_ skipped redundant esp_wifi_set_max_tx_power calls, but an AP→STA cascade / reconnect / AP shutdown resets the radio's TX-power while the cached value stayed equal to the desired one, so syncTxPower() short-circuited and the cap never re-landed. Fix: every wifiStaStop()/wifiApStop() also invalidates the cache (= -1).

Core/light type boundary: light_types.h split + preview decouple

src/core/types.h had grown into a junk-drawer of light-domain types (nrOfLightsType, CoordCallback, defaultGridSize, HEAP_RESERVE, lengthType) alongside Dim. Split so each symbol lives with its owner, done in three passes:

• Pass 1 (no core consumer): nrOfLightsType/CoordCallback/defaultGridSize → src/light/light_types.h; HEAP_RESERVE → platform.h (a platform memory constraint, not Layer's, even though Layer was its only caller, ownership follows concept, not call count).
• Pass 2 (lengthType, by removing its incidental core consumers): Control.h only mentioned it in a comment; HttpServerModule::put16 only took it because it serialised PreviewFrame, a light struct sitting in core. Introduced BinaryBroadcaster (core interface, ~6 lines: "send these bytes to all WS clients"); moved PreviewFrame.h → src/light/, where PreviewDriver now owns the 13-byte header and pushes bytes (replacing the old PreviewFrame::ready poll). lengthType → light/light_types.h, zero core users left.
• Pass 3 (Dim, and the deletion of core/types.h): ModuleFactory::registerType<T> probed dimensions() naming Dim in the constraint, but the next line did static_cast<uint8_t>(...), the name was incidental. Loosened the probe to requires { static_cast<uint8_t>(t.dimensions()); }; Dim moved to light/light_types.h, core/types.h was empty and deleted. Verified /api/types still reports dim 3/2/0 unchanged.

Every tie here turned out incidental and severable: a comment, a serializer following a misplaced struct's field type, and a SFINAE constraint that named a type it immediately discarded. End state: no core/types.h; core names zero light types.

A static "current instance" pointer needs re-election, not just claim/vacate

AudioModule::latestFrame() hands effects the active mic via a process-wide static AudioModule* active_. "setup() claims, teardown() vacates" silently breaks with two mics: removing the one holding active_ leaves the seat null while a second running mic sits captured-but-unread, and every audio effect goes silent. Fix is a three-part protocol: the first live module claims in setup(), teardown() vacates, and any running module re-claims an empty seat in loop(), so the survivor takes over on its next tick for any add/remove order. (unit_AudioModule pins the two-mic first-wins + re-election.) A tempting CodeRabbit "fix" (gate the claim on inited_) was rejected: a claimed-but-uninited module publishes valid silence (the documented contract), and a mic-less board running simulate publishes synth frames without being inited_.

An "Effect-role" module is not guaranteed to be an EffectBase

DemoReel hosts a child effect and needs its dimensions() to extrude it. The first cut did static_cast<EffectBase*>(current_)->dimensions() each frame on the MoonModule* child, which crashed (SIGBUS in RTTI/vtable) when the eligible list included a test EffectStub that registers with ModuleRole::Effect but is a bare MoonModule. Two lessons: role is a registration property, not a type guarantee (a downcast keyed on role is UB); and the data was already available RTTI-free, the factory probes dimensions() at registration via if constexpr and stores it, so ModuleFactory::typeDim(index) gives it (ESP32 builds -fno-rtti, so dynamic_cast isn't even an option).

uint16 intermediate overflow blanks the display, and a status check doesn't prove the render works (MultiplyModifier)

MultiplyModifier at 8×8×4 black-screened the no-PSRAM Olimex while desktop showed it working. Layer::rebuildLUT computed maxDest = logicalCount * mod->maxMultiplier() in nrOfLightsType; on no-PSRAM that's uint16_t, and 256 * 256 = 65536 wraps to 0, so the LUT was sized to ~nothing and almost every light mapped nowhere. On desktop (uint32_t) the product fits, so the bug was invisible. Fix: compute in uint64_t, clamp to the ceiling, narrow back.

The harder lesson is verification: the agent's hardware test asserted the Layer status string ("16×16×1"), which was correct, while the render (driven by the corrupted LUT) was black; the product owner caught it by eye. A correctness test for a mapping/effect must assert the buffer or LUT is non-empty with expected coverage (the regression counts LUT destinations == physical light count), not just the declared dimensions. A width-sensitive bug is invisible on the uint32 desktop build; such paths need a uint16-typed unit test or hardware confirmation.

Lessons from the repo-transfer + v1.0.0 release branch

Moving ewowi/projectMM → MoonModules/projectMM and cutting v1.0.0 surfaced three CI failures from the infrastructure around the release, not the diff.

1. windows-latest migrated from VS 2022 to windows-2025-vs2026 mid-release, breaking a green tree two ways: (a) package_desktop.py hard-coded -G "Visual Studio 17 2022" → CMake couldn't find VS 2022; (b) once the pin was dropped, the new VS 2026 MSVC STL emitted C5285 ("specializing std::tuple is forbidden") on vendored doctest.h, fatal under /WX. Fixes: drop the generator pin (auto-detect survives migration); add /wd5285 (third-party header, GCC/Clang never warn).
2. A Pages-only environment: github-pages on the release job made a tag fail the gate before any step ran, dropping all release assets. The job did asset upload AND Pages deploy under one environment whose protection rule allowed only main; the v1.0.0 tag failed it, so the whole job (including "Publish GitHub release") was rejected in 2 seconds, a published release with zero binaries, and a red X blaming Pages. Fix: split into a release job (no environment, contents: write, runs on tags) and a deploy-pages job (needs: [release], if: ref==main). Recovery without re-tagging: gh workflow run release.yml -f tag=vX.Y.Z.
3. Three "failures" were the environment, not the change, a 120µs tick contract "failed" at 536µs under concurrent-build load (118µs isolated); Improv reported UNABLE_TO_CONNECT while the device was provisioned and reachable (async-confirmation timeout); MoonDeck showed 0/0 online while the device served HTTP 200 (an active network record with an empty subnet). None were defects.

Lessons from the LCD_CAM WS2812 driver bench debug (LcdLedDriver, S3)

Bringing the 8-lane LCD_CAM driver from "compiles and ticks" to "strip animates" took three stacked root causes, each masked by the one before; every layer of indirection hid a failure the layer above couldn't see.

1. The i80 peripheral requires ALL bus_width data GPIOs, a partial bus never exists. esp_lcd_new_i80_bus rejects GPIO_NUM_NC entries, so a 1-pin config never initialized: no bus, dark strip, while the UI showed an enabled driver. Fix: demand exactly 8 pins and report LCD bus needs exactly 8 pins; unused lanes take 0 and idle LOW.
2. "Capacity still fits" is not "config unchanged." reinit() skipped the bus rebuild when the DMA buffer was big enough, correct for grid resizes, wrong for pin edits (moving lane 0 from GPIO 13 to 18 keeps the frame size, so the bus kept clocking the OLD pins). Fix: record the pin set the live bus was built with, compare on every reinit, any difference forces the rebuild. Same family as the S3-DevKit stale-cache lesson: the cached check was true about the wrong invariant.
3. Gate on CONFIG_SOC_LCDCAM_I80_LCD_SUPPORTED, NOT CONFIG_SOC_LCD_I80_SUPPORTED. The classic ESP32 also defines SOC_LCD_I80_SUPPORTED=1 for its unrelated I2S-LCD peripheral, so gating on it wired LcdLedDriver at boot on classic, and esp_lcd_new_i80_bus() hung the watchdog → boot-loop on every classic ESP32. The correct macro is defined only on chips with the real LCD_CAM (S3/P4). Bisect-proven, hardware-verified.
4. A self-test that passes while the device fails is telling you what the test doesn't cover. The first loopback transmitted a 3-byte synthetic pattern through a 136-byte transfer and PASSED while the strip washed out max-white. Upgrading the test to transmit the driver's REAL frame (5.4 KB, multi-descriptor GDMA chain, sustained cadence) and bit-verify the whole capture eliminated two suspects in one run, leaving timing.
5. Modern WS2812B reads a 412 ns "0" as "1", T0H max is ~380 ns on newer revisions, and 3.3 V direct drive eats the margin. The classic 2.4 MHz encoding (slot ≈ 416 ns) decoded perfectly on RMT RX yet rendered max white with flicker; the RMT driver's 350 ns zeros ran clean, isolating timing. Fix: pclk 2.67 MHz → 375 ns slots ("0"=375, "1"=750, bit 1125 ns), inside every WS2812B revision's window; latch pad resized to keep ≥300 µs. The lineage gets away with 416 ns behind a 74HCT level shifter.

Bench notes: a board that drops STA mid-session falls back to softAP silently, poll reachability before reading an unanswered API call as a wedge. The differential test that cracked it twice: drive the same pin/strip with the proven RMT driver, it exonerates wiring, power, and strip in one move.

Diagnosing LED flicker: eliminate firmware with hardware tests before blaming (or fixing) the wire

A classic ESP32 driving WS2812 on RMT showed random wrong colours on LEDs the effect left black. Instead of guessing (WiFi, buffering, timing), a four-step elimination each a measurement: (1) capture the source/preview buffer, clean, so corruption is downstream of the logical buffer; (2) whole-frame loopback self-test through a jumper, bit-exact PASS even under WiFi load, so the firmware/peripheral is innocent; (3) sweep txPowerSetting 20→1 dBm, flicker constant, so it's NOT WiFi coupling; (4) check pulse timing, 350/700/1250 ns spec-exact, not a timing-margin bug. Every firmware cause eliminated by test, the remaining cause is the physical data path, and "constant regardless of TX power" fingers signal integrity, a missing 3.3→5 V level shift. Fix is hardware (documented in LED signal integrity).

Red-herring note: the flicker's appearance shifted (blue-only → random) after an unrelated lightPreset RGB→GRB default change plus a pin swap. The electrical fault was identical; only the colour mapping changed. A changed symptom is not proof a code change caused it.

ESP32-P4 support, round 1, per-board Ethernet pin config, and the P4's WiFi reality

Adding the Waveshare ESP32-P4-NANO (round 1 of 4: board + Ethernet-only).

• The P4 has no native WiFi. SOC_WIFI_SUPPORTED is absent on esp32p4 (it has EMAC, RMT, LCD_CAM, Parlio, no radio). WiFi comes from an on-board ESP32-C6 over SDIO via esp_wifi_remote/esp-hosted (a managed component, not in mainline v6.1-dev). Round 1 ships Ethernet-only. C6 SDIO pins on the P4-NANO (so round-2 Parlio avoids them): CLK 18, CMD 19, D0-D3 14-17, C6 reset 54.
• Ethernet pins became a per-target compile-time config, not scattered #ifdefs. ethInit() had Olimex RMII/PHY pins baked as literals; the P4-NANO needs IP101 PHY addr 1, MDC 31, MDIO 52, reset 51, and an external 50 MHz RMII clock fed IN on GPIO50 (EMAC_CLK_EXT_IN, opposite of Olimex's EMAC_CLK_OUT). An EthPinConfig struct + constexpr ethPins = isEsp32P4 ? {…} : {…} lives in platform_config.h; ethInit reads it. Keeps the platform-boundary rule (if constexpr, not #ifdef in domain code).
• The IP101 PHY driver is a managed component in IDF v6 (espressif/ip101); the generic PHY (Olimex LAN8720) stays in core, so only the P4 build pulls it, behind if constexpr (ethPins.isIp101). A build-script trap: the eth-fragment detector matched only .eth, so sdkconfig.defaults.esp32p4-eth would have set MM_NO_ETH, the matcher now accepts both .eth and -eth.

ESP32-P4 round 2, Parlio LED driver: a simpler peripheral, the same encoder

Adding ParlioLedDriver (round 2 of 4) was mostly subtraction from the LCD driver.

• Parlio is simpler than the i80 bus: no sacrificial WR/DC pins (Parlio clocks internally via output_clk_freq_hz, so no clockPin/dcPin), and no exactly-8-pins rule (Parlio takes data_gpio_nums[] with unused = -1, so 1-8 lanes work).
• The encoder is shared, not duplicated. LcdSlots.h::encodeWs2812LcdSlots outputs one bus byte per slot, and a Parlio bus byte is byte-identical to an i80 one, so ParlioLedDriver reuses the same encoder and unit_LcdLedEncoder.cpp test, zero new encode code.
• One Parlio constraint: data_width must be a power of two, ≤ SOC_PARLIO_TX_UNIT_MAX_DATA_WIDTH. Always create at data_width = 8 (matching the encoder's 8-bit byte), unused lanes' GPIOs -1.
• Clock math is identical: Parlio's PLL_F160M ÷ 60 = 2.67 MHz = the same 375 ns slot the LCD driver settled on for the T0H-margin reason.
• Default pins must dodge the P4's strapping GPIOs. The first cut defaulted 36,37,38,… but GPIO 34-38 are strapping pins (boot-mode control). It inited fine on the bench (nothing drove them during boot) but defaulting LED output onto strapping pins is a latent footgun. Fixed to pins="20,21,22,23,24,25,26,27" with all 64 lights on lane 0 (the other 7 idle LOW at zero cost, the parallel DMA transfer time is set by the longest lane, not the lane count). Clear P4-NANO GPIOs after Ethernet, C6 SDIO, I2C, and strapping: 20-27, 32-33, 39-48. Pull the strapping-pin list from the datasheet before picking defaults for a new chip.

Audio input (INMP441 mic), ship the manual core, design fresh from the datasheet

The first audio-reactive capability: AudioModule (a SystemModule Peripheral) reads an INMP441 I²S mic and publishes an AudioFrame (level + 16-band spectrum + peak) that AudioVolumeEffect/AudioSpectrumEffect consume.

• Two seams, everything else host-tested. Only the I²S read and the FFT kernel sit behind the platform boundary (i2s_std + esp-dsp's float dsps_fft2r_fc32). All signal math (DC strip, RMS, Hann window, magnitude→16-band log map) is header-only domain code. The desktop audioFft stub is a real naive O(n²) DFT, so the whole pipeline runs in CI on synth sines. esp-dsp float (not fixed-point) is right on an FPU chip.
• Effects reach the producer via AudioModule::latestFrame() (a static accessor), not a boot setter, because an audio effect can be added through the UI after boot and a setter only wired the boot instance. The active mic registers in setup(), clears in teardown(), returns a static silent frame when there's no mic.
• Two hardware-only bugs, now pinned: a missing registerType<AudioModule> made create(...)->markWiredByCode() deref null and boot-loop; the I²S read blocked the tick ~7.7 ms at a 20 ms timeout (fixed to non-blocking + a cross-tick sample accumulator, dropping to ~400 µs). The INMP441 is on the left I²S slot here (right reads empty), the first bench suspect when level floors with sound present.
• Shipped the manual core; the adaptive conditioner was prototyped and removed. Auto noise-floor + AGC + smoothing needs tuning in a quiet room; the dev environment (a campground van with strong varying low-frequency rumble) was the adversarial worst case that kept it from settling. Land the manual core, treat adaptive auto-tuning as its own increment, tune it where the noise floor is real-quiet. Also: level is overall RMS (independent of the FFT), don't derive it from the bands or it stops fluctuating with volume.
• Designed fresh from the datasheet + textbook DSP, not a prior project. The datasheet made even a behaviour-reference unnecessary: a flat ±3 dB mic has no per-frequency error to correct, so the hand-tuned band-correction table years of prior-project work produced was the wrong tool; the textbook defaults sufficed. The DSP choices and why live in AudioModule.md.

ESP32-P4 round 3, WiFi via the C6: the abstraction the earlier round feared wasn't needed

Round 1 predicted round 3 would need a WiFi abstraction seam; the actual implementation was far smaller.

• esp_wifi_remote is API-compatible, so there was no seam to build. You still call esp_wifi_init()/esp_wifi_connect()/…; the component forwards them to the C6 over SDIO. The entire WiFi platform layer (~230 lines) compiles unchanged. The only new code is a two-call prelude (esp_hosted_init() + esp_hosted_connect_to_slave()) run before esp_wifi_init(), behind if constexpr (platform::usesRemoteWifi) (= isEsp32P4 && hasWiFi). Before designing a two-backend abstraction, check whether the vendor already made them API-identical.
• Init ordering is the whole risk. esp_hosted must be fully up before anything touches the WiFi stack; wrong ordering surfaces as NVS errors / asserts / a silent hang, not a clean error. The prelude-before-esp_wifi_init placement is the mitigation and the first thing to check if a P4-wifi boot misbehaves.
• Pulled P4-only via a rules gate (if: "target == esp32p4" in idf_component.yml), so other targets never fetch or compile the managed components.
• A deliberate v6.0-floor exception: these components live outside mainline v6.0. The floor rule's documented-exception path applies, accepted consciously, recorded at every introduction site, scoped to the P4. A floor rule with a documented-exception path beats a rigid one.

RMT timeout: don't cancel a stuck transfer, the cancel crashes classic ESP32

A deferred "fuller error handling" item for rmtWs2812Show/rmtWs2812Wait (🐇 CodeRabbit PR#17) was mostly already done and partly actively harmful. Most had landed in the multi-pin work: rmtWs2812Transmit already returns the rmt_transmit result, and RmtLedDriver::loop() already tracks started[] so it only waits on channels whose transmit succeeded. The remaining item, cancel the in-flight transfer on timeout via rmt_disable(), is a trap: rmt_disable() while a transmission is active triggers an interrupt-WDT panic on classic ESP32 (espressif/esp-idf#17692; classic-only). That trades a self-healing dropped frame for a crash, strictly worse. The right change was to NOT add the cancel and document why the current code is already right (next tick re-encodes and re-transmits; a still-busy channel fails its rmt_transmit cleanly, started[] skips the wait). A deferred-improvement note is a hypothesis, not a spec, verify the improvement is real AND safe on every target before implementing.

Pin defaults: assign one only when it cannot do harm

A mic-less classic ESP32 boot-looped (TG1WDT_SYS_RESET at ~736 ms, silent hang). Bisect (clean-built known-good still looped; disabling AudioModule wiring booted clean) fingered AudioModule: auto-wired via addChild + markWiredByCode() (gated on platform::hasI2sMic), it ran setup() → reinit() → audioMicInit() → i2s_channel_enable(), which on the classic's older I²S driver blocks forever when no mic clocks the pins. The P4 was never affected (newer I²S returns silence or fails cleanly).

The fix (a design fix, not a band-aid): don't auto-wire AudioModule (register it in the factory so it's user-addable); default the mic pins to unset (0); reinit() no-ops on any unset pin (setStatus("set …"); return;). Classic then boots 191 FPS, 0 WDT resets. This drove the general rule now in architecture.md § Config provenance and coding-standards.md § Defaults: chip-/board-fixed pins (RMII Ethernet) must default (omitting them is a chicken-and-egg lockout); user-soldered pins (mic, LED strands) stay empty until set. Never auto-run a peripheral whose init can block on absent hardware.

Live reconfiguration falls out of the prepare-pass for free, MoonLight's "initless" goal, a different mechanism

projectMM reconfigures every module live the instant a control changes (pins, leds-per-pin, output protocol, mic pin/rate), no reboot. The design note lives in architecture.md § Live reconfiguration. The lineage is MoonLight's "initless drivers" (a driver with no addLeds/initLed step, reading a mutable Context at show() time). projectMM reaches the same outcome by a different mechanism: our drivers do have an explicit rebuild (RmtLedDriver::reinit(), the i80/Parlio DMA-bus rebuild), but it's driven by the generic tier-3 onBuildState() sweep, not hand-built per driver, so any module returning true from controlChangeTriggersBuildState inherits live-reconfig, spanning drivers, audio, effects, layouts, modifiers, and network I/O alike. Credit the lineage for the idea, but name the property by what the user sees when the mechanism differs.

Lessons from the catalog-driven installer branch (3-layer device model)

The installer was reworked so a board catalog sets a device up for its hardware at install time. The mechanics live in the installer README; these are the principles.

• Inject from data, don't bury in code (vs MoonLight's ModuleIO.h). MoonLight hardcoded ~20 boards' pin presets in firmware C++ behind #ifdef CONFIG_IDF_TARGET_*; adding a board is a recompile and every binary ships every board's table. projectMM injects the same info from the catalog after flash, the firmware is a generic engine, the data specialises it. Adding a board is a JSON edit; board definitions become community-contributable data.
• Investigate before building, the device side was already done. The original plan was a new POST /api/preset endpoint; investigation found three install clients already fan a catalog's controls out as POST /api/control calls, and the real gap (a control write 404s on a fresh flash where the module doesn't exist) was solved by the already-existing idempotent POST /api/modules the clients weren't driving. A day of mapping the existing code replaced a new core endpoint (and a forbidden JSON-array parser).
• loopbackTxPin, verify the claim against the code before deciding twice. A loopbackTxPin override was proposed, dropped on "it only fits RMT's single pin, not Parlio/Lcd's lane array," then re-added after reading ParallelLedDriver::runLoopbackSelfTest: the Parlio/Lcd loopback only drives lane 0, so a single TX override substituting for lane 0 works on all three drivers. The "doesn't fit lane arrays" objection was an assumption, not a fact.
• Board vs Device is a completeness spectrum, not two schemas. A carrier/shield PCB an ESP32 module plugs into is a Board; a vendor-finished all-in-one (QuinLED Dig-2-Go) is a Device, but they're the same catalog entry shape, a Device is just a Board entry with more optional modules/controls filled in. So no separate-schema devices.json. A board's pins fall into three categories: always-fixed (LED outputs, status LED, default freely), board-optional (a populated-or-not W5500/IR, can't default blind because the same board name ships both ways), and user-soldered (always unset).
• Per-board capability spec'n'test loop. Each board's pin-layout image and product url are inputs to a repeatable loop: read capabilities off the image + link; for a capability we offer (I²S mic → AudioModule) wire it into modules/controls with real pins; for one we don't (IR receive) write a proposal, spec it, create test scripts, iterate, then add it. An un-implemented capability is a recorded proposal, not a half-wired control. Specs before code at board granularity.
• Drivers became catalog-added, with an OTA nuance. LED/network drivers stopped being boot-wired (only Preview stays, since it needs the HTTP broadcaster the catalog can't supply); each board declares its driver(s) in modules. A fresh-erased board boots with Drivers = [Preview] only. The nuance: an OTA update without erase keeps a device's previously-persisted drivers (saved config the new firmware reloads), so "out-of-box" and "after-update" differ.
• A persistence overlay must distinguish "key absent" from "value 0". The runtime-Ethernet-PHY work moved pin/PHY config into persisted controls with non-zero per-chip defaults (P4 IP101 ethType 2). applyControlValue used json::parseInt, which returns 0 for an absent key, indistinguishable from a real 0, and wrote that 0 under the Clamp policy, so loading an older <Module>.json clobbered the default. On the P4 this zeroed ethType (2→0=none): link LEDs on, no DHCP. Invisible on classic (eth defaults mostly 0) and on main (still read constexpr ethPins). Fix: a json::hasKey() guard in applyControlValue. Any "control resets to 0 after reboot" symptom is a persistence-overlay smell. The decisive move was a std::printf over the P4's secondary USB-Serial-JTAG console after a git worktree bisect.
• A GPIO pin is its own control type (ControlType::Pin), not an overloaded int16. Pins added as addInt16 -1..48 rendered as a slider (meaningless for a GPIO) and the cap excluded the P4's high pins (MDIO 52, clk 50); dropping the range didn't help because the UI's int16 case always draws a slider. Fix: a dedicated Pin type (int8_t storage, a GPIO never exceeds ~54; −1 = unused; the UI renders a plain number input keyed off the "pin" type string; min/max are a server-side write-clamp only). Every future pin control migrates to addPin for free.
• deviceName (identity) vs deviceModel (product) vs board (bare PCB), one term was doing three jobs. "Board" had meant per-unit network identity, the catalog key, AND the bare PCB. Untangled: deviceName is the per-unit identity (drives mDNS/AP/DHCP, RFC-1123-coerced); deviceModel is the hardware product (catalog key, spaces allowed, never a hostname); "device" is the umbrella; "board" means only the bare PCB. Drove the BoardModule→SystemModule fold, the board→deviceModel rename (SET_BOARD→SET_DEVICE_MODEL, byte 0xFE unchanged), and the eth pin-map clarification (driver = firmware, pin map = firmware-seeded but deviceModel-authoritative).
• "Improv = REST over serial", one apply-core, two transports. The HTTPS installer couldn't POST to an http:// device (mixed-content), and the ?deviceModel= pull only ran if the user opened that exact link. Fix: the installer owns the USB serial port during provisioning, so push config over it as the same REST ops, a new APPLY_OP (0xFC) Improv vendor RPC whose payload is the same JSON a POST /api/modules//api/control body carries. On device it routes to one transport-free apply-core (HttpServerModule::applyAddModule/applySetControl/…) the HTTP handlers also call, so serial and network execute identical code. This deleted the whole browser handoff. The hard part (chunk reassembly + out-of-order/duplicate sequence guard) was extracted to src/core/ImprovOpReassembler.h (header-only state machine) so it unit-tests on desktop, and the frame builders (device C++, Python, JS) are pinned by one shared golden vector in test/python + test/js. A hard mechanism buried in a platform .cpp that "can only be tested on hardware" is a smell, extract its pure core.
• A periodic re-broadcast to let late joiners catch up is a hack wearing a keepalive costume. The 3D preview re-sent the whole coordinate table every ~1 s "so a client that connected after the last rebuild catches it", rebuilding the full table from the layout every tick-second forever, on the hot path, whether or not anyone connected or anything changed. The correct construct is event-driven: send the table when it changes (onBuildState) and when a client asks (a new WS connection bumps BinaryBroadcaster::clientGeneration(), which PreviewDriver::loop() watches), strictly less code and zero idle cost. It sneaked past review because it worked in casual testing and its cost was invisible until a later change made each rebuild heavier. "Re-send periodically so it eventually syncs" is the polling-instead-of-events smell. Guard: a test that advances the clock several seconds with no client change and asserts the table is NOT re-sent.
• When a working seam regresses after your "fix," suspect the fix, and measure with a tool faithful to what the user sees. A later attempt to route the coordinate table + downsampled frame through the resumable send path (removing the synchronous spin-and-close) regressed every board into an intermittent stall through several variants. Three compounding lessons: (1) stop at the first failed fix on a working path (revert to known-good at attempt two, per Anti-stalling); (2) a plain one-shot WebSocket probe gave up on close where a browser reconnects, so it reported stalls users never saw and missed blips they did, the fix was a browser-faithful probe (moondeck/diag/preview_health.py: reads binary frames, 25 s keepalive ping, auto-reconnect, exactly app.js's connectWs); (3) a 38-hour-old desktop binary still held port 8080 so the freshly-built one couldn't bind, before bisecting a "rebuild didn't fix it" bug, confirm the artifact under test is the one you built (check process uptime / build timestamp / what's bound to the port).
• Don't hold a vendor library's async handle across your own event loop, it races the library's internal timers. The mDNS browse (peer discovery, distinct from advertise) used mdns_query_async_new, whose handle DevicesModule held across ticks and polled with a 0 ms timeout. The mDNS component's own task frees that handle's queue when the 3 s window expires, so a poll landing after expiry asserts on a freed queue (xQueueSemaphoreTake queue.c:1709), intermittent and grid-size-sensitive (a bigger grid widens the gap). A first fix (a cancel-before-mutate guard, assuming a service-table mutation tore it down) did nothing, because the freeing party is the expiry timer, not our code. Real fix: replace start/poll/stop with one synchronous mdnsBrowse() holding no handle across ticks. Synchronous mdns_query_ptr blocks the full window (~80 ms, charged to the tick), so throttle: browse one service type every ~8th tick with a ~60 ms timeout. A library's async/iterator handle is only valid between its lifecycle events; an intermittent load-dependent crash in a vendor backtrace is a lifecycle race, not a component bug (find the actual concurrent actor before fixing). Hardware-only; the reproduction (concurrent WS churn at a large grid) is the proof.
• A dead control that was always meant to be functional belongs in the mechanism that already expresses it. Six persisted-but-ignored Layer controls (percent region carving) were due to be wired into rebuildLUT; the product owner's question, can a modifier already do this?, was the better path. ModifierBase's two virtuals (logicalDimensions, mapToPhysical) express carving exactly, so it shipped as a modifier and the Layer controls were deleted. The implementation forced the boundary rounding: inclusive-ceil made two abutting layers (0..50, 50..100) overlap by one pixel at the seam, so the product owner chose half-open [start, end) (with a min-1-pixel floor). "Make the default fastest" is best met by making the default the absence of the feature (full coverage = no modifier = the existing fast path). Reach for half-open intervals whenever regions tile a space.

ESP32-S31 RGMII Ethernet bring-up: four bugs between "code compiles" and "link up"

Bringing up the S31's on-chip 1 Gb RGMII Ethernet took four fixes, none of them the C++, each a layer below the feature code that a build can't catch. For a hardware bring-up, "it compiles" tells you almost nothing; the truth is only in the boot log on the actual board.

• A MM_NO_ETH-style gate keyed on a filename silently omits a new board. build_esp32.py decided "does this firmware have Ethernet?" by pattern-matching the sdkconfig fragment filename for .eth; the S31 enables its EMAC in sdkconfig.defaults.esp32s31 (no .eth), so the gate compiled the ethInit() stub and the board fell back to WiFi with zero eth log. A gate asking "does X enable feature Y?" must read what X contains (CONFIG_ETH_USE_*=y), never what it's named.
• A schematic/reference pin table is a hypothesis until the boot log confirms it, ours was systematically off by one. MDC/MDIO/TXD were all one GPIO low. Wrong data pins failed loudly (invalid TX_CTL GPIO number), but wrong MDIO failed as "No PHY device detected", three inference-steps from the typo. The chip's own IDF IO_MUX table (esp32s31/emac_periph.c) is the authority for RGMII data pins (fixed pads); verify SMI pins against ETH_ESP32_EMAC_DEFAULT_CONFIG; use PHY-addr auto-detect (-1).
• A shared clock is a contended resource, the RGMII 125 MHz Tx clock can't come from a PLL already spoken for. The default (AUTO) sourced it from the MPLL, but PSRAM already ran the MPLL at 400 MHz (no integer path to 125) and CPLL couldn't synthesise 125 on the 40 MHz XTAL grid; only the fractional APLL works. On an SoC where one PLL feeds multiple peripherals, "pick a clock source" is a conflict-resolution decision, check what already owns each PLL.
• Changing a Kconfig choice in a defaults fragment needs a clean build. Editing CONFIG_ETH_EMAC_RGMII_TX_CLK_SRC_* and building incrementally kept the old choice (the build dir's sdkconfig already had a value; defaults don't override an existing one). Two flash cycles were spent "testing APLL" that were still CPLL. rm -rf the build dir when a sdkconfig choice changes.

W5500 SPI Ethernet (LightCrafter / SE16): interrupt service, IDF-v6 config, and a per-board reset pin

Bringing up the two Limpkin ESP32-S3 boards' external W5500 (SPI) Ethernet surfaced three traps between "the driver installs" and "the link is up."

• Interrupt mode needs gpio_install_isr_service(), miss it and the IRQ silently degrades to slow polling. The W5500 has a real INT pin (SE16 GPIO18, LightCrafter GPIO45), but the IDF driver's gpio_isr_handler_add() requires the per-pin ISR service already installed, which nothing did. The add failed with a one-line log, the interrupt never fired, and RX serviced only on the slow fallback → a ~1 s sawtooth ping latency. Install the ISR service once before driver init (ESP_ERR_INVALID_STATE = already installed = fine). The int_gpio=-1 polling fallback is a workaround that hides this.
• IDF v6's W5500 driver rejects int_gpio_num < 0 unless a poll period is also set. int_gpio_num = -1 alone returns invalid configuration argument combination; you must also set w5500_config.poll_period_ms (e.g. 10). The pair is the API contract.
• A failed Ethernet init that cascades to WiFi masks the eth bug, "why are we on WiFi?" is the tell. Both bugs above manifested as the board quietly running on WiFi with a healthy HTTP server and no eth log. The robustness cascade is correct behaviour but turns a loud init failure into a silent capability loss; a board that should be on Ethernet but is on WiFi is the signal to read the eth init log.
• A 0x00 "version mismatched" read from an SPI PHY means the chip is held in reset. The LightCrafter's WIZ850IO holds its own nRST until GPIO3 is driven HIGH; without releasing it the ESP32 read 0x00 from the W5500's version register (expected 0x04, got 0x00). The SE16's W5500 self-resets, which is why this was board-specific. Setting ethRstGpio (→ the driver's reset_gpio_num) fixed it. An all-zeros register read from an SPI peripheral is "the chip isn't talking"; the first suspect after wiring is a reset/enable line the host must drive.

An RMT/IR done-callback runs in ISR context: signal only, decode on the task

The first NEC-decoder version crash-looped the board (USB dropped, port re-enumerated) because the RMT on_recv_done callback both decoded the frame (calling non-IRAM_ATTR helpers) and re-armed rmt_receive() from interrupt context. On ESP32, an ISR that jumps to flash-resident code while the cache is disabled, or re-enters a driver call, faults. The existing working RX capture (rmtWs2812RxCapture) already showed the safe shape: the ISR only records the symbol count and signals a queue; the decode + re-arm happen on the task (irRead, from the render loop). An RMT/GPIO done-callback may touch only its stack, the passed event data, and an ISR-safe FreeRTOS primitive (xQueueSendFromISR); parsing, logging, and re-arming move to the waking task. When in doubt, copy the codebase's already-proven callback.

Don't promote one specific board's pins to "the chip default"

NetworkModule::ethType defaulted to a per-chip ethConfigDefault, and the classic / P4 / S31 entries were, on inspection, one specific board's wiring each (classic = Olimex, P4 = Waveshare NANO, S31 = Function-CoreBoard) dressed up as a chip property. Two failures: a WiFi-only board (LOLIN-D32 Shelly) wasted an Ethernet-PHY init on hardware it lacks; and the QuinLED Dig-Octa (custom MDC/MDIO, no software eth reset, GPIO5 is an LED output there) would have inherited reset=GPIO5 and driven an LED pin as an Ethernet reset. Fix: ethType defaults to None, Ethernet is opt-in, every board declares its own ethType + wiring pins in deviceModels.json, and check_devices.py enforces "ethType set → wiring pins present." A "default" that is really one board's values is bespoke masquerading as standard; make the capability opt-in and require each consumer to state its own values, so a missing declaration fails loudly instead of inheriting a stranger's wiring.

A new firmware default only reaches a factory-fresh device; persistence wins on an upgrade

After changing the ethType firmware default to None and reflashing an already-provisioned board, the device still reported the old value, persistence (FilesystemModule overlays every saved control at boot) restores the value captured during earlier provisioning, and a plain flash write keeps the filesystem. The same shape bit the MQTT prefix (a stale stored prefix a reflash didn't clear). This is correct persistence behaviour (saved config wins over a constructor default), but a default change is invisible to any device that has already run, only an erase-flash or explicit re-set surfaces it. When you change a default to fix a bug, the fix lands only on factory-fresh devices; an already-provisioned fleet needs an explicit migration, and when debugging "my fix didn't take," check the persisted .config/*.json before the firmware default.

Fixing "blocking in the tick" at the connect misses the same violation at the write

The MQTT client's first version blocked the render loop twice, a synchronous connect() (with DNS) and an unbounded blocking write() retry, both inside loop1s (which runs in Scheduler::tick). The first review caught the connect (reworked to non-blocking connectStart/connectPoll); the re-review caught that conn_.write() was still there: a zero-window broker (accepts TCP, stops reading) fills the send buffer and the next write never returns, freezing the render loop permanently, and the keepalive timeout can't even fire because control never comes back. Fix: sends go through a non-blocking writeSome with an all-or-reset rule (control packets ≤256 B, so atomic-send is realistic). A "no blocking in the hot path" audit must sweep every syscall the path can reach, connect, DNS, read, and write, not just the loudest one; the re-review exists to catch exactly the partial fix that reads as complete.

A retract/cleanup path gated on "connected" strands the resource when disconnected

HA MQTT discovery's toggle-off publishes an empty retained config (so HA removes the entity) then frees the 768 B discovery buffers. The first cut guarded the whole path on state_ == Conn::Connected, so toggling discovery off while the socket was mid-reconnect returned before freeing, stranding the buffers until teardown, and never sent the tombstone, so the broker kept the stale retained config and HA kept the entity. Found live on P4/S31 hardware; every unit test had driven retract from the connected state, so the disconnected path had zero coverage. Fix: the retract frees unconditionally (freeing local memory needs no socket) and CONNACK retracts when discovery is off (so a reconnect clears a config left retained from a previous session); only the empty-retained PUBLISH itself needs a live link. General shape: a cleanup/retract path that also releases a resource must not gate the release on connection state, only the network send; and a path only ever exercised in one state (connected) needs an explicit test in the other. (unit_MqttModule "retract frees even while disconnected" pins it.)

Raw-HTML <img src> is not validated by MkDocs --strict, so a wrong ../ depth ships a silent 404

The catalog/summary pages hand-author each card image as a raw <img src="../../assets/...">, and the build hook moves the tag into a table cell without touching its src. MkDocs --strict validates markdown ![]() links but NOT raw-HTML <img src>, so 34 card images with a wrong ../ depth (the catalog pages sit one level shallower than the generated moxygen pages, so they need ../../assets, not the moxygen pages' ../../../assets) resolved to a nonexistent repo-root /assets/ and 404'd on the live site while every gate passed green. The desktop build, ctest, and --strict all miss it; only loading the deployed page shows it. Fix + guard: a test/python check resolves every catalog <img src> on disk (test_mkdocs_slug.py::test_catalog_card_images_resolve_on_disk). General: a generated-site check that only validates the syntaxes its linter knows leaves the others as silent-404 territory, verify the actual rendered output, or gate the un-validated syntax explicitly.

Lessons from the HA-finetuning bench pass (ha-mqtt branch, WLED /json + diagnostics)

A live-in-Home-Assistant debugging pass over the WLED-compat surface, plus a mic bring-up that ate an afternoon. The gotchas that a green build didn't catch.

• HA reads WLED info.leds.seglc[] as a per-segment capability CODE, not an LED count. The /json writer put the LED count in seglc (seglc:[24] on a 24-LED strip). HA's WLED integration indexes seglc[segment_id] and maps it through LIGHT_CAPABILITIES_COLOR_MODE_MAPPING (a small enum: 1 = RGB, etc.) — a 24 has no mapping, so WLEDSegmentLight ends up with no supported colour modes, HA raises does not set supported color modes, and the light entity stays restored/unavailable while the sensors keep working. It hid for ages because a single-light board (count:1) sends seglc:[1], which is accidentally the valid RGB code, so those boards worked; only a multi-LED board tripped it. Fix: seglc is the constant 1, matching lc; the count lives only in count. General shape: when impersonating another device's API, a field that looks numeric may be an enum/bitmask in the consumer — validate against the consumer's parser (here frenck/python-wled + ha-core wled/const.py), not against what the number "obviously" means. Pinned by test/python/test_wled_json_shape.py parsing a golden /json through the real library.
• HA's WLED light entity needs a complete segment, not a valid one. After seglc was fixed the light was still stuck, because the segment carried pal but no fx — a shape real WLED never emits (it always reports both). python-wled's Segment half-populated, and setup still failed silently (no log line). Real WLED always pairs effect + palette; sending one without the other is the trap. Fix: emit fx:0 alongside pal. When mimicking a device, match the full field set of the real thing for any object the consumer parses, not the subset you happen to use.
• A WLED-shim device shows up in HA over TWO independent discovery paths, and both fire at once. A projectMM board announces via WLED (mDNS _wled._tcp + /json, no broker) and, if the haDiscovery control is on, via MQTT discovery (retained homeassistant/light/<id>/config). HA creates a light entity from each — so the device lists twice (one platform=wled, one platform=mqtt), which reads as a bug. The WLED path is richer (colour, palette, sensors) and needs no broker; MQTT is the fallback for broker-only / cross-subnet setups where mDNS can't reach. Resolution: haDiscovery now defaults off so a device appears once by default; MQTT discovery is opt-in. (Recorded as a decision in ADR-0012.) General: when a device speaks two auto-discovery protocols to the same hub, exactly one should be the default or the hub double-lists it.
• An Ethernet device sending a zeroed wifi block makes HA render greyed Wi-Fi sensors; omit the block instead. /json unconditionally emitted wifi:{bssid:00:…, rssi:0, channel:0, signal:0} — correct-but-empty on an eth board (no AP). HA's info.wifi is optional in python-wled, so a present-but-zero block still spawns Wi-Fi RSSI/BSSID/channel sensors that sit greyed. Fix: on ethConnected(), drop the wifi object entirely — HA then creates no Wi-Fi sensors for an eth device (what a real WLED-on-eth does). General: for an optional field a consumer keys entity-creation off, absent and present-but-empty are different — absent is the honest signal for "this capability doesn't exist here."
• A silkscreen 45 read as 15 put an I²S mic pin on a boot strap, and the mic read dead-silent — not a bad mic. Hours went into a "broken" mic (dead-flat RMS on every pin, phantom signal that hopped between pins run-to-run) before the cause turned out to be the WS wire on GPIO45 (an S3 boot strap, driven at reset → silent) misread off the header as GPIO15. Two diagnostic traps compounded it: (1) RMS>0 — or even RMS varies — does NOT confirm a mic, because a floating/strap/mis-numbered input pins RMS at ~255 or bounces erratically; only RMS tracking sound on ONE fixed pin does; (2) an earlier actual solder fault on the SD line masked the real issue for a while. Fixes: a mic-health status (no samples = clocks dead / data line silent = SD dead) that self-reports which wire is at fault; the working bench config is SCK=47/WS=48/SD=21 (48 is free on this rev v1.0 board, LED on 38). General: re-read physical pin numbers against the silkscreen first on a dead peripheral, and A/B against a known-good board on the same firmware before suspecting software.

Lessons from this branch (windows-esp32-fixes)

Windows ESP32 bring-up end-to-end plus the beta1 pin-bump aftermath. Four gotchas each layered under a working build.

• A "private" LL_ HAL symbol can be the de-facto stable interface, while its public SOC_ replacement is transient. ESP-IDF's HAL docs mark RMT_LL_TX_CANDIDATES_PER_INST (hal/rmt_ll.h) as internal, but it's shipped consistently across every v5.x and v6.x snapshot this codebase has been built against — except one in-between v6.1-dev commit that briefly removed it in favour of SOC_RMT_TX_CANDIDATES_PER_GROUP (soc/soc_caps.h). Following the docs and switching to the SOC symbol broke on beta1, which restored RMT_LL_* and removed the SOC one. When a codebase already uses an "internal" HAL symbol proven stable across releases, upstream docs calling it internal aren't binding — the moving target may actually be the "public" alternative during a component-split refactor. Cross-check what Espressif ships, not what they document. The net-zero-code round-trip on platform_config.h in this branch is the receipt.
• Two build_esp32.py invocations against the same project race on esp32/managed_components/. IDF's component manager writes to a project-scope directory, not per-build-dir. Kicking off the S31 build while S3-n16r8 was still resolving components produced a partial managed_components/ where one component's CMakeLists.txt pointed at source files that didn't land. Symptom: CMake Error: Cannot find source file mid-configure. Serialize IDF builds — either wait for one to finish before the next, or introduce a lock at wrapper level. moondeck/build/build_esp32.py has no queue today; serialisation is the caller's responsibility.
• A failed set-target leaves the build dir with a fallback esp32 sdkconfig that the wrapper's fast path trusts silently. build_esp32.py's "dir exists → skip set-target" fast path assumes the existing dir is coherent for the requested firmware. It isn't when a previous set-target esp32p4 bailed early (e.g. the RISC-V toolchain wasn't in PATH) after IDF wrote a partial sdkconfig defaulting to esp32. The next invocation skips set-target and builds a classic esp32 binary in a build/esp32-esp32p4-eth/ directory. Symptom: an esp32p4-labelled artifact esptool refuses to flash to a P4 ("chip is ESP32-P4, not ESP32"). Fix in this branch: stale_feature_cache() gains a chip param and reads IDF_TARGET:STRING= from CMakeCache.txt before the existing feature-flag checks; a mismatch (or missing target line) forces the wipe-and-reconfigure path. Generalisation: any fast path that skips work because "the previous run left state" needs to validate that state is coherent, not just present.
• Python scripts emitting non-ASCII crash on Windows when stdout is piped. sys.stdout on Windows defaults to cp1252, which has no ⚠, →, box-drawing, or emoji. Direct-to-terminal often works because the Windows terminal negotiates UTF-8 on modern versions, but piping (Tee-Object, redirection, CI log capture) uses the raw stdio encoding and raises UnicodeEncodeError mid-message. sys.stdout.reconfigure(encoding="utf-8", errors="replace") at script entry is a one-line fix, no-op on POSIX. Every projectMM Python script that may emit non-ASCII AND be piped needs the reconfigure — worth generalising into a shared helper when the next script hits the same trap.

Disable-releases-resources: one router (applyState), not a per-module enabled() check

"Disabling a module frees its hardware" (so the pin map's freed-pin display is truthful and a shared GPIO is genuinely reusable) resolves to a single core primitive, not per-module bookkeeping — but getting there passed through a wrong turn worth remembering.

• The shipped design. MoonModule::applyState() is the sole resource-lifecycle router: it calls onBuildState() (pure build — acquire) on an effectivelyEnabled() node and teardown() (release) otherwise, recursing the tree so each child is routed by its own effective-enabled. A module's onBuildState() therefore contains no enabled() check — it just builds; core decides whether to build or tear down. effectivelyEnabled() walks parent_, so a disabled parent releases its whole subtree (a disabled Layer frees its effects' heap), and a respectsEnabled()==false ancestor (Network/HttpServer) is neutral, never forcing children on. This mirrors the loop gate (tickChildren / Scheduler::shouldRun): the same "one place owns the enabled decision" shape, now applied to acquire/release.
• The wrong turn: per-module if (enabled()). The first cut at disable-releases-resources had each resource-holder self-gate — if (enabled()) reinit() pasted into setup(), onBuildState(), onCorrectionChanged(), and the buffer setters (~24 sites). It worked but pushed orchestration into catalog modules (a driver "remembering" a rule that isn't its job) and it was inconsistent (some if(!enabled()) return;, some if(enabled()) …). The lift into applyState() deleted all of it: the release code was already in each module's teardown(), so the onBuildState disabled-branch was a duplicate — removing it is pure subtraction. (This is the Complexity lives in core clause on "extend the core mechanism to the sibling path, don't re-implement per module" — the canonical repo example.)
• The cold-boot symptom that started it (keep — it's the reproduction). On the P4 (MHC-WLED shield, GPIO 20 = O20 level-shifted output), a disabled RmtLed on pin 20 grabbed the RMT channel at boot, so the enabled ParlioLed on the same pin drove nothing — LEDs dark. Toggling RmtLed off→on→off "fixed" it live (the disable freed the RMT channel, then ParlioLed's next enabled sweep re-inited over the now-free pin) — which is exactly what masked it during runtime-only testing: a toggle test never exercises a cold boot. The lesson: a lifecycle bug that a live toggle papers over still bites on the boot path — test persisted-disabled-at-boot explicitly (a disabled module's applyState() must route to teardown, asserted host-side).
• The gotcha applyState sidesteps: teardown() is not always a clean inverse of setup(). HueDriver's teardown() frees its light-name buffers, but its setup() (inherited no-op) does NOT re-fetch them — loop1s()/onBuildControls() rebuild them lazily. Under the router this is fine (release lives in teardown, rebuild is lazy, onBuildState is pure), but it's why "disable = blindly call teardown then setup" can't be a blanket rule: route through applyState (build-or-teardown), never a mechanical setup↔teardown swap.
• IrService + the active_ election show release lives in teardown. IrService has no channel member — its RMT-RX channel is a static behind the platform layer, keyed by pin — so its teardown() calls the platform::irStop() seam (the existing closeChannel()); re-acquire is lazy on the next irRead. AudioService/DevicesModule vacate their active_ singleton seat in teardown() (and in a destructor, so a bare destruction never dangles the static). All acquisitions belong in onBuildState, all releases in teardown — applyState picks which runs.

"Port enumerates but the chip is dead-silent" = the board's power/enable jumper (S31 J5), not the cable

An S31 Function-CoreBoard would not flash: the CP2102N UART port (/dev/cu.usbserial-…) enumerated fine, but esptool got No serial data received at every baud (115200/460800/921600), every reset method (default-reset / no-reset / usb-reset / a software EN pulse), and even with BOOT physically held through the connect window. A raw 20 s serial read while tapping RST returned zero bytes — a running or bootlooping ESP is loud (ROM reset banner every cycle), so total silence ruled out a bootloop too. The port even renamed (…340→…434) and dropped mid-session. Every software angle pointed at "the chip's UART0 isn't reaching the bridge," which reads as a cable/jack fault — but it wasn't. The cause was a missing 2-pin jumper, J5, next to the 5V→3.3V DC/DC converter (U4) and the 3.3V power-on LED (D11) on the power-input side. With J5 out the board is half-powered: the CP2102N runs off USB VBUS directly (so it enumerates), but the ESP32-S31's 3.3V/EN rail is incomplete, so the chip never drives UART0 — the exact "port present, chip silent, nothing in download mode" signature. Reinstalling J5 and esptool talked to the chip on the first try (same port, same cable). The lesson: when a board's USB port enumerates but the MCU emits nothing — not even a boot banner, not in forced download mode — suspect the board's own power/enable jumper or supply path before the cable or the USB jack. A bridge chip powered from VBUS will happily enumerate a dead board. (Diagnostic that would have saved the hour: a raw serial read on the first reset — zero bytes with the power LED lit means "not powered/enabled," not "reset timing.") Bench note for the S31 specifically: J5 = power/enable; leave it installed.

S31 LED-output validation: RmtLed + ParlioLed pass loopback, LcdLed is S3-only

Loopback self-test on the S31 (a WS2812 frame TX'd on one pin, read back on a jumpered RX pin — Tx=48 / Rx=47, the two stacked pins of J2 column 7, bridgeable with one cap): RmtLed PASS, ParlioLed PASS on RISC-V silicon. LcdLed reports "no valid pins" — correct, not a bug: LcdLedDriver is the ESP32-S3-specific LCD_CAM i80-bus driver, and the S31 (RISC-V) has no LCD_CAM peripheral, so platform::lcdLanes is 0 there. So the S31's supported WS2812 output drivers are RmtLed + ParlioLed; LcdLed is not one of them. One gotcha surfaced while testing several drivers in a row: they all defaulted their loopback to the same GPIO 48, so only one driver can hold the loopback pin at a time — toggle each driver's loopbackTest off before testing the next, or give them distinct jumper pairs (the same single-pin-owner rule the pin map enforces everywhere).

tickTimeUs is CPU-in-the-call, NOT frame time — the async-transmit measurement trap that nearly buried a working 48→76 fps win

The parallel-driver async double-buffer (Step 1.5: encode frame N+1 into a second DMA buffer while frame N clocks out, wait deferred to the start of the next reuse) was almost reverted as a no-op because the A/B was read off the wrong meter. The trap, and the corrections that untangled it:

• A module's tickTimeUs is CPU time spent inside its tick() call, which includes a blocking wait but excludes a deferred one — so it is NOT the output frame time. Synchronous (async OFF) the ParlioLed tick was ~10,800 µs = encode (~2,800) + blocking on the WS2812 wire (~8,000); the CPU sits in busWait, so the wire time is in the tick, and 1e6/10800 ≈ 92 reads like a real rate. Async ON the tick was ~3,790 µs — encode only, because busTransmit returns immediately and the wait, deferred to the next tick, finds the DMA already finished (it drained during the other modules' ~9 ms of ticks). 1e6/3790 = 264 — which is physically impossible: 256 lights × ~29 µs/light = ~7,474 µs is a hard 133 fps wire ceiling. That impossibility is the tell that tickTimeUs had stopped measuring frame time and started measuring only CPU. The real frame rate is the system tick (the whole render loop): 48 fps sync → 76 fps async. The driver's "264" is CPU headroom, not fps.
• The first "no gain" A/B compared two async-ON states. The initial P4 firmware defaulted async ON, so flipping the runtime toggle never cleanly reached the synchronous 10.8 ms path — both sides of the "A/B" were ~3,790 µs, so it looked flat. The clean A/B needs the two paths genuinely distinct: toggle OFF must run the literal original encode→transmit→wait (which is why the shipped tick() has two explicit branches, tickSync/tickAsync, not one path with a stale flag — the sync branch is provably the pre-double-buffer timing, no regression).
• The rigorous baseline was a git worktree at the previous commit, flashed to the same board+config. The doubt ("we started at 93, now 76, now 48 — only lower numbers?!") was three different quantities wearing the label "fps": 93 = driver-only sync fps (1e6/10.8ms), 76 = system fps async, 48 = system fps sync. To kill it, git worktree add <tmp> <prev-commit>, build, flash, measure — same meters, same board. Result: the previous commit was always 48 fps overall (never 76 or 93 as a whole-system number), and today's async-OFF reproduced it to the microsecond (10,820 µs vs 10,787). No regression ever existed; the confusion was entirely meter-mixing. General: when a number looks like a regression, measure the prior commit on the identical target before touching anything — a worktree does it without disturbing the working tree.
• The fix that makes the trap un-fall-into-able: measure the wire explicitly. Added a wireUs read-only KPI = the DMA transfer duration timed start-of-busTransmit → done-callback (an in-order 2-slot completion FIFO in the platform pairs each done with its start stamp; esp_timer_get_time() is ISR-safe). It reads the pure WS2812 output floor independent of render load — live on the P4 at 16×256: "7474 µs (133 fps max)". Now the ceiling is measured, not assumed, the "impossible 264" can't masquerade as fps, and it doubles as the overclock instrument (raise the slot rate → wireUs drops directly). General: when a derived metric (fps-from-tick) can silently change what it measures under an optimization, add a metric for the underlying physical quantity so the derived one can't lie.
• The feature is real and defaults ON. 48→76 fps on the P4, no regression when off, and it costs one extra DMA buffer + one frame (~8 ms) of latency — hence OFF is the sound-reactive opt-out (allocation follows the flag: OFF allocates one buffer, so off truly costs nothing). The remaining gap to the 133 fps wire floor is the render loop itself (the effect at ~7.3 ms is the next bottleneck), which is the multicore work (Step 2) — and wireUs is now the target line to watch the system tick fall toward.

A fixed-stride per-light scratch buffer silently caps channel count — a >4-channel preset overran it and heap-corrupted into a bootloop

Setting a multi-channel preset (a moving-head / DMX fixture, or even RGBCCT at 5 channels) on a WS2812 LED driver bootlooped the SE16 (S3/LCD_CAM) with a LoadProhibited panic. The hunt and the design lesson:

• The bug: a fixed 4-byte-per-light wire scratch, assuming ≤4 channels (RGBW). The per-light encode ran correction_.apply(src, wire + lane * 4) into wire[kMaxLanes * 4] (parallel) / wire[4] (RMT), and encodeWs2812LcdSlots read wire[lane * 4 + ch] — the stride hard-coded to 4. A correction with outChannels > 4 wrote past its 4-byte slot, overflowing into adjacent lanes and off the array end → heap/stack corruption. outChannels is set from the preset's role count, so a fixture preset (Pan/Tilt/Dimmer/… = 8–16+ ch) — or RGBCCT (5) — walks straight off the end. The persisted preset re-applied every boot, so it crashed, rebooted, crashed = bootloop.
• The reproducing bootloop was the ideal debugger, not a disaster. Same panic every boot → each hypothesis got a definitive yes/no. addr2line on the backtrace + a one-line ESP_LOGW dumping the state pre-crash (st, io, buf, bytes, fifo head/tail) turned "it crashes somewhere in tx_color" into hard facts. Two decisive isolation flashes (queue_depth 2→1, then PSRAM→internal buffer) ruled OUT the new Step 1.5 double-buffer as the cause — the crash was identical. Then flashing the previous commit in a git worktree crashed too, but landed in tlsf_malloc (remove_free_block) — the unmistakable signature of a corrupted heap, i.e. a buffer overflow, present before Step 1.5. General: a crash inside the allocator on a later malloc is almost never the allocator's fault — it's an earlier overflow that trashed the free-list; find the writer, not the reader. And a reliable repro earns instrumentation, not guesswork (the anti-stalling rule cuts both ways: >2 blind fix attempts = stop, but a repro you can measure is exactly when to keep going).
• The fix is NOT a channel cap — it's sizing the scratch to the channel count (no maxed caps). The first instinct (reject outChannels > 4 with a status) was wrong: RGBCCT is already 5, and projectMM's principle is no arbitrary caps — the only limit is memory (CLAUDE.md). So wire became a heap buffer sized to kMaxLanes × outChannels (parallel) / outChannels (RMT), allocated off the hot path (in parseConfig/resizeSymbols, grows-only, freed on release), and the encoder's lane stride became the channels parameter instead of a literal 4. Now any channel count lays out without overrun; the driver drives a 16-channel fixture, it doesn't refuse it. Heap in the hot path? No: allocation is off the hot path (once per config change), tick() only reads the member pointer — same pattern NetworkSendDriver's corrected_ and the DMA buffer already use, so no per-frame alloc and no cache penalty at these sizes (tens–hundreds of bytes stays in L1). Verified end-to-end: the SE16, still holding the multi-channel preset that bootlooped it, now boots clean and reports driving 2304 of 16384 lights.
• General: a fixed-stride scratch is a silent cap. Any buf[N * FIXED_STRIDE] where the real element size is runtime-derived is a maxed cap hiding as a buffer — it works until an input exceeds the stride, then it's a memory-corruption bug, not a clean rejection. Either size it to the actual stride (off the hot path) or assert the input fits; never let it overrun. The tell that this class of bug exists: a driver that assumes RGB(W) but reads its channel count from a user-settable preset.

Six dead hypotheses on one bug: what a debugging session looks like when the method is broken

The 74HCT595 shift-register transport bug (frames reaching the strands garbled, thousands of GDMA lli full mount failures) survived six confident explanations in a single session. Every one of them explained the symptom; every one died on contact with a real test. The bug is still open — but the method failure is the transferable lesson, and it is worth more than the eventual fix.

The corpse pile, in order: (1) the descriptor pool is too small → doubled it, no change; (2) the DMA never hands descriptors back → refuted, avail climbs steadily; (3) an off-by-one in the frame's node count → refuted, a fresh bus mounts its full count fine; (4) a silent S3 FIFO underrun → built on the S3's inability to report underruns, i.e. on the absence of evidence, and falsified by a P4 (which can report) running the identical frame perfectly with zero underruns; (5) the frame must live in internal RAM, not PSRAM → refuted twice (the P4 works from PSRAM; the S3 failed from internal RAM) and, worse, a dead end by design — internal DMA RAM is ~110 KB, so it would cap the feature below what already works, sawing off the branch the driver stands on; (6) PSRAM is too slow → never plausible, the board's octal PSRAM has bandwidth to spare, and one look at the datasheet number would have killed it before any code was written.

What actually went wrong, and the rules that fall out:

• A theory that "explains everything" is not evidence. Each hypothesis was adopted because it fit the symptom, then code was changed to act on it, then the bench was read through the theory. That is the loop that produces six deaths. The discipline: before changing code, name the measurement that would refute the hypothesis, and take it. If no such measurement exists, the hypothesis is not yet worth acting on.
• Absence of evidence is not evidence. The "silent underrun" story existed only because the S3 has no underrun interrupt. A chip that cannot report X is not telling you X is happening. The moment a control condition appeared (a P4, which does report), the theory evaporated.
• Check the datasheet number before theorising about it. "PSRAM is too slow" cost real time and was refutable in ten seconds by looking up the octal-PSRAM bandwidth against the 26.67 MB/s the frame needs.
• A metric you never validated is not a meter. The GDMA error count was used as the success signal for most of the session — and then the product owner reported visually clean, flicker-free strands in a state that still logged 121 mount failures. The proxy and the reality had come apart, which means every "confirmed" and "no change" read off that proxy was worthless. Validate a proxy against ground truth once, early, or don't lean on it.
• The PO's eyes found in one sentence what six theories missed. "I put asyncTransmit off and it's MUCH better" and, later, a blind sweep of ledsPerPin landing exactly on the allocator's internal/PSRAM boundary (96 good / 128 bad, against a ~38 KB largest free internal block) — both were better data than anything the agent produced. When a human at the bench reports a reproducible observation, that is the primary instrument; the logs are supporting evidence. (This is the Invite the product owner to test — then STOP and wait rule earning its place: the rule exists because the agent kept racing past the one measurement that mattered.)
• A fix that shrinks the feature below its own baseline is not a fix. Internal-RAM-first "worked" and was nearly shipped as the answer — while capping a shift display at ~1,500 lights, fewer than direct mode already drives. It ships as an explicitly-labelled stopgap, and the PO's push-back ("not scalable — check you're not running in a very bad circle") is what caught it.
• Prune the record as hypotheses die. docs/history/shift-register-driver-analysis.md § 7.5 had accumulated three successive "the mechanism is X" write-ups, each stated with confidence and each wrong — actively poisoning the next attempt. It is now split into what is TRUE (measured) and what is FALSE (guessed, then refuted — do not re-derive). A design doc that records dead theories as findings is worse than no doc.

The one thing that is solid after all that: the frame renders correctly from internal RAM and incorrectly from PSRAM on the S3 — established by the PO's blind sweep, not by any of the reasoning above — while the same PSRAM frame runs perfectly on a P4. The mechanism remains unknown, and the next attempt starts from the PO's asyncTransmit observation, not from a seventh theory.

A device that loses WiFi must climb back by itself — three stacked bugs made a dropout permanent

Two bench boards were found rendering happily at full frame rate with no IP and no retry, one of them with POWERON as its last reset reason — i.e. it had lost its network at some point and simply never come back. In a light installation (a controller in a ceiling) that is a brick that needs a ladder. Three independent defects had to line up:

• ESP-IDF does not auto-reconnect, and nothing was calling esp_wifi_connect(). The WIFI_EVENT_STA_DISCONNECTED handler logged the event, cleared a flag, and stopped. Espressif's own wifi_station example calls esp_wifi_connect() right there; without it a single dropped association is permanent. (Do not vTaskDelay to pace the retry — the handler runs on IDF's event-loop task, which also carries the Ethernet and IP events; blocking it stalls the whole stack. The driver's own ~1–2 s association timeout paces the retries for free.)
• A single failed poll tore down a working STA. State::ConnectedSta fell straight to AP mode the first time wifiStaConnected() read false — no grace period. One lost beacon and the device abandoned a perfectly good network. Now it gets the same 10 s grace State::WaitingSta already used for the initial connect (one named constant, kStaGraceMs, shared by both — the question is identical, so the answer should be too).
• AP mode was a dead end. The fallback called wifiStaStop(), which deinits the STA radio — so wifiStaConnected() could never become true again, and State::AP's only upgrade check was… wifiStaConnected(). The device would sit on its own AP forever. It now retries STA every 60 s while parked in AP (long, because each attempt bounces the AP and the causes it recovers from — a rebooting router, a device carried back into range — play out over minutes).

General: any fallback state must carry a path back to the state it fell from, and that path must not depend on a signal the fallback itself made unreachable. Grep for the shape: if (!connected()) { stop(); startFallback(); } where the fallback's only exit test is connected().

The catalog was right and the reader was wrong: a silent lookup miss cost a board its network

The hpwit shift-register board could associate and was then dropped by the AP within a second, repeatedly — chased for an hour as a router problem, a DHCP-lease problem, an RF problem. The actual cause: it was transmitting at full 20 dBm because its TX-power cap was never applied, and the cap was never applied because improv_provision.py looked for it in the wrong place:

cap = entry.get("controls", {}).get("Network", {}).get("txPowerSetting")   # no entry has a flat `controls`

Every catalog entry stores per-module settings in a modules list ({"type": "NetworkModule", "controls": {"txPowerSetting": 8}}), so the chained .get() resolved to None — silently, for every board, forever. deviceModels.json had the correct value the whole time.

General: a chain of dict.get(k, {}) cannot fail loudly — it turns a schema mismatch into a None and a missing feature. When the value is load-bearing (a TX-power cap, a pin map, a baud rate), either assert it was found or log what was looked up and what was there. And when a device misbehaves in a way its catalog entry claims to prevent, suspect the reader before the hardware. (The PO's question — "did you inject txpower 8? the board needs that!" — found it in one line.)

The shift-ring "wedge" was two mundane bugs wearing a scary mask — total-free vs largest-block, and a double-rebuild

The MoonI80 shift ring "wedged" after a loopback ON→OFF (and after any control edit): status stuck at "output stalled — the bus is not delivering frames," LEDs dark until a reboot — a no-reboot-principle violation chased for two sessions with theories about stale LCD_CAM shift-clock state, a detached GDMA interrupt, and an unreleased clock-source enable. Three prior speculative fixes (a busBusy_ gate, adding loopbackTest to affectsPrepare, moving reinit() out of onControlChanged) all failed. Bench instrumentation (an ISR-bumped EOF counter + printf at each destroy/create) found it was two ordinary bugs, neither peripheral-level:

• InternalFits tested total free, not the largest contiguous block. A 144 KB shift frame (16 strands × 128) reported "fits internal" because total internal DMA free was megabytes — but the largest block was ~62 KB. So wantsRing() returned false, the whole-frame path took over, its single contiguous 144 KB alloc failed, fell back to PSRAM, and stalled at the expander clock (the exact thing the ring exists to avoid). The heap has the space; it just isn't contiguous. General: when code allocates ONE contiguous block, gate on heap_caps_get_largest_free_block, never heap_caps_get_free_size — total-free says "there's room" while the single alloc that faces fragmentation says "no." This also produced the maddening "128 worked a bit then stalled": the answer flip-flopped with heap fragmentation around the 144 KB line.

• One control edit rebuilt the bus TWICE, and the rapid second rebuild wedged. Within a single Scheduler::prepareTree() sweep, the Drivers container's prepare() pushes every child's correction (passBufferToDrivers → rebuildCorrection → the driver's onCorrectionChanged(), which did a full reinit()), and THEN the same sweep runs the child's own prepare() → reinit() again. Two peripheral teardown+rebuilds back-to-back; the second came up before the first's ring had fired a single frame, and its GDMA EOF never fired. The fix is a dirty-gate: onCorrectionChanged() reinits only when frameBytes_ actually changed (a real RGB↔RGBW switch), so a light-count/window edit — which the correction push does NOT resize — rebuilds once, in prepare(). General: a cross-cutting "rebuild on change" that fires from two independent triggers in the same sweep must be idempotent or gated on a real delta; two reinits of a stateful peripheral in quick succession is its own failure mode, distinct from either reinit alone.

Both bugs presented identically ("output stalled, wireUs=—, reboot to fix"), which is why the peripheral-level theories stuck: the symptom was at the peripheral, the cause was two layers up in memory-fit routing and control-flow. General: a symptom that manifests in an interrupt/peripheral is not evidence the bug lives there — instrument the decision that led to the peripheral call (which path was chosen, how many times it ran) before diffing registers. The false-positive risk cuts the other way too: the bench probe flagged "WEDGE? first frame advanced EOF by 0" on every rebuild, which was just the first-frame-has-no-predecessor artifact of the probe, not a real wedge — the ring was driving fine underneath.