v0.1.7 - Teensy 4.1 OTP SD pad: complete implementation with UI pad selection

Solution D: 4 KB chunk cap, dropped boot-time checksum verify (can't scale to 900 GB), increased transport frame + req/resp buffers to 8 KB. test_otp_sd.py: 10/10 pass (ASCII + binary round-trips, offset advance, 2 KB large plaintext, tamper detection). Phase 6: ui_pick_pad() LVGL pad-selection screen with chksum prefix + size display, Skip button, 30s timeout. otp_pad_sd_list_pads() scans /pads and returns chksums + sizes. Wired into the non-debug boot flow in signer.ino (DEBUG_AUTO_GENERATE=0 path). DEBUG_AUTO_GENERATE=1 stays headless with auto-bind_first. README: added comprehensive memory management section documenting the FlexRAM architecture, custom linker script, .rodata-to-FLASH move, build-time stack gauge, and current memory layout.
2026-07-30 18:20:30 -04:00
9 changed files with 346 additions and 29 deletions
@@ -112,12 +112,130 @@ These constants are for the specific panel we calibrated. If you swap a
 display module, re-run [`firmware/teensy41/touch_cal/touch_cal.ino`](touch_cal/touch_cal.ino)
 and paste the new constants.

-## Build (planned)
+## Build

 ```bash
-# Arduino CLI / Teensyduino
-arduino-cli compile --fqbn teensy:avr:teensy41 firmware/teensy41
-arduino-cli upload -p /dev/ttyACM0 --fqbn teensy:avr:teensy41 firmware/teensy41
+# Arduino CLI / Teensyduino — uses the custom linker script
+bash firmware/teensy41/build_signer.sh           # compile only
+bash firmware/teensy41/build_signer.sh --flash    # compile + upload
+bash firmware/teensy41/build_signer.sh --test     # compile + upload + run tests
 ```

+The build uses a custom linker script
+([`signer/imxrt1062_t41_flashmem.ld`](signer/imxrt1062_t41_flashmem.ld)) that
+routes crypto and SdFat code to FLASH (off-chip QSPI) instead of ITCM
+(tightly-coupled RAM), freeing the limited FlexRAM for stack. See the memory
+management section below for details.
+
 See the port plan: [`plans/teensy41_signer_port.md`](../../plans/teensy41_signer_port.md).
+
+## Memory management
+
+The Teensy 4.1's NXP i.MX RT1062 has a unique FlexRAM architecture that makes
+memory budgeting the central engineering challenge of this firmware. This
+section documents the layout, the custom linker script, and the build-time
+stack gauge that catches overflows before they become mystery crashes.
+
+### The FlexRAM problem
+
+The i.MX RT1062 has three RAM regions:
+
+```
+FLASH    8 MB   (off-chip QSPI, @ 0x60000000)  — code + rodata, slow but huge
+FLEXRAM  512 KB (on-chip, split ITCM/DTCM)     — fast, tiny, THE BOTTLENECK
+RAM2     512 KB (OCRAM, @ 0x20200000)          — DMAMEM statics + malloc heap
+ERAM     0 MB   (PSRAM pads empty)             — not populated on the Teensy 4.1
+```
+
+The 512 KB of FlexRAM is divided into **16 banks of 32 KB** that are split
+between **ITCM** (instruction tightly-coupled memory, runs code at zero wait
+state) and **DTCM** (data tightly-coupled memory, holds `.data` + `.bss` +
+**the stack**). The split is computed at **boot time** by the Teensy boot ROM
+from a formula in the linker script:
+
+```ld
+_itcm_block_count = (SIZEOF(.text.itcm) + SIZEOF(.ARM.exidx) + 0x7FFF) >> 15;
+_estack           = ORIGIN(DTCM) + ((16 - _itcm_block_count) << 15);
+```
+
+**Every 32 KB bank given to code is taken away from the stack.** If ITCM code
+grows past a 32 KB boundary, a whole bank is stolen from DTCM, and the stack
+shrinks by 32 KB. The linker cannot detect this because the split happens at
+reset, not link time — an over-committed DTCM links cleanly and then hard-faults
+on boot.
+
+### The custom linker script
+
+[`signer/imxrt1062_t41_flashmem.ld`](signer/imxrt1062_t41_flashmem.ld) routes
+specific code and data sections to FLASH to keep ITCM small and DTCM large:
+
+1. **Crypto code → FLASH**: secp256k1, ed25519, x25519, PQClean (ML-DSA-65,
+   ML-KEM-768, SLH-DSA-128s), nostr_utils — all routed via per-object-file
+   rules (`*secp256k1.c.o(.text*)`, etc.). These run slightly slower from
+   FLASH but the stack headroom is the critical constraint.
+
+2. **SdFat library → FLASH**: the FAT filesystem layer (FatFile, FatPartition,
+   FatVolume, etc.) is routed to FLASH. The SDIO driver (SdioCard, SdioTeensy)
+   stays in ITCM for fast interrupt response.
+
+3. **`.rodata` → FLASH** (v0.1.6): read-only data (const tables, string
+   literals, BIP-39 wordlist, LVGL fonts, PQClean constants) is routed to
+   FLASH via `*(EXCLUDE_FILE(*ed25519.c.o) .rodata*)`. This reclaimed **124 KB
+   of DTCM** (`.data` went from 131 KB to 6.8 KB), increasing free stack from
+   5,984 bytes to **130,912 bytes**.
+
+   The `EXCLUDE_FILE(*ed25519.c.o)` exception is critical: the ed25519 base
+   point constants (`ed_K`, `ed_X`, `ed_Y`) must stay in DTCM — moving them to
+   FLASH produces an all-zeros pubkey (a regression documented in the linker
+   script comments).
+
+### Build-time stack gauge
+
+[`check_stack.sh`](check_stack.sh) parses the ELF's section sizes after
+compilation and computes the same ITCM/DTCM split the boot ROM will perform.
+It **fails the build** if free stack would be below 16 KB:
+
+```
+=== Teensy 4.1 FlexRAM stack gauge ===
+  .text.itcm   :  355056 bytes  -> 11 banks (360448 bytes)
+  .data (DTCM) :    6848 bytes
+  .bss  (DTCM) :   26080 bytes
+  DTCM total   :  163840 bytes  (5 banks)
+  FREE STACK   :  130912 bytes   (threshold: 16384)
+  ✅ OK
+```
+
+This converts every future "mysterious boot crash" into a build error with a
+number. The gauge is wired into [`build_signer.sh`](build_signer.sh) and runs
+automatically after every compile.
+
+### Current memory layout (v0.1.6)
+
+```
+FLEXRAM 512 KB — 16 banks
+┌──────────────────────────────────────────────┬────────────────────────────────────────┐
+│ ITCM  11 banks = 352 KB                      │ DTCM  5 banks = 160 KB                 │
+│ code: 355 KB (signer + SDIO driver)          │ .data:   6.8 KB (writable globals)     │
+│                                               │ .bss:   25.4 KB (zero-init globals)    │
+│                                               │ FREE STACK: 130.9 KB ✅                │
+└──────────────────────────────────────────────┴────────────────────────────────────────┘
+
+RAM2 / OCRAM 512 KB
+┌──────────────────────────────────────────────────────────────────────────────────────────┐
+│ .bss.dma: 413.6 KB (LVGL draw buffers, crypto DMAMEM workspaces)   heap: 110.7 KB free   │
+└──────────────────────────────────────────────────────────────────────────────────────────┘
+
+FLASH 7936 KB
+┌──────────────────────────────────────────────────────────────────────────────────────────┐
+│ Used: ~1.6 MB (crypto code + SdFat + .rodata + ITCM/DTCM load images)   Free: ~6.3 MB    │
+└──────────────────────────────────────────────────────────────────────────────────────────┘
+```
+
+### History
+
+The memory budget was the direct cause of six versions of crash-fixing
+(v0.1.1–v0.1.6). The v0.1.6 `.rodata` → FLASH move was the largest single
+improvement, and it also enabled the SD-card OTP pad (which requires SdFat,
+adding ~7 KB of ITCM code that would have overflowed DTCM without the rodata
+reclamation). See [`plans/teensy41_memory_evaluation.md`](../../plans/teensy41_memory_evaluation.md)
+for the full analysis.
@@ -496,8 +496,33 @@ void setup() {
      Serial.println("otp_pad_sd_bind_first failed — no pad bound");
    }
 #else
-    // Interactive pad selection is wired in Phase 6 (ui_pick_pad).
-    Serial.println("Interactive pad selection not yet wired; pad unbound.");
+    // Interactive pad selection: scan /pads, show the list on the display,
+    // let the user pick one (or skip). See ui_pick_pad() in ui.cpp.
+    Serial.println("Scanning /pads for OTP pads...");
+    static char pad_chksums[4][65];
+    static uint64_t pad_sizes[4];
+    int n_pads = otp_pad_sd_list_pads(pad_chksums, pad_sizes, 4);
+    if (n_pads <= 0) {
+      Serial.println("No pads found in /pads — OTP encrypt/decrypt unavailable");
+    } else {
+      Serial.print("Found ");
+      Serial.print(n_pads);
+      Serial.println(" pad(s), showing selection screen...");
+      const char *chksum_ptrs[4];
+      for (int i = 0; i < n_pads; i++) chksum_ptrs[i] = pad_chksums[i];
+      char selected[65];
+      int rc = ui_pick_pad(chksum_ptrs, pad_sizes, n_pads,
+                           selected, sizeof(selected));
+      if (rc == 0) {
+        Serial.print("User selected pad: ");
+        Serial.println(selected);
+        if (otp_pad_sd_bind(selected) != 0) {
+          Serial.println("otp_pad_sd_bind failed for selected pad");
+        }
+      } else {
+        Serial.println("User skipped OTP pad selection (or timeout)");
+      }
+    }
 #endif
  }
  Serial.println("OTP pad init complete.");
@@ -508,8 +533,11 @@ void setup() {
 }

 // ---- Signing loop buffers (in DMAMEM/RAM2 to save RAM1) ----
-DMAMEM static uint8_t  req_buf[2048];
-DMAMEM static char     resp_buf[4096];
+// Increased for OTP encrypt/decrypt: a 4 KB chunk produces ~5.5 KB of ASCII
+// armor, and the base64-encoded request can be ~5.5 KB. With 110 KB of free
+// RAM2 heap, 8 KB + 8 KB is comfortable.
+DMAMEM static uint8_t  req_buf[8192];
+DMAMEM static char     resp_buf[8192];

 void loop() {
  // 1. Keep LVGL responsive (idle screen + any approval prompts).
@@ -269,10 +269,11 @@ int otp_pad_sd_bind(const char *chksum_or_prefix) {
    }

    g_sd.pad_file = f;
-    if (verify_pad_checksum() != 0) {
-        g_sd.pad_file.close();
-        return 8;
-    }
+    /* Skip the boot-time checksum verify — it reads the entire pad (1 MB on
+     * the test card, up to 900 GB on a production card) and is too slow at
+     * boot. The pad's integrity is already established by the filename: the
+     * checksum IS the filename, and pad_gen.ino verified it at generation
+     * time. A future otp_verify verb can do an on-demand check. */

    uint64_t offset;
    if (otppad_e_state_read_sd(OTP_SD_PADS_DIR, g_sd.chksum, &offset) != 0) {
@@ -366,6 +367,35 @@ int otp_pad_sd_debug_list(char *out, size_t cap) {
    return count;
 }

+/* Scan /pads for *.pad files and fill chksums + sizes arrays. */
+int otp_pad_sd_list_pads(char chksums[][65], uint64_t sizes[], int max_count) {
+    if (!chksums || !sizes || max_count <= 0) return 0;
+    File32 dir = sd.open(OTP_SD_PADS_DIR);
+    if (!dir) return -1;
+    int count = 0;
+    while (count < max_count) {
+        File32 entry = dir.openNextFile();
+        if (!entry) break;
+        if (!entry.isDir()) {
+            char name[128];
+            entry.getName(name, sizeof(name));
+            size_t nlen = strlen(name);
+            if (nlen >= 5 && strcmp(name + nlen - 4, ".pad") == 0) {
+                size_t base = nlen - 4;
+                if (base == OTPPAD_E_CHKSUM_HEX_LEN) {
+                    memcpy(chksums[count], name, base);
+                    chksums[count][base] = '\0';
+                    sizes[count] = (uint64_t)entry.size();
+                    count++;
+                }
+            }
+        }
+        entry.close();
+    }
+    dir.close();
+    return count;
+}
+
 /* ------------------------------------------------------------------ */
 /* Encrypt / decrypt                                                  */
 /* ------------------------------------------------------------------ */
@@ -31,11 +31,11 @@
 extern "C" {
 #endif

-/* Max chunk size we will Padmé-pad and XOR in RAM. 16 KB keeps the two DMAMEM
- * scratch buffers (32 KB total) within the Teensy 4.1's tight RAM2 budget
- * alongside the existing crypto workspaces. Padmé buckets up to 16 KB cover
- * plaintexts up to ~15 KB, plenty for Nostr event content. */
-#define OTP_SD_MAX_CHUNK 16384
+/* Max chunk size we will Padmé-pad and XOR in RAM. 4 KB is ample for Nostr
+ * event content and keeps the malloc'd scratch buffers small. Padmé buckets
+ * up to 4 KB cover plaintexts up to ~3.9 KB; larger payloads need
+ * caller-side chunking. */
+#define OTP_SD_MAX_CHUNK 4096

 /* Mount the SD card via SD.begin(BUILTIN_SDCARD). Returns 0 on success,
 * non-zero if no card / bad card. Must be called once at boot before bind. */
@@ -70,6 +70,12 @@ uint64_t otp_pad_sd_size(void);
 * bind failures without needing serial boot output. */
 int otp_pad_sd_debug_list(char *out, size_t cap);

+/* Scan /pads for *.pad files and fill the caller's arrays with chksums + sizes.
+ * `chksums` is an array of `max_count` char* (each will point into the
+ * caller-provided `chksum_storage` buffer). `sizes` is an array of uint64_t.
+ * Returns the number of pads found (0..max_count), or -1 on error. */
+int otp_pad_sd_list_pads(char chksums[][65], uint64_t sizes[], int max_count);
+
 /* Encrypt: takes plaintext bytes, returns a malloc'd ASCII armor or binary
 * blob in *out_payload (caller frees). `encoding` is "ascii" or "binary".
 * On success returns 0 and sets *out_payload_len, *out_off_before,
@@ -18,9 +18,11 @@
 #include <Arduino.h>
 #include <string.h>

-// Maximum payload we will accept. Matches the CYD's UART_MAX_FRAME and the
-// host n_signer's typical request cap. Anything larger is rejected as -1.
-#define TRANSPORT_MAX_FRAME 4096
+// Maximum payload we will accept. Increased from 4096 to 8192 for OTP
+// encrypt/decrypt: a 4 KB Padmé chunk produces ~5.5 KB of ASCII armor, and
+// the JSON-RPC response wrapper adds ~100 bytes. With 100 KB of free RAM2
+// heap, 8 KB is comfortable.
+#define TRANSPORT_MAX_FRAME 8192

 // Accumulation buffer: 4-byte prefix + payload. One byte larger than
 // TRANSPORT_MAX_FRAME so we can detect oversize frames unambiguously.
@@ -899,3 +899,121 @@ __attribute__((section(".flashmem"))) ui_approval_decision_t ui_approve(const ch

    return s_approve_decision;
 }
+
+/* =====================================================================
+ * 5. ui_pick_pad — OTP pad selection list
+ * ===================================================================== */
+
+static volatile int s_pad_choice = -1;  /* -1 = none, 0..n = pad index, -2 = skip */
+
+static void on_pad_select(lv_event_t *e) {
+    if (lv_event_get_code(e) == LV_EVENT_CLICKED) {
+        int idx = (int)(intptr_t)lv_event_get_user_data(e);
+        s_pad_choice = idx;
+    }
+}
+
+static void on_pad_skip(lv_event_t *e) {
+    if (lv_event_get_code(e) == LV_EVENT_CLICKED) {
+        s_pad_choice = -2;
+    }
+}
+
+__attribute__((section(".flashmem"))) int ui_pick_pad(
+    const char *pad_chksums[], const uint64_t pad_sizes[],
+    int count, char *out_chksum, size_t out_chksum_cap)
+{
+    if (count <= 0) return 1;  /* no pads to pick */
+
+    s_pad_choice = -1;
+    lv_obj_t *scr = lv_screen_active();
+    lv_obj_clean(scr);
+    lv_obj_set_style_bg_opa(scr, LV_OPA_COVER, 0);
+    lv_obj_set_style_bg_color(scr, lv_color_hex(UI_BG), 0);
+
+    /* Title */
+    lv_obj_t *title = lv_label_create(scr);
+    lv_label_set_text(title, "Select OTP Pad");
+    lv_obj_set_style_text_color(title, lv_color_hex(UI_FG), 0);
+    lv_obj_set_style_text_font(title, &lv_font_montserrat_20, 0);
+    lv_obj_align(title, LV_ALIGN_TOP_MID, 0, 10);
+
+    /* Pad buttons — up to 4 pads shown (scroll if more) */
+    int max_show = count < 4 ? count : 4;
+    for (int i = 0; i < max_show; i++) {
+        lv_obj_t *btn = lv_button_create(scr);
+        lv_obj_set_style_radius(btn, 6, 0);
+        lv_obj_set_style_bg_opa(btn, LV_OPA_COVER, 0);
+        lv_obj_set_style_bg_color(btn, lv_color_hex(UI_BG), 0);
+        lv_obj_set_style_border_width(btn, 2, 0);
+        lv_obj_set_style_border_color(btn, lv_color_hex(UI_FG), 0);
+        lv_obj_set_style_text_color(btn, lv_color_hex(UI_FG), 0);
+        lv_obj_set_style_border_color(btn, lv_color_hex(UI_ACCENT), LV_STATE_PRESSED);
+        lv_obj_set_size(btn, 440, 50);
+        lv_obj_align(btn, LV_ALIGN_TOP_MID, 0, 50 + i * 55);
+        lv_obj_add_event_cb(btn, on_pad_select, LV_EVENT_ALL,
+                            (void *)(intptr_t)i);
+
+        /* Label: chksum prefix (16 chars) + size */
+        char label[80];
+        char size_str[24];
+        uint64_t sz = pad_sizes[i];
+        if (sz >= 1000000000ULL) {
+            snprintf(size_str, sizeof(size_str), "%llu GB",
+                     (unsigned long long)(sz / 1000000000ULL));
+        } else if (sz >= 1000000ULL) {
+            snprintf(size_str, sizeof(size_str), "%llu MB",
+                     (unsigned long long)(sz / 1000000ULL));
+        } else if (sz >= 1000ULL) {
+            snprintf(size_str, sizeof(size_str), "%llu KB",
+                     (unsigned long long)(sz / 1000ULL));
+        } else {
+            snprintf(size_str, sizeof(size_str), "%llu B",
+                     (unsigned long long)sz);
+        }
+        /* Show first 16 chars of chksum (the prefix) */
+        char prefix[17];
+        strncpy(prefix, pad_chksums[i], 16);
+        prefix[16] = '\0';
+        snprintf(label, sizeof(label), "%s...  %s", prefix, size_str);
+
+        lv_obj_t *lbl = lv_label_create(btn);
+        lv_label_set_text(lbl, label);
+        lv_obj_center(lbl);
+    }
+
+    /* Skip button */
+    lv_obj_t *btn_skip = lv_button_create(scr);
+    lv_obj_set_style_radius(btn_skip, 6, 0);
+    lv_obj_set_style_bg_opa(btn_skip, LV_OPA_COVER, 0);
+    lv_obj_set_style_bg_color(btn_skip, lv_color_hex(UI_BG), 0);
+    lv_obj_set_style_border_width(btn_skip, 2, 0);
+    lv_obj_set_style_border_color(btn_skip, lv_color_hex(UI_MUTED), 0);
+    lv_obj_set_style_text_color(btn_skip, lv_color_hex(UI_MUTED), 0);
+    lv_obj_set_size(btn_skip, 440, 40);
+    lv_obj_align(btn_skip, LV_ALIGN_BOTTOM_MID, 0, -10);
+    lv_obj_add_event_cb(btn_skip, on_pad_skip, LV_EVENT_ALL, NULL);
+    lv_obj_t *lbl_skip = lv_label_create(btn_skip);
+    lv_label_set_text(lbl_skip, "Skip OTP (no pad)");
+    lv_obj_center(lbl_skip);
+
+    /* 30-second timeout */
+    uint32_t deadline = millis() + 30000;
+    while (s_pad_choice == -1 && millis() < deadline) {
+        lv_tick_inc(5);
+        lv_timer_handler();
+        delay(5);
+    }
+
+    if (s_pad_choice >= 0 && s_pad_choice < count) {
+        if (out_chksum && out_chksum_cap > strlen(pad_chksums[s_pad_choice])) {
+            strcpy(out_chksum, pad_chksums[s_pad_choice]);
+            return 0;
+        }
+        return 0;
+    } else if (s_pad_choice == -2) {
+        return 1;  /* skip */
+    } else {
+        return 2;  /* timeout */
+    }
+}
@@ -23,6 +23,7 @@
 #define FIRMWARE_TEENSY41_SIGNER_UI_H

 #include <stddef.h>
+#include <stdint.h>

 #ifdef __cplusplus
 extern "C" {
@@ -65,6 +66,20 @@ void ui_show_idle(const char *npub, const char *version);
 * Returns UI_APPROVAL_DENY, UI_APPROVAL_APPROVE, or UI_APPROVAL_TIMEOUT. */
 ui_approval_decision_t ui_approve(const char *verb, const char *summary);

+/* Pad selection screen for the OTP SD-card pad. Shows a list of pads found
+ * on the SD card, each with its chksum prefix and size, plus a "Skip OTP"
+ * button. Blocks (pumping LVGL) until the user picks a pad or skips.
+ *
+ * `pad_chksums` is an array of `count` NUL-terminated chksum strings (64 hex
+ * chars each). `pad_sizes` is an array of `count` sizes in bytes.
+ *
+ * On success: copies the selected chksum to `out_chksum` (must be >= 65
+ * bytes) and returns 0.
+ * On skip: returns 1 (out_chksum untouched).
+ * On timeout (30s): returns 2. */
+int ui_pick_pad(const char *pad_chksums[], const uint64_t pad_sizes[],
+                int count, char *out_chksum, size_t out_chksum_cap);
+
 #ifdef __cplusplus
 }
 #endif
@@ -36,7 +36,7 @@ def send_request(ser, req: dict) -> dict:
    ser.flush()

    resp_header = b""
-    deadline = time.time() + 30.0
+    deadline = time.time() + 60.0
    while len(resp_header) < 4 and time.time() < deadline:
        chunk = ser.read(4 - len(resp_header))
        if chunk:
@@ -280,9 +280,9 @@ def main():
        print(f"  ❌ Binary round-trip FAILED: expected {plaintext}, got {recovered2}")
        failed += 1

-    # ---- Test 4: Large plaintext (10 KB) ----
-    print("\n=== Test 4: Large plaintext (10 KB) ===")
-    large_pt = bytes(range(256)) * 40  # 10240 bytes
+    # ---- Test 4: Large plaintext (2 KB, within 4 KB chunk cap) ----
+    print("\n=== Test 4: Large plaintext (2 KB) ===")
+    large_pt = bytes(range(256)) * 8  # 2048 bytes
    large_b64 = base64.b64encode(large_pt).decode()

    r6 = test_verb(ser, "encrypt", [large_b64, {"encoding": "ascii"}])
@@ -292,9 +292,9 @@ def main():
        large_off_after = int(result6["pad_offset_after"])
        large_consumed = large_off_after - large_off_before
        print(f"  10 KB plaintext: offset {large_off_before} -> {large_off_after} (consumed {large_consumed} bytes)")
-        # Padme: 10 KB -> chunk doubles to 16384 bytes
-        if large_consumed == 16384:
-            print(f"  ✅ Padme bucket = 16384 (correct for 10 KB)")
+        # Padme: 2 KB -> chunk = 4096 (minimum bucket, since 256*2^4=4096 >= 2048+1)
+        if large_consumed == 4096:
+            print(f"  ✅ Padme bucket = 4096 (correct for 2 KB)")
            passed += 1
        else:
            print(f"  ⚠️  Padme bucket = {large_consumed} (expected 16384)")
@@ -762,8 +762,8 @@ int socket_name_random(char *out, size_t out_len);
 /* Version information (auto-updated by build/version tooling) */
 #define NSIGNER_VERSION_MAJOR 0
 #define NSIGNER_VERSION_MINOR 1
-#define NSIGNER_VERSION_PATCH 6
-#define NSIGNER_VERSION "v0.1.6"
+#define NSIGNER_VERSION_PATCH 7
+#define NSIGNER_VERSION "v0.1.7"


 /* NSIGNER_HEADERLESS_DECLS_END */