n_signer/plans/teensy41_otp_sd_pad.md

# Plan: Real SD-card OTP pad for the Teensy 4.1 signer

## Goal

Replace the Teensy 4.1 firmware's throwaway HKDF-derived 1024-byte in-RAM OTP
pad ([`firmware/teensy41/signer/src/otp_pad.cpp`](firmware/teensy41/signer/src/otp_pad.cpp))
with a **real SD-card pad** that reads `<chksum>.pad` / `<chksum>.state` from
the Teensy's built-in SD slot, bit-compatible with the `otp` project and the
host `n_signer` ([`src/otp_pad.c`](src/otp_pad.c)) via [`libotppad`](libotppad/libotppad.h).

This means the firmware's `encrypt`/`decrypt` verbs must gain:
- Padmé padding (ISO/IEC 9797-1 Method 2) with exponential bucketing.
- ASCII armored output (`-----BEGIN OTP MESSAGE-----` + base64) **and** binary
  `.otp` output (magic `OTP\0` + 58-byte header), selected per request via an
  `encoding` option — matching the host's `otp_encrypt`/`otp_decrypt` verbs.
- Per-pad offset persistence in `<chksum>.state` (atomic write-temp-then-rename
  on the SD card).
- Pad binding at startup (mount SD, find pad by chksum, verify checksum, read
  offset).

The interactive "look for existing pads and ask the user to confirm one" UI
flow is the **last** phase. Until then, a debug auto-bind path lets us test
encrypt/decrypt round-trips over USB CDC without touching the screen.

## Non-goals

- Multi-device offset coordination (deferred in
  [`plans/otp_nostr_integration.md`](plans/otp_nostr_integration.md)).
- Nostr kind-30078 event wrapping (caller's job, same as host).
- Production pad entropy: the test pad is `/dev/urandom`-sourced, fine for
  validating the format and round-trips.

## Architecture

```mermaid
flowchart LR
  SD[(SD card exFAT<br/>pads/chksum.pad<br/>pads/chksum.state)] -->|SD.begin BUILTIN_SDCARD| M[otp_pad_sd.cpp<br/>mount + bind + seek/read]
  M -->|otppad_embedded| L[libotppad port<br/>XOR, base64, Padme, armor, checksum]
  L --> E[otp_pad_encrypt/decrypt<br/>encoding: ascii or binary]
  E --> D[dispatch.cpp<br/>encrypt/decrypt verbs]
  D -->|USB CDC framed JSON-RPC| Host[test_otp_sd.py]
  Boot[signer.ino boot flow] -->|after seed| M
  Boot -.->|last phase| UI[ui_pick_pad<br/>LVGL list + confirm]
```

### Module layout

- [`firmware/teensy41/signer/src/otp_pad_sd.h`](firmware/teensy41/signer/src/otp_pad_sd.h) —
  new public API: `otp_pad_sd_mount()`, `otp_pad_sd_bind(chksum)`,
  `otp_pad_sd_bind_first()` (debug), `otp_pad_sd_unbind()`,
  `otp_pad_sd_encrypt(pt, len, encoding, out, out_len)`,
  `otp_pad_sd_decrypt(input, len, encoding, out, out_len)`,
  `otp_pad_sd_ready()`, `otp_pad_sd_chksum()`, `otp_pad_sd_offset()`,
  `otp_pad_sd_size()`.
- [`firmware/teensy41/signer/src/otp_pad_sd.cpp`](firmware/teensy41/signer/src/otp_pad_sd.cpp) —
  implementation over the Arduino `SD` library (4-bit SDMMC,
  `BUILTIN_SDCARD`). Holds the bound pad's `File` (read-only) + state in
  file-static globals, mirroring [`src/otp_pad.c`](src/otp_pad.c)'s
  `otp_pad_state_t`.
- [`firmware/teensy41/signer/src/otppad_embedded.h`](firmware/teensy41/signer/src/otppad_embedded.h) /
  `.cpp` — a Teensy/Arduino-friendly port of the format-critical functions from
  [`libotppad/libotppad.c`](libotppad/libotppad.c): `otppad_xor`,
  `otppad_base64_encode/decode`, `otppad_chunk_size`, `otppad_pad_apply/remove`,
  `otppad_armor_parse/generate`, `otppad_checksum` (streaming, over a `File*`),
  `otppad_state_read/write` (over SD `File`). No POSIX `FILE*`/`malloc`/`strtok`
  dependencies that don't exist on Teensy; uses `malloc`/`free` (available via
  newlib) and Arduino `String`/manual parsing where needed. **Bit-identical
  output to libotppad** — same constants, same byte order, same header layout.

### Wire format (align with host `otp_encrypt`/`otp_decrypt`)

The existing Teensy `encrypt`/`decrypt` verbs return raw base64 XOR +
`pad_offset_before`/`pad_offset_after`. The host verbs return ASCII armor or a
binary `.otp` blob with the offset embedded. To be bit-compatible and reusable
with the existing [`tools/otp_roundtrip_test.py`](tools/otp_roundtrip_test.py)
pattern, the Teensy verbs will be upgraded to match the host:

- `encrypt` params: `[plaintext_b64, {"encoding": "ascii"|"binary"}]`
  → result JSON: `{"ciphertext": "<armor or b64-of-blob>", "pad_chksum": "<64hex>", "pad_offset_before": N, "pad_offset_after": N}`.
- `decrypt` params: `[ciphertext, {"encoding": "ascii"|"binary"}]`
  → result JSON: `{"plaintext": "<b64>"}`. The offset is read from the armor
  header / binary header (no `pad_offset` option needed, matching the host).

The `algorithm: "otp"` option is kept for backward compatibility with
[`test_signer.py`](firmware/teensy41/test_signer.py) but is optional.

### Memory budget

The Teensy 4.1 has ~110 KB free heap (RAM2/DMAMEM) and ~9.6 KB free DTCM stack.
The pad is **never** loaded whole. Each request:
1. Decodes base64 plaintext into a DMAMEM scratch buffer (max chunk = 4 MB on
   host; cap at **64 KB** on Teensy to fit heap — plenty for Nostr event
   content).
2. Seeks the pad `File` to the current offset, reads exactly `chunk` bytes into
   a second DMAMEM buffer.
3. XORs in place, encodes output, zeroizes scratch, advances offset in
   `.state`.

All large buffers go in `DMAMEM` (RAM2), matching the existing crypto
workspace pattern in [`signer.ino`](firmware/teensy41/signer/signer.ino:63).

## Phased implementation

### Phase 0 — Cleanup

- [ ] Delete [`firmware/teensy41/otp_card_probe/`](firmware/teensy41/otp_card_probe/) (the throwaway probe sketch).
- [ ] Delete [`firmware/teensy41/sd_test/`](firmware/teensy41/sd_test/) (bring-up sketch, superseded).
- [ ] Confirm the smaller card is still readable by re-running the probe logic
      once inside the real firmware's SD mount (no separate sketch).

### Phase 1 — Generate a test pad on the smaller card

The Teensy's SD slot isn't accessible from the host, so the pad must be
generated on-device. Two options (pick one):

- **A. One-time pad-generator sketch** `firmware/teensy41/pad_gen/pad_gen.ino`:
  mounts the SD card, writes `<chksum>.pad` (e.g. 1 MB from the Teensy's TRNG /
  `analogRead` noise + `LibRandom` if available, else `/dev/urandom`-equivalent
  PRNG seeded from `ENTROPY` registers), computes the XOR checksum, writes
  `<chksum>.state` with `offset=32\n`. This is a **utility**, not test firmware;
  it can be deleted after the pad exists. Uses the same checksum algorithm as
  [`tools/make_test_pad.c`](tools/make_test_pad.c:39) so the pad is
  bit-compatible.
- **B. Host generation via USB reader**: if a USB SD reader is available, pop
  the card, run `make_test_pad <mount>/pads 1048576` on the host, reinsert.

Default: **A** (no USB reader assumed). The generator is clearly marked as a
setup utility and removed in Phase 0 of a future cleanup once the pad exists.

- [ ] Write `firmware/teensy41/pad_gen/pad_gen.ino` (1 MB pad, 32-byte header,
      checksum-named, `offset=32\n` state).
- [ ] Flash + run it; record the generated `<chksum>` for use in tests.

### Phase 2 — Port libotppad to the firmware (`otppad_embedded`)

- [ ] Create `otppad_embedded.h` declaring the format-critical functions.
- [ ] Port `otppad_xor`, `otppad_base64_encode/decode` (reuse the existing
      `b64_encode`/`b64_decode` in dispatch.cpp if bit-identical, else port
      libotppad's tables).
- [ ] Port `otppad_chunk_size`, `otppad_pad_apply`, `otppad_pad_remove` (Padmé).
- [ ] Port `otppad_armor_parse` / `otppad_armor_generate` (replace `strtok` with
      manual line splitting; replace `snprintf` with Arduino `sprintf`).
- [ ] Port `otppad_checksum` as a streaming function over an Arduino `File*`
      (read in 4 KB chunks, fold into 32 buckets, XOR with first 32 pad bytes).
- [ ] Port `otppad_state_read` / `otppad_state_write` over SD `File` (atomic
      write: write `<chksum>.state.tmp`, `SD.rename` over `<chksum>.state`).
- [ ] Add a host-buildable unit test
      `firmware/teensy41/signer/tests/host_test_otppad_embedded.c` that links
      `otppad_embedded.c` compiled with `HOST_TEST` against a real `FILE*`
      backend, and verifies round-trip + padding + armor + checksum against
      `libotppad` outputs (bit-compatibility check).

### Phase 3 — `otp_pad_sd.cpp` (bind + seek/read + encrypt/decrypt)

- [ ] Create `otp_pad_sd.h` with the bind/encrypt/decrypt API above.
- [ ] Implement `otp_pad_sd_mount()` — `SD.begin(BUILTIN_SDCARD)`, report
      failure over Serial.
- [ ] Implement `otp_pad_sd_bind(chksum)` — open `<chksum>.pad` read-only,
      verify checksum via `otppad_checksum`, read offset from `.state` (default
      to 32 if missing), store pad size + chksum in globals.
- [ ] Implement `otp_pad_sd_bind_first()` — scan root for `*.pad`, bind the
      first one (debug auto-bind path).
- [ ] Implement `otp_pad_sd_encrypt` — Padmé-pad, seek+read pad slice, XOR,
      encode (ascii/binary), advance offset atomically, zeroize scratch.
- [ ] Implement `otp_pad_sd_decrypt` — parse armor/binary header, seek+read pad
      slice, XOR, strip Padmé, zeroize scratch. Does **not** advance offset
      (decrypt is non-consuming, matching host).
- [ ] Implement `otp_pad_sd_unbind` — close `File`, zeroize state.
- [ ] All scratch buffers in `DMAMEM`; cap chunk at 64 KB.

### Phase 4 — Wire into dispatch + boot flow (debug auto-bind)

- [ ] In [`signer.ino`](firmware/teensy41/signer/signer.ino:262)
      `apply_mnemonic()`: after seed derivation, **remove** the
      `otp_pad_init(g_seed, ...)` HKDF call. Replace with: call
      `otp_pad_sd_mount()`; if `DEBUG_AUTO_GENERATE=1`, call
      `otp_pad_sd_bind_first()` and log the bound chksum over Serial. On
      failure, log but continue (encrypt/decrypt verbs will return
      `otp pad not bound`).
- [ ] In [`dispatch.cpp`](firmware/teensy41/signer/src/dispatch.cpp:1694)
      `encrypt`/`decrypt` verbs: replace the in-RAM `otp_pad_apply`/`seek` path
      with calls to `otp_pad_sd_encrypt`/`otp_pad_sd_decrypt`. Parse
      `encoding` option (`"ascii"` default, `"binary"`). Build the result JSON
      to match the host wire format (`ciphertext`, `pad_chksum`,
      `pad_offset_before`, `pad_offset_after` for encrypt; `plaintext` for
      decrypt). Keep `algorithm: "otp"` optional for backward compat.
- [ ] Remove the old [`otp_pad.cpp`](firmware/teensy41/signer/src/otp_pad.cpp) /
      [`otp_pad.h`](firmware/teensy41/signer/src/otp_pad.h) HKDF implementation
      (superseded by `otp_pad_sd.*`).

### Phase 5 — Test harness + hardware round-trips

- [ ] Write `firmware/teensy41/test_otp_sd.py` — over USB CDC, framed JSON-RPC:
  - `get_info` (sanity).
  - `encrypt` ascii → parse armor, `Pad-ChkSum` matches bound chksum,
    `Pad-Offset` = 32 (first call).
  - `decrypt` ascii → recovered plaintext matches.
  - `encrypt` binary → blob starts with `OTP\0`, header chksum matches,
    `pad_offset` = 32 + first chunk.
  - `decrypt` binary → recovered plaintext matches.
  - Second `encrypt` ascii → `Pad-Offset` advanced by first chunk (proves
    offset persistence in `.state`).
  - Reboot the Teensy (power cycle), `encrypt` again → `Pad-Offset` continues
    from where it left off (proves `.state` survives power cycle).
  - Large plaintext (e.g. 10 KB) → Padmé bucket doubles to 16 KB, round-trip
    OK.
  - Tamper test: flip one byte in the armor base64 → decrypt returns error
    (padding removal fails) or wrong plaintext (detected).
- [ ] Run it against the flashed firmware; iterate on failures.

### Phase 6 — Interactive pad selection UI (LAST)

- [ ] Add `ui_pick_pad(pads_list, count) -> selected_chksum` to
      [`ui.h`](firmware/teensy41/signer/src/ui.h) / `ui.cpp`: an LVGL list
      screen showing each pad's chksum prefix + size + used%, with a "Use this
      pad" / "Skip OTP" choice. Blocks (pumps LVGL) until the user picks.
- [ ] In `signer.ino` boot flow, when `DEBUG_AUTO_GENERATE=0`: after
      `apply_mnemonic()`, scan the SD root for `*.pad`, build the list, call
      `ui_pick_pad()`. If the user picks one, `otp_pad_sd_bind(chksum)`. If
      "Skip OTP" or no pads found, continue without a bound pad.
- [ ] Keep `DEBUG_AUTO_GENERATE=1` → `otp_pad_sd_bind_first()` as the test path
      so Phase 5 tests still run headless.

## Test commands (Phase 5)

```bash
# Build + flash the real firmware
bash firmware/teensy41/build_signer.sh --flash

# OTP SD round-trip suite
python3 firmware/teensy41/test_otp_sd.py --port /dev/ttyACM0

# Host-side bit-compatibility check for otppad_embedded
cc -O2 -Wall -Wextra -D HOST_TEST -o host_test_otppad_embedded \
    firmware/teensy41/signer/src/otppad_embedded.c \
    firmware/teensy41/signer/tests/host_test_otppad_embedded.c -lm
./host_test_otppad_embedded
```

## Status (2026-07-28)

### Done

- **Phase 0**: Throwaway sketches deleted.
- **Phase 1**: [`pad_gen.ino`](firmware/teensy41/pad_gen/pad_gen.ino) written,
  compiled, flashed, and run on the smaller card. Generated a 1 MB TRNG-sourced
  pad `4ec4e221...b0ca78.pad` + `.state` (offset=32). The Teensy 4.1's hardware
  TRNG (`TRNG_ENT0..15` registers) works — two runs produced different pads.
- **Phase 2**: [`otppad_embedded.{h,c}`](firmware/teensy41/signer/src/otppad_embedded.h)
  ported from libotppad. Host bit-compat test
  [`host_test_otppad_embedded.c`](firmware/teensy41/signer/tests/host_test_otppad_embedded.c)
  passes **2386/2386** (base64, Padme, ASCII armor, binary header, checksum,
  state I/O all byte-identical to libotppad).
- **Phase 3**: [`otp_pad_sd.{h,cpp}`](firmware/teensy41/signer/src/otp_pad_sd.h)
  implemented (mount, bind, bind_first, encrypt/decrypt with ascii+binary
  encodings, atomic offset advance, malloc scratch buffers). Code is complete.
- **Phase 4**: [`signer.ino`](firmware/teensy41/signer/signer.ino) and
  [`dispatch.cpp`](firmware/teensy41/signer/src/dispatch.cpp) wired to the new
  `otp_pad_sd` API with host-compatible wire format. Old
  [`otp_pad.{cpp,h}`](firmware/teensy41/signer/src/otp_pad.cpp) deleted.

### BLOCKER: `<SD.h>` crashes the signer firmware

Including `<SD.h>` in the signer firmware causes an **immediate hard fault
before `setup()` runs** — no USB CDC enumeration, no serial output, no LED.
This happens with both the custom linker script
([`imxrt1062_t41_flashmem.ld`](firmware/teensy41/signer/imxrt1062_t41_flashmem.ld))
and the default Teensy linker script. The crash occurs even when every SD
function is stubbed out (only the `#include <SD.h>` is present).

The same `<SD.h>` works fine in standalone sketches:
- [`pad_gen.ino`](firmware/teensy41/pad_gen/pad_gen.ino) — mounts SD, writes a
  1 MB pad, reads it back, verifies checksum. Runs perfectly.
- The deleted `sd_test.ino` / `otp_card_probe.ino` — mounted the 1 TB card,
  listed files, wrote+read a 64-byte test file. All worked.

**Root cause (likely):** The Teensy 4.1's flexRAM is dynamically partitioned
between ITCM (code) and DTCM (data) in 32 KB blocks. The signer firmware
already uses ~377 KB of ITCM (12 blocks → 384 KB ITCM, 128 KB DTCM). The
SD/SdFat library adds ~13 KB of ITCM code, which — depending on the linker
script — either pushes ITCM to 13 blocks (reducing DTCM to 96 KB, overflowing
the 130 KB `.data` section) or doesn't change the block count but the
additional `.data`/BSS overflows DTCM. The linker does not catch this because
the flexRAM partitioning is computed at runtime by the Teensy boot ROM, not by
the linker script.

**Current workaround:** [`otp_pad_sd.cpp`](firmware/teensy41/signer/src/otp_pad_sd.cpp)
has `OTP_SD_ENABLED 0` — all SD functions are stubbed, `<SD.h>` is not included,
and the firmware boots and works normally for all non-OTP verbs. The
encrypt/decrypt verbs return `otp pad not bound (no SD pad)`.

### Path forward (to unblock)

1. **Route SdFat code to FLASH via `.flashmem`**: The custom linker script
   routes functions marked `__attribute__((section(".flashmem")))` to FLASH
   instead of ITCM. The SD/SdFat library functions are not marked `.flashmem`,
   so they land in ITCM. Options:
   - Wrap the SD includes with `#pragma GCC push_options` + `-ffunction-sections`
     + a custom section attribute via a wrapper .cpp that re-exports the SD
     calls from a `.flashmem`-marked translation unit.
   - Fork/patch SdFat to add `.flashmem` attributes (heavy).
2. **Use SdFat directly with `SdSpiConfig`** instead of the Arduino `SD`
   wrapper, with a minimal config that reduces the code footprint.
3. **Reduce the signer's own ITCM usage** to make room for the SD library's
   ~13 KB (e.g., move more crypto code to FLASH).
4. **Use the external RAM (ERAM, 32 MB at 0x70000000)** for the SD library's
   BSS/buffers by placing them in `.bss.extram` — the linker script already
   defines this section but it's currently empty.

## Risks / open questions

- **SD library flexRAM crash** (see BLOCKER above) — the main blocker.
- **exFAT rename atomicity**: `SD.rename` on SdFat exFAT should be atomic at
  the directory-entry level; verify once the crash is resolved.
- **Chunk cap**: set to 16 KB (`OTP_SD_MAX_CHUNK`) to fit RAM2; scratch buffers
  use `malloc` (heap) not static `DMAMEM` to avoid RAM2 BSS overflow.
- **Checksum over a 1 MB pad on-device**: streaming 4 KB reads, fast enough.
  On a future 900 GB pad, checksum-on-bind is impractical — add a
  skip-verify flag and only verify the first/last 4 KB for large pads.
- **Pad generation entropy**: the Teensy 4.1's hardware TRNG
  (`TRNG_ENT0..15` registers) is used directly in `pad_gen.ino` and works.