n_signer/plans/teensy41_otp_sd_pad.md

Plan: Real SD-card OTP pad for the Teensy 4.1 signer

Goal

Replace the Teensy 4.1 firmware's throwaway HKDF-derived 1024-byte in-RAM OTP pad (firmware/teensy41/signer/src/otp_pad.cpp) with a real SD-card pad that reads <chksum>.pad / <chksum>.state from the Teensy's built-in SD slot, bit-compatible with the otp project and the host n_signer (src/otp_pad.c) via libotppad.

This means the firmware's encrypt/decrypt verbs must gain:

• Padmé padding (ISO/IEC 9797-1 Method 2) with exponential bucketing.
• ASCII armored output (-----BEGIN OTP MESSAGE----- + base64) and binary .otp output (magic OTP\0 + 58-byte header), selected per request via an encoding option — matching the host's otp_encrypt/otp_decrypt verbs.
• Per-pad offset persistence in <chksum>.state (atomic write-temp-then-rename on the SD card).
• Pad binding at startup (mount SD, find pad by chksum, verify checksum, read offset).

The interactive "look for existing pads and ask the user to confirm one" UI flow is the last phase. Until then, a debug auto-bind path lets us test encrypt/decrypt round-trips over USB CDC without touching the screen.

Non-goals

• Multi-device offset coordination (deferred in plans/otp_nostr_integration.md).
• Nostr kind-30078 event wrapping (caller's job, same as host).
• Production pad entropy: the test pad is /dev/urandom-sourced, fine for validating the format and round-trips.

Architecture

flowchart LR
  SD[(SD card exFAT<br/>pads/chksum.pad<br/>pads/chksum.state)] -->|SD.begin BUILTIN_SDCARD| M[otp_pad_sd.cpp<br/>mount + bind + seek/read]
  M -->|otppad_embedded| L[libotppad port<br/>XOR, base64, Padme, armor, checksum]
  L --> E[otp_pad_encrypt/decrypt<br/>encoding: ascii or binary]
  E --> D[dispatch.cpp<br/>encrypt/decrypt verbs]
  D -->|USB CDC framed JSON-RPC| Host[test_otp_sd.py]
  Boot[signer.ino boot flow] -->|after seed| M
  Boot -.->|last phase| UI[ui_pick_pad<br/>LVGL list + confirm]

Module layout

• firmware/teensy41/signer/src/otp_pad_sd.h — new public API: otp_pad_sd_mount(), otp_pad_sd_bind(chksum), otp_pad_sd_bind_first() (debug), otp_pad_sd_unbind(), otp_pad_sd_encrypt(pt, len, encoding, out, out_len), otp_pad_sd_decrypt(input, len, encoding, out, out_len), otp_pad_sd_ready(), otp_pad_sd_chksum(), otp_pad_sd_offset(), otp_pad_sd_size().
• firmware/teensy41/signer/src/otp_pad_sd.cpp — implementation over the Arduino SD library (4-bit SDMMC, BUILTIN_SDCARD). Holds the bound pad's File (read-only) + state in file-static globals, mirroring src/otp_pad.c's otp_pad_state_t.
• firmware/teensy41/signer/src/otppad_embedded.h / .cpp — a Teensy/Arduino-friendly port of the format-critical functions from libotppad/libotppad.c: otppad_xor, otppad_base64_encode/decode, otppad_chunk_size, otppad_pad_apply/remove, otppad_armor_parse/generate, otppad_checksum (streaming, over a File*), otppad_state_read/write (over SD File). No POSIX FILE*/malloc/strtok dependencies that don't exist on Teensy; uses malloc/free (available via newlib) and Arduino String/manual parsing where needed. Bit-identical output to libotppad — same constants, same byte order, same header layout.

Wire format (align with host otp_encrypt/otp_decrypt)

The existing Teensy encrypt/decrypt verbs return raw base64 XOR + pad_offset_before/pad_offset_after. The host verbs return ASCII armor or a binary .otp blob with the offset embedded. To be bit-compatible and reusable with the existing tools/otp_roundtrip_test.py pattern, the Teensy verbs will be upgraded to match the host:

• encrypt params: [plaintext_b64, {"encoding": "ascii"|"binary"}] → result JSON: {"ciphertext": "<armor or b64-of-blob>", "pad_chksum": "<64hex>", "pad_offset_before": N, "pad_offset_after": N}.
• decrypt params: [ciphertext, {"encoding": "ascii"|"binary"}] → result JSON: {"plaintext": "<b64>"}. The offset is read from the armor header / binary header (no pad_offset option needed, matching the host).

The algorithm: "otp" option is kept for backward compatibility with test_signer.py but is optional.

Memory budget

The Teensy 4.1 has ~110 KB free heap (RAM2/DMAMEM) and ~9.6 KB free DTCM stack. The pad is never loaded whole. Each request:

1. Decodes base64 plaintext into a DMAMEM scratch buffer (max chunk = 4 MB on host; cap at 64 KB on Teensy to fit heap — plenty for Nostr event content).
2. Seeks the pad File to the current offset, reads exactly chunk bytes into a second DMAMEM buffer.
3. XORs in place, encodes output, zeroizes scratch, advances offset in .state.

All large buffers go in DMAMEM (RAM2), matching the existing crypto workspace pattern in signer.ino.

Phased implementation

Phase 0 — Cleanup

• Delete firmware/teensy41/otp_card_probe/ (the throwaway probe sketch).
• Delete firmware/teensy41/sd_test/ (bring-up sketch, superseded).
• Confirm the smaller card is still readable by re-running the probe logic once inside the real firmware's SD mount (no separate sketch).

Phase 1 — Generate a test pad on the smaller card

The Teensy's SD slot isn't accessible from the host, so the pad must be generated on-device. Two options (pick one):

• A. One-time pad-generator sketch firmware/teensy41/pad_gen/pad_gen.ino: mounts the SD card, writes <chksum>.pad (e.g. 1 MB from the Teensy's TRNG / analogRead noise + LibRandom if available, else /dev/urandom-equivalent PRNG seeded from ENTROPY registers), computes the XOR checksum, writes <chksum>.state with offset=32\n. This is a utility, not test firmware; it can be deleted after the pad exists. Uses the same checksum algorithm as tools/make_test_pad.c so the pad is bit-compatible.
• B. Host generation via USB reader: if a USB SD reader is available, pop the card, run make_test_pad <mount>/pads 1048576 on the host, reinsert.

Default: A (no USB reader assumed). The generator is clearly marked as a setup utility and removed in Phase 0 of a future cleanup once the pad exists.

• Write firmware/teensy41/pad_gen/pad_gen.ino (1 MB pad, 32-byte header, checksum-named, offset=32\n state).
• Flash + run it; record the generated <chksum> for use in tests.

Phase 2 — Port libotppad to the firmware (otppad_embedded)

• Create otppad_embedded.h declaring the format-critical functions.
• Port otppad_xor, otppad_base64_encode/decode (reuse the existing b64_encode/b64_decode in dispatch.cpp if bit-identical, else port libotppad's tables).
• Port otppad_chunk_size, otppad_pad_apply, otppad_pad_remove (Padmé).
• Port otppad_armor_parse / otppad_armor_generate (replace strtok with manual line splitting; replace snprintf with Arduino sprintf).
• Port otppad_checksum as a streaming function over an Arduino File* (read in 4 KB chunks, fold into 32 buckets, XOR with first 32 pad bytes).
• Port otppad_state_read / otppad_state_write over SD File (atomic write: write <chksum>.state.tmp, SD.rename over <chksum>.state).
• Add a host-buildable unit test firmware/teensy41/signer/tests/host_test_otppad_embedded.c that links otppad_embedded.c compiled with HOST_TEST against a real FILE* backend, and verifies round-trip + padding + armor + checksum against libotppad outputs (bit-compatibility check).

Phase 3 — otp_pad_sd.cpp (bind + seek/read + encrypt/decrypt)

• Create otp_pad_sd.h with the bind/encrypt/decrypt API above.
• Implement otp_pad_sd_mount() — SD.begin(BUILTIN_SDCARD), report failure over Serial.
• Implement otp_pad_sd_bind(chksum) — open <chksum>.pad read-only, verify checksum via otppad_checksum, read offset from .state (default to 32 if missing), store pad size + chksum in globals.
• Implement otp_pad_sd_bind_first() — scan root for *.pad, bind the first one (debug auto-bind path).
• Implement otp_pad_sd_encrypt — Padmé-pad, seek+read pad slice, XOR, encode (ascii/binary), advance offset atomically, zeroize scratch.
• Implement otp_pad_sd_decrypt — parse armor/binary header, seek+read pad slice, XOR, strip Padmé, zeroize scratch. Does not advance offset (decrypt is non-consuming, matching host).
• Implement otp_pad_sd_unbind — close File, zeroize state.
• All scratch buffers in DMAMEM; cap chunk at 64 KB.

Phase 4 — Wire into dispatch + boot flow (debug auto-bind)

• In signer.ino apply_mnemonic(): after seed derivation, remove the otp_pad_init(g_seed, ...) HKDF call. Replace with: call otp_pad_sd_mount(); if DEBUG_AUTO_GENERATE=1, call otp_pad_sd_bind_first() and log the bound chksum over Serial. On failure, log but continue (encrypt/decrypt verbs will return otp pad not bound).
• In dispatch.cpp encrypt/decrypt verbs: replace the in-RAM otp_pad_apply/seek path with calls to otp_pad_sd_encrypt/otp_pad_sd_decrypt. Parse encoding option ("ascii" default, "binary"). Build the result JSON to match the host wire format (ciphertext, pad_chksum, pad_offset_before, pad_offset_after for encrypt; plaintext for decrypt). Keep algorithm: "otp" optional for backward compat.
• Remove the old otp_pad.cpp / otp_pad.h HKDF implementation (superseded by otp_pad_sd.*).

Phase 5 — Test harness + hardware round-trips

• Write firmware/teensy41/test_otp_sd.py — over USB CDC, framed JSON-RPC:
  • get_info (sanity).
  • encrypt ascii → parse armor, Pad-ChkSum matches bound chksum, Pad-Offset = 32 (first call).
  • decrypt ascii → recovered plaintext matches.
  • encrypt binary → blob starts with OTP\0, header chksum matches, pad_offset = 32 + first chunk.
  • decrypt binary → recovered plaintext matches.
  • Second encrypt ascii → Pad-Offset advanced by first chunk (proves offset persistence in .state).
  • Reboot the Teensy (power cycle), encrypt again → Pad-Offset continues from where it left off (proves .state survives power cycle).
  • Large plaintext (e.g. 10 KB) → Padmé bucket doubles to 16 KB, round-trip OK.
  • Tamper test: flip one byte in the armor base64 → decrypt returns error (padding removal fails) or wrong plaintext (detected).
• Run it against the flashed firmware; iterate on failures.

Phase 6 — Interactive pad selection UI (LAST)

• Add ui_pick_pad(pads_list, count) -> selected_chksum to ui.h / ui.cpp: an LVGL list screen showing each pad's chksum prefix + size + used%, with a "Use this pad" / "Skip OTP" choice. Blocks (pumps LVGL) until the user picks.
• In signer.ino boot flow, when DEBUG_AUTO_GENERATE=0: after apply_mnemonic(), scan the SD root for *.pad, build the list, call ui_pick_pad(). If the user picks one, otp_pad_sd_bind(chksum). If "Skip OTP" or no pads found, continue without a bound pad.
• Keep DEBUG_AUTO_GENERATE=1 → otp_pad_sd_bind_first() as the test path so Phase 5 tests still run headless.

Test commands (Phase 5)

# Build + flash the real firmware
bash firmware/teensy41/build_signer.sh --flash

# OTP SD round-trip suite
python3 firmware/teensy41/test_otp_sd.py --port /dev/ttyACM0

# Host-side bit-compatibility check for otppad_embedded
cc -O2 -Wall -Wextra -D HOST_TEST -o host_test_otppad_embedded \
    firmware/teensy41/signer/src/otppad_embedded.c \
    firmware/teensy41/signer/tests/host_test_otppad_embedded.c -lm
./host_test_otppad_embedded

Status (2026-07-28)

Done

• Phase 0: Throwaway sketches deleted.
• Phase 1: pad_gen.ino written, compiled, flashed, and run on the smaller card. Generated a 1 MB TRNG-sourced pad 4ec4e221...b0ca78.pad + .state (offset=32). The Teensy 4.1's hardware TRNG (TRNG_ENT0..15 registers) works — two runs produced different pads.
• Phase 2: otppad_embedded.{h,c} ported from libotppad. Host bit-compat test host_test_otppad_embedded.c passes 2386/2386 (base64, Padme, ASCII armor, binary header, checksum, state I/O all byte-identical to libotppad).
• Phase 3: otp_pad_sd.{h,cpp} implemented (mount, bind, bind_first, encrypt/decrypt with ascii+binary encodings, atomic offset advance, malloc scratch buffers). Code is complete.
• Phase 4: signer.ino and dispatch.cpp wired to the new otp_pad_sd API with host-compatible wire format. Old otp_pad.{cpp,h} deleted.

BLOCKER: <SD.h> crashes the signer firmware

Including <SD.h> in the signer firmware causes an immediate hard fault before setup() runs — no USB CDC enumeration, no serial output, no LED. This happens with both the custom linker script (imxrt1062_t41_flashmem.ld) and the default Teensy linker script. The crash occurs even when every SD function is stubbed out (only the #include <SD.h> is present).

The same <SD.h> works fine in standalone sketches:

• pad_gen.ino — mounts SD, writes a 1 MB pad, reads it back, verifies checksum. Runs perfectly.
• The deleted sd_test.ino / otp_card_probe.ino — mounted the 1 TB card, listed files, wrote+read a 64-byte test file. All worked.

Root cause (likely): The Teensy 4.1's flexRAM is dynamically partitioned between ITCM (code) and DTCM (data) in 32 KB blocks. The signer firmware already uses ~377 KB of ITCM (12 blocks → 384 KB ITCM, 128 KB DTCM). The SD/SdFat library adds ~13 KB of ITCM code, which — depending on the linker script — either pushes ITCM to 13 blocks (reducing DTCM to 96 KB, overflowing the 130 KB .data section) or doesn't change the block count but the additional .data/BSS overflows DTCM. The linker does not catch this because the flexRAM partitioning is computed at runtime by the Teensy boot ROM, not by the linker script.

Current workaround: otp_pad_sd.cpp has OTP_SD_ENABLED 0 — all SD functions are stubbed, <SD.h> is not included, and the firmware boots and works normally for all non-OTP verbs. The encrypt/decrypt verbs return otp pad not bound (no SD pad).

Path forward (to unblock)

1. Route SdFat code to FLASH via .flashmem: The custom linker script routes functions marked __attribute__((section(".flashmem"))) to FLASH instead of ITCM. The SD/SdFat library functions are not marked .flashmem, so they land in ITCM. Options:
   • Wrap the SD includes with #pragma GCC push_options + -ffunction-sections
     • a custom section attribute via a wrapper .cpp that re-exports the SD calls from a .flashmem-marked translation unit.
   • Fork/patch SdFat to add .flashmem attributes (heavy).
2. Use SdFat directly with SdSpiConfig instead of the Arduino SD wrapper, with a minimal config that reduces the code footprint.
3. Reduce the signer's own ITCM usage to make room for the SD library's ~13 KB (e.g., move more crypto code to FLASH).
4. Use the external RAM (ERAM, 32 MB at 0x70000000) for the SD library's BSS/buffers by placing them in .bss.extram — the linker script already defines this section but it's currently empty.

Risks / open questions

• SD library flexRAM crash (see BLOCKER above) — the main blocker.
• exFAT rename atomicity: SD.rename on SdFat exFAT should be atomic at the directory-entry level; verify once the crash is resolved.
• Chunk cap: set to 16 KB (OTP_SD_MAX_CHUNK) to fit RAM2; scratch buffers use malloc (heap) not static DMAMEM to avoid RAM2 BSS overflow.
• Checksum over a 1 MB pad on-device: streaming 4 KB reads, fast enough. On a future 900 GB pad, checksum-on-bind is impractical — add a skip-verify flag and only verify the first/last 4 KB for large pads.
• Pad generation entropy: the Teensy 4.1's hardware TRNG (TRNG_ENT0..15 registers) is used directly in pad_gen.ino and works.