n_signer/plans/cyd_algorithm_api_upgrade.md

Plan: Bring firmware/cyd_esp32_2432s028 Up to Date with the Algorithm-Based API

Context

The main project (src/dispatcher.c) has fully migrated to the algorithm-based API documented in README.md §4. The migration is marked COMPLETED in plans/legacy_verb_aliases.md: legacy verb names are gone from the wire protocol, implementation, tests, clients, and docs.

The CYD firmware at firmware/cyd_esp32_2432s028/main/main.c was not updated and still speaks the legacy verb API:

Firmware verb (current)	Main-project verb (target)
get_public_key (+ nostr_index)	split → get_public_key (+ algorithm) and nostr_get_public_key (+ nostr_index)
sign_event	nostr_sign_event
nip04_encrypt / nip04_decrypt	nostr_nip04_encrypt / nostr_nip04_decrypt
nip44_encrypt / nip44_decrypt	nostr_nip44_encrypt / nostr_nip44_decrypt
(missing)	sign, verify, encapsulate, decapsulate, derive_shared_secret, derive, encrypt/decrypt (otp), nostr_mine_event

Good news: the crypto primitives already exist

The firmware already has every underlying primitive needed — only the dispatch layer in main.c is stale:

• key_derivation.h: derive_nostr_key_index, derive_ed25519_key, derive_x25519_key, derive_ml_dsa_65_key, derive_slh_dsa_128s_key, derive_ml_kem_768_key, schnorr_sign32, ed25519_sign32
• pq_crypto_firmware.h: fw_pq_ml_dsa_65_sign/verify, fw_pq_slh_dsa_128s_sign/verify, fw_pq_ml_kem_768_encaps/decaps
• nostr_core_lib nip004/nip044 already linked for the Nostr verbs

key_id convention

The main project defines key_id as the first 16 hex characters of the public key (see src/dispatcher.c ~line 1788). The firmware must match this so clients can correlate keys across targets.

Scope

Target: firmware/cyd_esp32_2432s028 only. The feather_s3_tft firmware is explicitly out of scope for this pass (it has the same gap but will be handled separately).

Architecture: dispatch flow after upgrade

flowchart TD
    A[recv frame] --> B[parse JSON-RPC]
    B --> C{auth envelope}
    C -->|fail| Z[auth error]
    C -->|ok| D{method}
    D -->|nostr_*| E[Nostr verb branch<br/>secp256k1 NIP-06<br/>nostr_index selector]
    D -->|alg verb| F[Algorithm verb branch<br/>algorithm + index selector]
    E --> G[derive_request_key<br/>nostr_index]
    F --> H{algorithm}
    H -->|secp256k1| H1[derive_nostr_key_index]
    H -->|ed25519| H2[derive_ed25519_key]
    H -->|x25519| H3[derive_x25519_key]
    H -->|ml-dsa-65| H4[derive_ml_dsa_65_key]
    H -->|slh-dsa-128s| H5[derive_slh_dsa_128s_key]
    H -->|ml-kem-768| H6[derive_ml_kem_768_key]
    H -->|otp| H7[bound pad]
    G --> I[enforcement matrix check]
    H1 --> I
    H2 --> I
    H3 --> I
    H4 --> I
    H5 --> I
    H6 --> I
    H7 --> I
    I -->|reject 1010| Z
    I -->|ok| J[approval prompt]
    J --> K[execute verb]
    K --> L[structured result JSON<br/>algorithm + key_id + field]

Implementation Steps

1. Add algorithm-name parsing helpers (main.c)

Add a small enum + parser mirroring the main project's algorithm set:

typedef enum {
    FW_ALG_SECP256K1 = 0,
    FW_ALG_ED25519,
    FW_ALG_X25519,
    FW_ALG_ML_DSA_65,
    FW_ALG_SLH_DSA_128S,
    FW_ALG_ML_KEM_768,
    FW_ALG_OTP,
    FW_ALG_UNKNOWN
} fw_alg_t;

• parse_algorithm_from_options(cJSON *options, fw_alg_t *out_alg, uint32_t *out_index) — reads algorithm (string) and index (number, default 0) from the trailing options object in params.
• fw_alg_name(fw_alg_t) → canonical string ("secp256k1", "ed25519", …) for response JSON.
• Keep the existing parse_nostr_index_from_params() for the nostr_* verbs.

2. Add a unified algorithm-key derivation + key_id helper

Add derive_alg_key(fw_alg_t alg, uint32_t index, ...) that dispatches to the right derive_*_key function and produces:

• the raw private key bytes (when applicable),
• the public key hex,
• the key_id (first 16 hex chars of the public key).

PQ algorithms have large key buffers (ml-dsa-65 sk = 4032 B, ml-kem-768 sk = 2400 B). Allocate these as static buffers (not on the stack) and secure_memzero after use, matching the existing s_privkey/s_pubkey pattern. SLH-DSA-128s keygen is slow (5–30 s) — log a warning and show a "deriving key…" UI screen before calling it.

3. Add a structured-result builder

Add build_alg_result_json(const char *alg_name, const char *key_id_16hex, const char *field_name, const char *field_value) → returns a JSON string like {"algorithm":"ed25519","key_id":"<16hex>","public_key":"<hex>"}. This mirrors build_alg_result_json in the main dispatcher.

4. Add the enforcement matrix

Add enforce_alg_verb(fw_alg_t alg, const char *verb) returning 0 / 1010, matching README.md §4.3 enforcement matrix:

Verb	Valid algorithms
sign / verify	secp256k1, ed25519, ml-dsa-65, slh-dsa-128s
encapsulate / decapsulate	ml-kem-768
derive_shared_secret	x25519
derive	secp256k1
encrypt / decrypt	otp
get_public_key	all key-deriving algorithms

Reject any unlisted pair with {"error":{"code":1010,"message":"algorithm_not_supported_for_verb"}}.

5. Rename the Nostr verbs (in-place, no compat shim)

In the dispatch if/else if chain in main.c ~lines 836–1230:

• get_public_key (nostr_index branch) → nostr_get_public_key
• sign_event → nostr_sign_event
• nip04_encrypt → nostr_nip04_encrypt
• nip04_decrypt → nostr_nip04_decrypt
• nip44_encrypt → nostr_nip44_encrypt
• nip44_decrypt → nostr_nip44_decrypt

The nostr_get_public_key verb should also honor the format option ("structured" → {"algorithm":"secp256k1","public_key":"<hex>","key_id":"<16hex>"}, default → bare 64-hex pubkey string), matching the main project.

6. Add the algorithm-based get_public_key verb

get_public_key with algorithm + index → derive the key, return structured JSON: {"algorithm":"<alg>","public_key":"<hex>","key_id":"<16hex>"}. For PQ algorithms the public_key is the full PQClean pubkey hex (1952 B for ml-dsa-65, 1184 B for ml-kem-768, 32 B for slh-dsa-128s) — ensure s_response_buf (currently 2048 B) is large enough, or emit via cJSON_PrintUnformatted into a larger static buffer.

7. Add sign and verify (algorithm-based)

• sign: params [<message_hex>, {algorithm, index, scheme?}]. scheme is secp256k1-only ("schnorr" default / "ecdsa"). For ed25519 use ed25519_sign32 (note: ed25519 signs the raw 32-byte message, not a pre-hash — match main project behavior). For ml-dsa-65 / slh-dsa-128s use fw_pq_*_sign. Response: {"algorithm":"<alg>","key_id":"<16hex>","signature":"<hex>"}.
• verify: params [<message_hex>, <signature_hex>, {algorithm, index, scheme?}]. Derive the signer's own pubkey and verify against it. Response: {"valid":true,"algorithm":"<alg>"}.

8. Add encapsulate / decapsulate (ml-kem-768)

• encapsulate: params [<peer_pubkey_hex>, {algorithm:"ml-kem-768"}] → fw_pq_ml_kem_768_encaps → {"ciphertext":"<hex>","shared_secret":"<hex>","algorithm":"ml-kem-768"}.
• decapsulate: params [<ciphertext_hex>, {algorithm:"ml-kem-768", index}] → derive kem keypair, fw_pq_ml_kem_768_decaps → {"shared_secret":"<hex>","algorithm":"ml-kem-768"}.

9. Add derive_shared_secret (x25519)

params [<peer_pubkey_hex>, {algorithm:"x25519", index}] → derive x25519 keypair, compute X25519 ECDH via mbedtls → {"shared_secret":"<hex>","algorithm":"x25519"}.

10. Add derive (secp256k1 HMAC-SHA256)

params [<data>, {algorithm:"secp256k1", index}] (index required) → derive secp256k1 privkey, compute HMAC-SHA256(privkey, data) via mbedtls → {"algorithm":"secp256k1","key_id":"<16hex>","digest":"<64hex>"}. See plans/derive_hmac.md for the spec.

11. Add nostr_mine_event (PoW)

params [<event_json>, {nostr_index, difficulty, timeout_sec, threads?}]. Reuse build_signed_event_json but iterate nonce in the event's tags until the leading-zero bits of the event id meet difficulty. ESP32 is slow — cap threads at 1 and enforce a firm timeout_sec (default 30). Show a "mining…" UI screen. If timeout, return error 1008 mining_failed. This mirrors src/miner.c.

12. Add encrypt / decrypt (otp)

The main project binds a pad from --otp-pad-dir + --otp-pad (a USB file). The CYD has no filesystem pad source. Decision: derive the OTP pad from the mnemonic seed via a SHAKE-256 / HKDF expansion keyed on algorithm:"otp" so the pad is deterministic per mnemonic and advances monotonically across requests (offset stored in a static variable, reported in every response). This keeps the wire contract identical (encrypt/decrypt with algorithm:"otp", encoding:"ascii"|""binary"") while fitting the embedded constraint. Document this divergence in firmware/README.md.

13. Bump FIRMWARE_VERSION and update firmware/README.md

• FIRMWARE_VERSION "0.0.1" → "0.0.2" (algorithm-based API).
• Document the new verb table, the OTP pad-derivation divergence, and the SLH-DSA-128s latency warning.

14. Update CYD-targeting examples / clients

Audit and update any example or client that speaks to the CYD over UART/Web-Serial and uses legacy verb names:

• examples/feather_get_public_key.py and examples/feather_sign_event.py (feather-targeting but the wire protocol is shared — note in README they need nostr_ prefixes for CYD after this change; leave feather examples alone since feather is out of scope, but add a CYD-specific example pair if none exists).
• client/ demos already use the new verbs (per plans/legacy_verb_aliases.md) — verify no CYD-specific legacy calls remain.

15. Add a CYD Web Serial test page covering all algorithms

The existing examples/feather_webusb_demo.html is WebUSB-only (feather's native USB) and uses the legacy verbs. The CYD's CH340 bridge (1a86:7523) is not a WebUSB device — it exposes a serial port, so the browser transport is Web Serial (navigator.serial), Chromium-only.

Create examples/cyd_webserial_demo.html — a single-file, dependency-light test page that:

Transport:

• navigator.serial.requestPort() → port.open({ baudRate: 115200 }) (matches uart_transport.c UART_BAUD_RATE).
• Same 4-byte big-endian length-prefix frame format as the feather demo (be32(), read loop reassembling frames).
• Read via a ReadableStream reader + length-prefix reassembly (Web Serial is stream-based, not packet-based like WebUSB transferIn).
• Same auth-envelope construction (kind 27235, nsigner_method / nsigner_body_hash tags, schnorr sign with a demo caller key) — reuse the buildAuth() logic verbatim.

UI sections (one card per verb family, all algorithms):

1. Connect — Connect Web Serial button + status.
2. Get Public Key — algorithm dropdown (secp256k1, ed25519, x25519, ml-dsa-65, slh-dsa-128s, ml-kem-768) + index → get_public_key. Also a nostr_get_public_key card with nostr_index + format (bare / structured) toggle.
3. Sign / Verify — algorithm dropdown (sig algs only) + index + scheme (schnorr/ecdsa, secp256k1-only) + message hex → sign; then verify with the returned signature.
4. KEM (ml-kem-768) — encapsulate with a peer pubkey (or self-pubkey from get_public_key) → ciphertext + shared secret; decapsulate with that ciphertext → shared secret (confirm match).
5. X25519 — derive_shared_secret with peer pubkey.
6. Derive (HMAC) — derive with data string + index → 64-hex digest.
7. Nostr Sign Event — nostr_sign_event (kind 1) with nostr_index.
8. Nostr Mine Event — nostr_mine_event with difficulty + timeout (low default, e.g. difficulty 4) — warn it's slow on ESP32.
9. NIP-04 / NIP-44 — nostr_nip04_encrypt/decrypt, nostr_nip44_encrypt/decrypt with nostr_index.
10. OTP — encrypt / decrypt with algorithm:"otp", encoding toggle (ascii/binary), base64 plaintext.

Each card shows the raw JSON-RPC request and response in a <pre> so the wire format is visible. Reuse the feather demo's CSS (dark theme, cards, .mono log) for consistency.

Link it from firmware/README.md in the CYD section (the "Quick validation" / Web Serial path), since the current README only points at the feather WebUSB demo.

16. Verification

• idf.py build in firmware/cyd_esp32_2432s028 compiles clean.
• Flash to the connected CYD board (CH340 on /dev/ttyUSB0) and smoke-test each verb:
  • Primary path: open examples/cyd_webserial_demo.html in Chrome/Edge, connect via Web Serial, exercise every card.
  • Secondary path: a small Python script over /dev/ttyUSB0 for headless confirmation.
  • Cover: get_public_key (each algorithm), sign/verify (each sig alg), encapsulate/decapsulate, derive_shared_secret, derive, nostr_get_public_key, nostr_sign_event, nostr_nip04_encrypt/decrypt, nostr_nip44_encrypt/decrypt, nostr_mine_event (low difficulty), encrypt/decrypt (otp).
  • Confirm key_id matches the first 16 hex of the returned pubkey for every alg.
  • Confirm an invalid (verb, algorithm) pair returns code 1010.
• grep -rn "sign_event\|nip04_encrypt\|nip04_decrypt\|nip44_encrypt\|nip44_decrypt" firmware/cyd_esp32_2432s028/ returns no matches (legacy names gone).

Open questions / decisions baked in

• OTP pad source: derived from mnemonic (no USB pad on CYD). Documented divergence.
• nostr_mine_event: implemented, single-threaded, hard 30 s default timeout, with a "mining…" UI screen. Not stubbed — the main project has it and the user asked to bring the firmware up to date.
• No legacy-verb compat shim: matches the main project's policy (plans/legacy_verb_aliases.md — "No backward- compatibility shim is needed").
• feather_s3_tft not touched in this pass.