n_os_tr/plans/binary_handling_strategy.md

# Binary Handling Strategy for n-OS-tr

> **Status:** Design. Defines how compiled binaries (`nsigner`, `nostr-id-tui`, `c-relay`, `ginxsom`, `fips`, `nak`, `nostr_core_lib`) get from upstream source into the n-OS-tr ISO. Establishes a single canonical pattern across all components, replacing the current ad-hoc mix of committed binaries, copy-pasted source trees, and an undocumented `/live-artifacts/` staging directory.
>
> Companion to [`plans/iso_architecture.md`](iso_architecture.md) (high-level ISO architecture) and [`plans/n_signer_integration.md`](n_signer_integration.md) (the integration this strategy unblocks).

## 1. Why this matters

n-OS-tr is a **security distro for cryptographic identity**. The trust story has to be: "the binary on this ISO was built deterministically from auditable source code pinned to a specific git commit." Anything weaker than that defeats the threat model in [`plans/threat_model.md`](threat_model.md).

The current state of [`includes/`](../includes/) and the build hook [`iso/config/hooks/live/0010-fetch-artifacts.hook.chroot`](../iso/config/hooks/live/0010-fetch-artifacts.hook.chroot) is a mix of three patterns with no documented rule:

| Component | Today's pattern | Trust story |
|---|---|---|
| [`includes/c_relay_static_x86_64`](../includes/c_relay_static_x86_64) | Pre-built binary committed to git | "Trust whoever built it." Weak. |
| [`includes/nak-v0.19.7-linux-amd64`](../includes/nak-v0.19.7-linux-amd64) | Pre-built binary committed to git | Same. |
| [`iso/config/includes.chroot/usr/local/bin/ginxsom/ginxsom-fcgi`](../iso/config/includes.chroot/usr/local/bin/ginxsom/ginxsom-fcgi) | Pre-built binary committed to git | Same. |
| [`includes/n_signer/`](../includes/n_signer/) | Full source tree copied in (no submodule) | Reproducible, but updates are manual file copies. |
| [`includes/c-relay/`](../includes/c-relay/) | Full source tree copied in | Same. |
| [`includes/ginxsom/`](../includes/ginxsom/) | Full source tree copied in | Same. |
| [`includes/fips/`](../includes/fips/) | Partial source (just `packaging/`) | Incomplete. |
| [`includes/nostr_core_lib/`](../includes/nostr_core_lib/) | Full source tree copied in | Reproducible, but tracked manually. |
| `nostr-id-tui` | Full source in [`stack/nostr-id-tui/`](../stack/nostr-id-tui/) | Reproducible. |
| `nostr_terminal` (`nt`) | **Not yet in repo** | Needs adding |
| `otp` | **Not yet in repo** | Needs adding |
| `didactyl` | **Not yet in repo** | Needs adding |

We also have a [`/live-artifacts/` staging convention](../iso/config/hooks/live/0010-fetch-artifacts.hook.chroot:5) where the build hook reads pre-built binaries from a chroot-internal directory populated by `iso/config/includes.chroot_before_packages/live-artifacts/` (gitignored, see [`.gitignore:26`](../.gitignore:26)). That convention is undocumented and the only way to get artifacts in there today is to drop them in manually before running `lb build`.

This is a mess. We standardize.

## 2. Decisions

### 2.1 Tier rules (in priority order)

Every binary that ends up in the ISO must follow one of these three tiers. **No exceptions for "we'll fix it later."**

#### Tier 1: Custom components — git submodule + build from source

For every component we author or substantially customize ([`n_signer`](../includes/n_signer/), [`c-relay`](../includes/c-relay/), [`ginxsom`](../includes/ginxsom/), [`nostr_core_lib`](../includes/nostr_core_lib/), [`nostr-id-tui`](../stack/nostr-id-tui/), [`fips`](../includes/fips/)):

- Vendored as a **git submodule** under [`includes/`](../includes/) pinned to a specific commit hash.
- Built from source during `lb build` via the component's own `build_static.sh` (Alpine + musl, Docker-driven).
- Output binary is staged into the chroot at a predictable path.
- **No pre-built binaries committed to this repo**. Ever.

This is the dominant tier — it covers everything we control.

#### Tier 2: Standard Debian software — apt packages

For software that's already in Debian's main repos (`nginx`, `socat`, `openssl`, `bash`, `xxd`, `coreutils`, etc.):

- Listed in [`iso/config/package-lists/*.list.chroot`](../iso/config/package-lists/).
- live-build installs them via `apt-get install` from Debian's signed mirrors.
- We never repackage Debian software; we consume it.

No change from today. Documented here for completeness.

#### Tier 3: Closed-source or unbuildable third-party — committed binary with SHA256 + provenance README

For things we genuinely cannot build from source (closed-source firmware blobs; third-party tools where building is impractical):

- The binary is committed to [`includes/third-party/<name>/`](../includes/) under Git LFS if larger than 1 MB.
- A `PROVENANCE.md` next to the binary states: upstream URL, version, SHA256, when fetched, by whom, what the build process upstream is.
- The SHA256 is also recorded in [`iso/SHA256SUMS`](../iso/SHA256SUMS) and verified by the build hook before staging.

Tier 3 should be **rare**. As of today, only [`nak`](../includes/nak-v0.19.7-linux-amd64) is a candidate, and even that we should audit (see §6).

### 2.2 The single artifact-staging convention

All binaries — Tier 1 build outputs, Tier 3 verified blobs — flow through one staging directory inside the chroot during `lb build`:

```
chroot:/live-artifacts/
├── nsigner                       # built from includes/n_signer        → /usr/local/sbin/nsigner
├── nostr-id-tui                  # built from includes/nostr-id-tui     → /usr/local/sbin/nostr-id-tui
├── c-relay                       # built from includes/c-relay          → /opt/c-relay/c-relay
├── ginxsom-fcgi                  # built from includes/ginxsom          → /usr/local/bin/ginxsom/ginxsom-fcgi
├── fips                          # built from includes/fips             → /usr/local/bin/fips (or .deb via dpkg)
├── didactyl                      # built from includes/didactyl         → /usr/local/bin/didactyl
├── nt                            # built from includes/nostr_terminal   → /usr/local/bin/nt
├── otp                           # built from includes/otp              → /usr/local/bin/otp
├── nak                           # tier-3 verified (or future Tier 1)   → /usr/local/bin/nak
└── SHA256SUMS                    # generated at build time, all built+staged binaries hashed
```

Daemons (nsigner, c-relay, ginxsom, fips, didactyl) → `/usr/local/sbin/` or `/opt/` per service convention. User-facing CLIs (nostr-id-tui, nt, otp, nak) → `/usr/local/bin/` so they're on `PATH`.

The current [`0010-fetch-artifacts.hook.chroot`](../iso/config/hooks/live/0010-fetch-artifacts.hook.chroot) already reads from `/live-artifacts/` — we keep that consumer-side convention. What changes is **how `/live-artifacts/` gets populated**: from a `lb` build hook that runs *before* the chroot runs, instead of the user dropping files there manually.

### 2.3 Pre-chroot build hook

Before `lb build` enters the chroot, a host-side script runs each component's `build_static.sh`, collects the output binaries, and stages them into `iso/config/includes.chroot_before_packages/live-artifacts/`. Live-build then copies that into `/live-artifacts/` inside the chroot, and the existing chroot-side `0010-fetch-artifacts.hook.chroot` does its job unchanged.

The pre-chroot script lives at [`scripts/build-binaries.sh`](../scripts/build-binaries.sh) (new). Outline:

```bash
#!/bin/bash
# Build all Tier-1 binaries from source, stage them for live-build.
set -euo pipefail

REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
STAGE="$REPO_ROOT/iso/config/includes.chroot_before_packages/live-artifacts"

mkdir -p "$STAGE"

# nostr_core_lib (C library; built once and consumed by downstream C builds at link time)
( cd "$REPO_ROOT/includes/nostr_core_lib" && ./build.sh x64 )
# (no binary to stage; consumed at link time by other builds)

# n_signer
( cd "$REPO_ROOT/includes/n_signer" && ./build_static.sh )
install -m 0755 "$REPO_ROOT/includes/n_signer/build/nsigner_static_x86_64" "$STAGE/nsigner"

# nostr-id-tui (n-OS-tr-native; lives at includes/nostr-id-tui/ after migration)
( cd "$REPO_ROOT/includes/nostr-id-tui" && ./build_static.sh )
install -m 0755 "$REPO_ROOT/includes/nostr-id-tui/build/nostr-id-tui_static_x86_64" "$STAGE/nostr-id-tui"

# c-relay
( cd "$REPO_ROOT/includes/c-relay" && ./build_static.sh )
install -m 0755 "$REPO_ROOT/includes/c-relay/build/c_relay_static_x86_64" "$STAGE/c-relay"

# ginxsom
( cd "$REPO_ROOT/includes/ginxsom" && ./build_static.sh )
install -m 0755 "$REPO_ROOT/includes/ginxsom/build/ginxsom-fcgi_static_x86_64" "$STAGE/ginxsom-fcgi"

# fips (Rust + cargo-deb produces a .deb)
( cd "$REPO_ROOT/includes/fips/packaging/debian" && ./build-deb.sh )
install -m 0644 "$REPO_ROOT/includes/fips/packaging/debian"/fips_*_amd64.deb "$STAGE/"

# didactyl (LLM-based agent daemon)
( cd "$REPO_ROOT/includes/didactyl" && ./build_static.sh )
install -m 0755 "$REPO_ROOT/includes/didactyl/build/didactyl_static_x86_64" "$STAGE/didactyl"

# nt (nostr_terminal — user CLI)
( cd "$REPO_ROOT/includes/nostr_terminal" && ./build_static.sh )
install -m 0755 "$REPO_ROOT/includes/nostr_terminal/build/nt_static_x86_64" "$STAGE/nt"

# otp (one-time pad CLI)
( cd "$REPO_ROOT/includes/otp" && make static )
install -m 0755 "$REPO_ROOT/includes/otp/build/otp-x86_64" "$STAGE/otp"

# Tier-3: verify + stage
sha256sum -c "$REPO_ROOT/iso/SHA256SUMS.tier3"   # fails if any committed blob has wrong hash
install -m 0755 "$REPO_ROOT/includes/third-party/nak/nak" "$STAGE/nak"

# Generate manifest
( cd "$STAGE" && sha256sum * > SHA256SUMS )
echo "Staged binaries:"
ls -la "$STAGE"
```

The `lb-wrapper.sh` (or `build.sh`) at the repo root invokes `scripts/build-binaries.sh` before calling `lb build`.

## 3. Migration plan: turning includes/ into submodules

### 3.1 Per-component conversion

For each component currently checked in as a plain directory under [`includes/`](../includes/):

1. Confirm there's an upstream git URL for the component.
2. `git rm -r includes/<name>` (this removes the files from this repo's tracking).
3. `git submodule add <upstream-url> includes/<name>`
4. `cd includes/<name> && git checkout <pinned-tag-or-commit>`
5. `cd ../.. && git commit -m "vendor <name> as submodule at <ref>"`

Result: this repo records *which commit* of the upstream we ship, but not the contents. `git clone --recurse-submodules` produces an identical working tree to today.

### 3.2 Component-by-component decisions

All upstream URLs are now confirmed (per user, 2026-05-04). They all live under `https://git.laantungir.net/laantungir/`:

| Component | Upstream URL | Action |
|---|---|---|
| `n_signer` | `https://git.laantungir.net/laantungir/n_signer` | Convert to submodule. Source already at [`includes/n_signer/`](../includes/n_signer/) but is **stale** (pre-v0.0.16, no `--preapprove`). Refresh from upstream during conversion. |
| `c-relay` | `https://git.laantungir.net/laantungir/c-relay` | Convert to submodule. |
| `ginxsom` | `https://git.laantungir.net/laantungir/ginxsom` | Convert to submodule. |
| `fips` | `https://git.laantungir.net/laantungir/fips` | Convert to submodule. **Note:** user does not control this repo (vendored from elsewhere); pin a known-good ref. |
| `nostr_core_lib` | `https://git.laantungir.net/laantungir/nostr_core_lib` | Convert to submodule. |
| `nostr_terminal` (`nt`) | `https://git.laantungir.net/laantungir/nostr_terminal` | **Add new submodule** at `includes/nostr_terminal/`. |
| `otp` | `https://git.laantungir.net/laantungir/otp` | **Add new submodule** at `includes/otp/`. |
| `didactyl` | `https://git.laantungir.net/laantungir/didactyl` | **Add new submodule** at `includes/didactyl/`. |
| `nostr-id-tui` | n-OS-tr-native (no upstream) | **Move** from [`stack/nostr-id-tui/`](../stack/nostr-id-tui/) to [`includes/nostr-id-tui/`](../includes/) for consistency. Stays in this repo as plain source — no submodule. |

**B1 closed.**

### 3.3 Things to delete after migration

Once submodules are in place and `scripts/build-binaries.sh` works:

- [`includes/c_relay_static_x86_64`](../includes/c_relay_static_x86_64) — pre-built binary, replaced by source-built output
- [`iso/config/includes.chroot/usr/local/bin/ginxsom/ginxsom-fcgi`](../iso/config/includes.chroot/usr/local/bin/ginxsom/ginxsom-fcgi) — same
- [`iso/config/includes.chroot/usr/local/bin/nak`](../iso/config/includes.chroot/usr/local/bin/nak) — moved to Tier 3 location with PROVENANCE.md (or built from Go source, see §6)
- [`includes/nak-v0.19.7-linux-amd64`](../includes/nak-v0.19.7-linux-amd64) — same
- [`includes/ginxsom/build/`](../includes/ginxsom/build/) — checked-in build output, regenerated from source
- [`stack/nostr-id-tui/build-host/`](../stack/nostr-id-tui/build-host/) — checked-in build output

### 3.4 Build host requirements (consequence of Tier 1)

Anyone building n-OS-tr from scratch needs:

- Linux host with Docker (for the Alpine+musl static builds)
- ~10 GB free disk for Docker layer cache
- Network access *the first time* (to pull `alpine:3.19`, plus apk packages during `docker build`); offline thereafter
- `live-build` (the Debian package) on the host
- `git` with submodule support (`git clone --recurse-submodules`)
- For Rust components like fips: the `rust` and `cargo-deb` packages, OR a Rust-capable Docker recipe (preferred, for consistency with the Tier 1 pattern)

This is documented in [`docs/RUNNING.md`](../docs/RUNNING.md) and [`docs/debian_live_notes.md`](../docs/debian_live_notes.md).

## 4. Reproducibility ladder

We aim for this property: **given a commit hash of n-OS-tr, two different people on two different days produce byte-identical ISOs**. Achieving the full level requires more than just "build from source"; it requires deterministic builds. The ladder:

| Level | Guarantee | Cost |
|---|---|---|
| **L0: source-pinned** | Same source produces same binary on the same machine | Free if you follow Tier 1 |
| **L1: same-platform reproducible** | Same source produces same binary on any Alpine 3.19 amd64 host | Requires `SOURCE_DATE_EPOCH`, no `__DATE__`/`__TIME__` macros |
| **L2: cross-platform reproducible** | Same source produces same binary on amd64 *and* arm64 hosts | Hard. Compiler must produce same output regardless of build host. |
| **L3: fully reproducible ISO** | Same source produces byte-identical ISO | Requires L2 + reproducible squashfs + reproducible bootloader timestamps |

Today we don't claim any level explicitly. **Target for v1: L1.** That's what most security distros (Tails, Qubes) achieve. L2 and L3 are nice-to-have and can come later.

The Tier 1 + `build_static.sh` foundation makes L1 the path of least resistance — `SOURCE_DATE_EPOCH` is set to the git commit timestamp at build time, and the Alpine/musl recipe doesn't pull in any non-deterministic dependencies.

## 5. Update workflow

### 5.1 To bump n_signer to a new version

```bash
cd includes/n_signer
git fetch
git checkout v0.0.11        # or whatever tag
cd ../..
./scripts/build-binaries.sh # verify it still builds
git add includes/n_signer
git commit -m "bump n_signer to v0.0.11"
```

The diff in this repo's git log shows exactly: "submodule pointer changed from <old-sha> to <new-sha>". Anyone reviewing the commit can `cd includes/n_signer && git log <old-sha>..<new-sha>` to see what actually changed.

### 5.2 To bump a Tier-3 binary (e.g., nak)

```bash
# Download new version, verify upstream signature/checksum
curl -O https://github.com/fiatjaf/nak/releases/download/v0.20.0/nak-v0.20.0-linux-amd64
sha256sum nak-v0.20.0-linux-amd64
# Compare against upstream-published checksum

mv nak-v0.20.0-linux-amd64 includes/third-party/nak/nak
sha256sum includes/third-party/nak/nak >> iso/SHA256SUMS.tier3  # remove old line
# Update PROVENANCE.md with new version, URL, hash, date
git add includes/third-party/nak iso/SHA256SUMS.tier3
git commit -m "bump nak to v0.20.0"
```

### 5.3 To add a new component

1. Decide tier (1, 2, or 3).
2. **Tier 1:** add submodule, write/verify `build_static.sh`, add stage step to `scripts/build-binaries.sh`, add chroot install step to `0010-fetch-artifacts.hook.chroot`.
3. **Tier 2:** add to a `*.list.chroot`.
4. **Tier 3:** put binary at `includes/third-party/<name>/`, write `PROVENANCE.md`, add SHA256 to `iso/SHA256SUMS.tier3`, add to `scripts/build-binaries.sh`.

## 6. The `nak` problem

[`includes/nak-v0.19.7-linux-amd64`](../includes/nak-v0.19.7-linux-amd64) is a Go binary built upstream by [fiatjaf/nak](https://github.com/fiatjaf/nak). Two paths:

**Path A: keep as Tier 3.** `nak` is a small CLI, fiatjaf is a known upstream, the binary is statically linked Go. Document with PROVENANCE.md, pin SHA256, done. Acceptable if we want to ship soon.

**Path B: move to Tier 1 (Go build).** Add a Go-capable Docker builder under [`stack/build-common/`](../stack/build-common/) (or a `Dockerfile.alpine-go` next to the existing musl one), make `nak` a submodule, build from source. Better for the trust story; more work.

**Recommendation: Path A for v1, Path B as a follow-up before Phase 2 launches.** The reasoning is that `nak` is currently used in [`n-os-tr-firstboot`](../iso/config/includes.chroot/usr/local/sbin/n-os-tr-firstboot) for one-shot operations and has been removed from the slimmed firstboot script (see [`plans/n_signer_integration.md`](n_signer_integration.md)). Whether it's even still needed should be checked — if not, we can just remove it.

> **Open question (B2):** Is `nak` actually used anywhere after the `n_signer` integration removes it from `n-os-tr-firstboot`? If no, delete it entirely.

## 7. Threat-model implications

This strategy strengthens [`plans/threat_model.md`](threat_model.md):

- **F26 (deterministic build).** Tier 1 + L1 reproducibility means anyone can re-build the ISO from a git checkout and get the same binaries. Pre-built binaries can never satisfy this; they're a permanent "trust me" hole.
- **F27 (signed releases).** Once we have Tier 1 across the board, we can sign the ISO with a key that vouches for the entire source tree. Without source pinning, a release signature only vouches for the publisher, not the contents.
- **Supply-chain attacks.** A compromised build host that produces malicious binaries committed to git is undetectable. A compromised build host that produces malicious binaries from source is detectable: rebuild on a clean host, compare. L1 reproducibility is the foundation.

## 8. Open questions

- **B1.** ~~Confirm canonical upstream URLs.~~ **Closed (2026-05-04):** all components live under `https://git.laantungir.net/laantungir/` with names matching the directory names. `nostr-id-tui` is n-OS-tr-native (no upstream). `nak` upstream is at `https://github.com/nostrapps/nostr-army-knife`.
- **B2.** Is `nak` still needed after the `n_signer` integration? **Tentative: yes** — useful CLI for users to inspect events from the shell on tty3+. Keep as Tier 3 with PROVENANCE.md until we know otherwise.
- **B3.** Should `nostr-id-tui` move from [`stack/`](../stack/) to [`includes/`](../includes/) for consistency? **Yes** — confirmed. Keep `stack/build-common/` for shared build infra; everything else under `includes/`.
- **B4.** Do we want `iso/SHA256SUMS` to track Tier-1 build outputs as well, for a single manifest? Trade-off: convenient end-state, but the file would change every commit since binaries are non-deterministic until we hit L1. **Defer until L1 is in place.**
- **B5.** Should `scripts/build-binaries.sh` be parallelized? Currently sequential. With nine components, sequential builds take ~10–20 minutes. Parallelizing across components could halve this. Optional optimization.
- **B6.** Is `cargo-deb` (used for fips) compatible with Alpine/musl? Or do we need a separate Debian-based builder for fips's `.deb`? **Likely: separate Debian builder.** Rust+musl works fine, but `cargo-deb` expects Debian packaging tooling (dpkg-dev, debhelper) which isn't trivially in Alpine. Cleanest answer: a `Dockerfile.debian-rust` next to the existing musl Dockerfiles, used only by fips. Investigate when we get to fips conversion.
- **B7.** What index should didactyl get in the role schedule? **Closed: index 5** ([`plans/n_signer_integration.md` §2](n_signer_integration.md#2-the-components-in-scope)).
- **B8.** What should be the user CLI signing UX (always-prompt vs. preapproved for the user's uid)? **Closed: preapproved** for `caller=uid:1000,nostr_index=0` ([`plans/n_signer_integration.md` §7.2](n_signer_integration.md#72-preapprove-for-the-users-uid)).
- **B9.** `nt` and `otp` need code changes upstream to use `n_signer` instead of their own mnemonic flow. Tracked as T11 in [`plans/upstream_nsigner_tickets.md`](upstream_nsigner_tickets.md).

## 9. Mermaid: artifact flow

```mermaid
flowchart TB
    subgraph repo[n_os_tr repo]
        SUB[includes/* git submodules<br/>pinned commits]
        STA[stack/build-common/<br/>Dockerfile.alpine-musl + build_static.sh]
        T3[includes/third-party/nak<br/>+ PROVENANCE.md + SHA256]
    end

    subgraph host[Build host]
        BUILD[scripts/build-binaries.sh]
        DCK[Docker daemon<br/>Alpine 3.19 musl]
        STAGE[iso/config/includes.chroot_before_packages/<br/>live-artifacts/]
    end

    subgraph lb[live-build]
        LB[lb build]
        CHROOT[chroot /live-artifacts/]
        HOOK[0010-fetch-artifacts.hook.chroot]
        ISO[n-os-tr.iso]
    end

    SUB --> BUILD
    STA --> BUILD
    T3 --> BUILD
    BUILD --> DCK
    DCK --> STAGE
    STAGE --> LB
    LB --> CHROOT
    CHROOT --> HOOK
    HOOK --> ISO
```

## 10. Summary

| Question | Answer |
|---|---|
| How do binaries get into the ISO? | Built from source at ISO build time via Docker+Alpine+musl |
| Where is source kept? | `includes/<name>/` as git submodules, pinned to commits |
| What about pre-built binaries in git? | **Forbidden** for Tier 1. Allowed only for Tier 3 (closed-source/unbuildable third-party) with PROVENANCE.md and SHA256. |
| What about apt packages? | Tier 2: standard Debian packages via `*.list.chroot`. No change. |
| Can the build run offline? | Once you've cloned `--recurse-submodules` and pulled `alpine:3.19` once, yes. |
| Does the ISO download anything at runtime? | No. Never. The whole point of an ISO is offline-bootable. |
| What's the trust story? | Git commit hash → submodule commit hashes → identical source → identical binaries (L1 reproducibility) → identical ISO. |