[1;33m[WARNING][0m No existing semantic tags found. Starting from v0.0.0

v0.0.1 - Reorganize ISO build tree, add docs/plans/scripts, and stage Phase 2 Slice B+C service integration artifacts
2026-04-29 13:09:28 -04:00
parent dc7af63e6b
commit 6e2164abe5
151 changed files with 8865 additions and 1222 deletions
--- a/plans/iso_architecture.md
+++ b/plans/iso_architecture.md
@@ -0,0 +1,382 @@
+# n-OS-tr ISO Architecture — Design Notes
+
+This document captures the key design decisions for the n-OS-tr Debian Live
+ISO build. It is the companion to the phased todo list and is intended to
+survive individual work sessions.
+
+## 1. Big-picture goal
+
+A single bootable/installable Debian Live ISO that, the moment it finishes
+booting, is running:
+
+- [`fips`](../includes/fips) — Rust mesh-routing daemon, npub-addressed IPv6 overlay
+- [`c-relay`](../includes/c-relay) — C Nostr relay with event-based config and embedded admin
+- [`ginxsom`](../includes/ginxsom) — C Blossom (blob) server as FastCGI behind nginx
+- `nginx` — TLS front door, static blob serving, reverse proxy
+
+No Docker. No `curl | bash` provisioning. All three apps run as native
+systemd services directly on the host.
+
+## 2. Repo layout (target)
+
+```
+/                              project root
+├── live-build/                upstream live-build vendored
+├── lb-wrapper.sh              thin wrapper for ./live-build/frontend/lb
+├── iso/                       the live-build config (promoted from tutorial2/)
+│   ├── auto/
+│   │   ├── config             lb config invocation with our flags
+│   │   ├── build              optional
+│   │   └── clean              optional
+│   ├── config/
+│   │   ├── package-lists/
+│   │   │   ├── base.list.chroot
+│   │   │   ├── fips-deps.list.chroot
+│   │   │   ├── tor.list.chroot            (optional)
+│   │   │   ├── gui.list.chroot            (optional)
+│   │   │   └── installer.list.chroot      (optional)
+│   │   ├── hooks/
+│   │   │   └── live/
+│   │   │       ├── 0010-fetch-artifacts.hook.chroot
+│   │   │       └── 0020-enable-services.hook.chroot
+│   │   ├── includes.chroot/
+│   │   │   ├── etc/systemd/system/*.service
+│   │   │   ├── etc/nginx/sites-available/n-os-tr.conf
+│   │   │   ├── etc/nginx/fastcgi_params
+│   │   │   ├── etc/nginx/mime.types
+│   │   │   ├── usr/local/sbin/n-os-tr-firstboot
+│   │   │   ├── var/www/html/index.html
+│   │   │   └── etc/issue
+│   │   └── SHA256SUMS                     pinned artifact digests
+│   └── local-artifacts/                   optional offline override (gitignored)
+├── includes/                  the three component projects
+│   ├── c-relay/
+│   ├── ginxsom/
+│   └── fips/
+├── plans/                     this doc and future design notes
+└── README.md
+```
+
+Legacy scripts ([`root.sh`](../root.sh), [`nginx.sh`](../nginx.sh),
+[`strfry.sh`](../strfry.sh), [`deploy.sh`](../deploy.sh),
+[`test.sh`](../test.sh), [`docker.sh`](../docker.sh),
+[`n-os-tr.sh`](../n-os-tr.sh)) are obsolete for the ISO build. They can
+move to `scratch/` for historical reference or be deleted.
+
+## 3. Runtime architecture on the booted ISO
+
+```mermaid
+flowchart TB
+    subgraph Ext[External]
+        U1([Internet peer])
+        U2([LAN host])
+        U3([Tor client])
+    end
+    subgraph Host[n-OS-tr booted ISO]
+        direction TB
+        Ng[nginx<br/>:80 :443]
+        C[c-relay<br/>127.0.0.1:8888]
+        G[ginxsom FastCGI<br/>/tmp/ginxsom-fcgi.sock]
+        F[fips daemon<br/>fips0 TUN]
+        Fd[fips-dns<br/>:5354]
+        Blobs[/var/www/blobs/]
+        RelayDb[/var/lib/c-relay/]
+        GinxDb[/var/lib/ginxsom/]
+        FbUnit[n-os-tr-firstboot<br/>oneshot]
+        FbUnit -.generates keys.-> KeyStore[/var/lib/n-os-tr/]
+        KeyStore -.EnvironmentFile.-> G
+        KeyStore -.identity.-> F
+        Ng -->|wss /relay| C
+        Ng -->|GET /sha256| Blobs
+        Ng -->|FastCGI PUT DELETE LIST HEAD| G
+        G --> Blobs
+        G --> GinxDb
+        C --> RelayDb
+    end
+    U1 --> Ng
+    U2 --> Ng
+    U3 -.onion.-> Ng
+    U1 -.npub IPv6.-> F
+```
+
+Notes:
+- `c-relay` generates its own admin keypair on first start and prints the
+  nsec to the journal once. This is c-relay's native behavior and we should
+  not override it.
+- `ginxsom` needs `--server-privkey` from **somewhere**. Today
+  [`ginxsom.service`](../includes/ginxsom/ginxsom.service:16) hardcodes one.
+  We replace that with an `EnvironmentFile` populated by the first-boot unit.
+- `fips` can run ephemeral by default; persistent mode drops a `fips.key`
+  next to [`fips.yaml`](../includes/fips/packaging/common/fips.yaml:1).
+
+## 4. Boot sequence
+
+```mermaid
+sequenceDiagram
+    participant LB as live-boot / initramfs
+    participant Sd as systemd PID 1
+    participant Fb as n-os-tr-firstboot.service
+    participant Fp as fips.service
+    participant Cr as c-relay.service
+    participant Gx as ginxsom.service
+    participant Ng as nginx.service
+
+    LB->>Sd: hand off
+    Sd->>Fb: start oneshot (Before= the app units)
+    alt First boot
+        Fb->>Fb: mkdir /var/lib/n-os-tr<br/>generate ginxsom key<br/>generate fips key<br/>write EnvironmentFile<br/>touch .provisioned
+    else Already provisioned
+        Fb-->>Sd: exit 0 (ConditionPathExists trip)
+    end
+    Fb-->>Sd: done
+    par
+        Sd->>Fp: start (reads /etc/fips/fips.yaml)
+        and
+        Sd->>Cr: start (self-provisions admin key to /var/lib/c-relay/)
+        and
+        Sd->>Gx: start (spawn-fcgi, reads env file for privkey)
+        and
+        Sd->>Ng: start (serves /var/www/blobs, proxies to c-relay and ginxsom)
+    end
+```
+
+## 5. Phase 0.A — Gitea artifact fetch
+
+### Problem
+Three binaries/packages are built in Gitea and must land inside the chroot
+during `lb build`. We will not commit binaries to this repo.
+
+### Strategy
+A chroot hook [`0010-fetch-artifacts.hook.chroot`](../iso/config/hooks/live/0010-fetch-artifacts.hook.chroot)
+that:
+
+1. Checks for [`iso/local-artifacts/`](../iso/local-artifacts) first (offline
+   dev builds). If present, copy from there and skip the network path.
+2. Otherwise `curl`s each artifact from Gitea release URLs. Auth via
+   `GITEA_TOKEN` environment variable passed through to the hook via
+   live-build's `--bootstrap-options` or a `.env`-style sourced file in
+   `auto/config`.
+3. Verifies each download against [`iso/SHA256SUMS`](../iso/SHA256SUMS).
+   Any mismatch fails the build.
+4. Installs:
+   - `fips_*.deb` → `dpkg -i` (pulls in systemd units, `fipsctl`, `fipstop`)
+   - `c_relay_x86` → `/opt/c-relay/c_relay_x86`, chmod 0755, owned by
+     system user `c-relay`
+   - `ginxsom-fcgi_static_x86_64` → `/usr/local/bin/ginxsom/ginxsom-fcgi`
+5. Removes the download staging directory and runs `apt-get clean`.
+
+### Open questions (in todos)
+- Are the Gitea repos/releases public or do they need a token?
+- How do we pass the token through to `lb build` (it runs as root)?
+- Do we build `fips` from source via `cargo-deb` inside the chroot as
+  a fallback, or only consume a prebuilt `.deb`?
+
+## 6. Phase 2.B — nginx + TLS template
+
+### Listening surface
+
+| Path | Method | Handler | Rationale |
+|---|---|---|---|
+| `/` | GET | static index.html from `/var/www/html/` | Landing page |
+| `/relay` | GET upgrade | `proxy_pass http://127.0.0.1:8888` with WebSocket upgrade headers | c-relay Nostr wss |
+| `/admin/` | GET | `proxy_pass http://127.0.0.1:8888/api/` | c-relay embedded admin |
+| `^/[a-f0-9]{64}$` | GET HEAD | `try_files /var/www/blobs/$uri =404` | Direct disk serve — [`ginxsom's`](../includes/ginxsom/README.md:33) core design |
+| `/upload` | PUT HEAD | `fastcgi_pass unix:/tmp/ginxsom-fcgi.sock` | Authenticated upload |
+| `^/[a-f0-9]{64}$` | DELETE | `fastcgi_pass unix:/tmp/ginxsom-fcgi.sock` | Authenticated delete |
+| `/list/` | GET | `fastcgi_pass unix:/tmp/ginxsom-fcgi.sock` | Per-pubkey listing |
+| `/mirror` | PUT | `fastcgi_pass unix:/tmp/ginxsom-fcgi.sock` | BUD-04 |
+| `/report` | PUT | `fastcgi_pass unix:/tmp/ginxsom-fcgi.sock` | BUD-09 |
+
+### TLS posture — chosen default
+
+**Self-signed cert generated on first boot**, valid for the box's hostname
+and any DNS names in `/etc/n-os-tr/tls-names`. Reason: the ISO has no idea
+what domain name it will live under, and most first users will run on a
+LAN or behind another reverse proxy. A self-signed cert means wss:// works
+out of the box for clients that trust it.
+
+We ship an enable-certbot helper script for users who have a real domain:
+
+```
+n-os-tr-certbot <domain>
+```
+
+This stops nginx, runs `certbot certonly --standalone`, rewrites the cert
+paths in the nginx config, and restarts nginx.
+
+### Key nginx template snippets
+
+WebSocket proxy to c-relay (the Upgrade dance is mandatory):
+
+```nginx
+location /relay {
+    proxy_pass http://127.0.0.1:8888;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_read_timeout 86400;
+}
+```
+
+Direct blob serve with FastCGI fallback for non-GET:
+
+```nginx
+# GET of a 64-hex path: serve straight from disk
+location ~ "^/(?<hash>[a-f0-9]{64})$" {
+    root /var/www/blobs;
+    try_files /$hash @ginxsom;
+}
+location @ginxsom {
+    fastcgi_pass unix:/tmp/ginxsom-fcgi.sock;
+    include fastcgi_params;
+}
+```
+
+Uploads:
+
+```nginx
+location /upload {
+    client_max_body_size 100m;
+    fastcgi_pass unix:/tmp/ginxsom-fcgi.sock;
+    include fastcgi_params;
+}
+```
+
+## 7. Phase 3.C — First-boot provisioning
+
+### Unit
+
+```ini
+# /etc/systemd/system/n-os-tr-firstboot.service
+[Unit]
+Description=n-OS-tr first-boot key provisioning
+ConditionPathExists=!/var/lib/n-os-tr/.provisioned
+Before=c-relay.service ginxsom.service fips.service nginx.service
+RequiredBy=c-relay.service ginxsom.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/n-os-tr-firstboot
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+```
+
+### The script
+
+Outline of `/usr/local/sbin/n-os-tr-firstboot`:
+
+```bash
+#!/bin/bash
+set -euo pipefail
+
+STATE=/var/lib/n-os-tr
+KEYS=$STATE/keys
+ENV=$STATE/env
+
+mkdir -p "$STATE" "$KEYS" /var/www/blobs /var/lib/c-relay /var/lib/ginxsom
+chmod 0700 "$KEYS"
+
+# 1. ginxsom server keypair
+if [[ ! -f $KEYS/ginxsom.nsec ]]; then
+    nak key generate > "$KEYS/ginxsom.nsec"
+    chmod 0400 "$KEYS/ginxsom.nsec"
+fi
+GINX_SEC=$(nak decode "$(cat "$KEYS/ginxsom.nsec")" | jq -r .private_key)
+
+# 2. fips identity (optional persistent mode)
+# If operator wants persistent identity, uncomment:
+# if [[ ! -f /etc/fips/fips.key ]]; then
+#     fips keygen > /etc/fips/fips.key
+#     chmod 0600 /etc/fips/fips.key
+# fi
+
+# 3. self-signed TLS cert if none
+if [[ ! -f /etc/ssl/n-os-tr/fullchain.pem ]]; then
+    mkdir -p /etc/ssl/n-os-tr
+    openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
+        -subj "/CN=$(hostname)" \
+        -keyout /etc/ssl/n-os-tr/privkey.pem \
+        -out   /etc/ssl/n-os-tr/fullchain.pem
+fi
+
+# 4. Write the environment file consumed by ginxsom.service
+cat > "$ENV" <<EOF
+GINXSOM_SERVER_PRIVKEY=$GINX_SEC
+EOF
+chmod 0400 "$ENV"
+
+# 5. Ownership
+chown -R www-data:www-data /var/www/blobs
+chown -R c-relay:c-relay   /var/lib/c-relay     2>/dev/null || true
+
+touch "$STATE/.provisioned"
+```
+
+Notes:
+- c-relay is intentionally **not** provisioned here. On first start it
+  generates its own admin keypair and prints the nsec to the journal. The
+  operator is expected to grab it from `journalctl -u c-relay`.
+- The `ginxsom.service` we ship is a modified version that replaces the
+  hardcoded `--server-privkey` with `--server-privkey ${GINXSOM_SERVER_PRIVKEY}`
+  and adds `EnvironmentFile=-/var/lib/n-os-tr/env`.
+
+### MOTD hint
+
+`/etc/motd` will include:
+
+```
+Welcome to n-OS-tr.
+
+Your Nostr relay admin key was generated on first boot. Retrieve it with:
+    journalctl -u c-relay | grep -i 'admin'
+Your Blossom server pubkey:
+    cat /var/lib/n-os-tr/keys/ginxsom.nsec
+Your FIPS identity:
+    fipsctl identity
+```
+
+## 8. Identity / persistence posture
+
+Three supported modes, one default:
+
+| Mode | Who it's for | What happens on reboot |
+|---|---|---|
+| **Ephemeral (default)** | Demos, privacy-maximal users, disposable relays | All keys regenerate, all data is lost |
+| **Persistence partition** | Users running from USB who want continuity | Second partition labeled `persistence` with `persistence.conf` listing `/var/lib/n-os-tr`, `/var/lib/c-relay`, `/var/lib/ginxsom`, `/var/www/blobs` as `union` mounts. Keys and data survive reboots of that stick. |
+| **Installed to disk** | Long-running nodes | Normal Debian behavior; user ran the debian-installer from the live session |
+
+All three work from the same ISO image. The user chooses posture by how
+they write the ISO and what partitions they create.
+
+## 9. Known traps / risks
+
+1. **c-relay has its own port** (8888) and its own `/api/` admin path,
+   while nginx also wants to serve `/admin/`. We either reverse-proxy
+   `/admin/` → `localhost:8888/api/` or expose c-relay's admin on a
+   separate port bound to localhost and forward it over SSH only.
+2. **`ginxsom` expects spawn-fcgi** per the unit file. Make sure
+   `spawn-fcgi` is in `base.list.chroot`.
+3. **libwebsockets ABI** — the c-relay binary is built static-musl per
+   [`STATIC_MUSL_GUIDE.md`](../includes/c-relay/STATIC_MUSL_GUIDE.md), so
+   we don't need to match the libwebsockets soname in Debian. Good.
+4. **ginxsom static binary** is x86_64 only today; arm64 comes in Phase 6.
+5. **`fips` Debian package** declares its own systemd units; don't
+   duplicate them via `includes.chroot` — install via `dpkg -i` and let
+   the postinst do the work.
+6. **Live-build caches aggressively**; remember `lb clean --purge` between
+   structural changes.
+
+## 10. Success criteria for the first real ISO
+
+A single end-to-end demo, run from a booted QEMU VM:
+
+- [ ] `systemctl status fips` → active, `fipsctl identity` prints an npub
+- [ ] `systemctl status c-relay` → active, `journalctl -u c-relay` shows the admin nsec printed once
+- [ ] `systemctl status ginxsom nginx` → both active
+- [ ] `curl -k https://localhost/` returns the landing page
+- [ ] `websocat wss://localhost/relay` accepts a NIP-01 `REQ` and returns an `EOSE`
+- [ ] `curl -k -X PUT https://localhost/upload --data-binary @foo.txt` with a signed authorization header stores a blob, then `curl -k https://localhost/<sha256>` retrieves the same bytes