didactyl/plans/startup_failure_admin_dm.md

Startup Failure Admin DM Notification

STATUS UPDATE (follow-up phase): The base feature (send a failure DM) is implemented and verified on VM410. However, because the failure happens inside the systemd Restart=on-failure loop (RestartSec=10), the admin now receives a new failure DM roughly every ~25 seconds, indefinitely (observed in the live message box). This follow-up adds throttling/de-duplication so the failure DM is sent once per failure type until the next successful startup, and makes the message more descriptive (per-relay connect state + the event kinds that were needed). See the "Follow-up: Throttling + Descriptive Failure DM" section at the end.

Startup Failure Admin DM Notification (original plan)

Goal

When the agent fails to start, send a best-effort encrypted DM to the administrator describing which startup step failed and why, so the operator learns about the failure without having to SSH in and read journalctl.

This directly addresses the real-world incident on VM410 where both didactyl.service and simon.service were stuck restart-looping at startup step 14 (Subscribe self-skill cache: failed (no skill events found after EOSE)) and silently stopped answering DMs.

Background / Code References

• Startup is a linear checklist in main(). Each step uses helpers:
  • startup_step_begin()
  • startup_step_ok()
  • startup_step_fail()
• On any fatal failure, the current pattern is: startup_step_fail(...) -> fprintf(stderr, ...) -> cleanup calls -> return 1; (e.g. the step-14 self-skill abort at src/main.c:1520).
• The success path already sends an admin "started up" DM at src/main.c:1663 via nostr_handler_send_dm_auto().
• DM delivery requires (see nostr_handler_send_dm_with_role()):
  • g_cfg and g_pool set (relay pool initialized),
  • at least one connected relay (otherwise the kind-4 event is dropped: "kind 4 event not queued: no connected relays" at src/nostr_handler.c:3203),
  • a valid cfg.admin.pubkey (validated at step 4, src/main.c:1310),
  • the private key derived (always done before the checklist).

Delivery-window reality

A failure DM is only deliverable from roughly step 11 onward (relay pool created + relays connected + admin known). The step-14 failure we actually hit is inside that window, so it will be delivered. Earlier failures (bad nsec, missing admin, LLM config, no relay connection) have no working channel; per decision, those are logged only (best-effort, no blocking retries).

Design Decision (confirmed)

Best-effort only. Attempt the failure DM whenever relays + admin are available; if the relay pool isn't connected or admin pubkey is unknown, skip the DM and rely on existing stderr/journal logging. No blocking retry loop, no self-healing of the underlying failure (that can be a separate plan).

Approach

Add a single helper that builds and sends the failure DM, then call it from every fatal startup-abort site (the points that currently do startup_step_fail(...) ... return 1;).

1. New helper in src/main.c

Add a static function near the other startup helpers (after startup_step_fail()):

static void notify_admin_startup_failure(const didactyl_config_t* cfg,
                                         int step,
                                         const char* label,
                                         const char* detail);

Behavior:

• Guard clauses (skip + DEBUG_WARN if any precondition is missing):
  • cfg == NULL or cfg->admin.pubkey[0] == '\0' -> log "admin unknown; cannot notify".
  • nostr_handler_connected_relay_count() <= 0 -> log "no connected relay; cannot notify".
• Build a concise human-readable message, e.g.:

  <DisplayName> FAILED to start at <YYYY-MM-DD HH:MM:SS>.
  Startup step [14] Subscribe self-skill cache failed: no skill events found after EOSE.
  Version <X>. Connected relays: <n>/<m>. The agent is NOT online and will not answer DMs until fixed.
  

  • Reuse nostr_handler_get_startup_display_name() and DIDACTYL_VERSION exactly like the success DM at src/main.c:1645 for naming/version consistency.
  • Include step, label, and detail so each failure site is self-describing.
• Send via nostr_handler_send_dm_auto(cfg->admin.pubkey, msg); if it returns non-zero, DEBUG_WARN that the failure DM could not be delivered (do not change the exit code).
• Pump the poll loop briefly after sending so the async publish can flush before process exit: call nostr_handler_poll(...) a few times (e.g. ~300--500 ms total) so the queued kind-4 event is actually written to the relay sockets before cleanup/return 1. (Without this, async publish may be discarded on immediate exit.)

2. Wire the helper into fatal abort sites

For each fatal startup_step_fail(...) that ends in return 1;, insert a call to notify_admin_startup_failure(&cfg, <step>, "<label>", "<detail>") immediately after the startup_step_fail(...) line and before the cleanup/return 1.

Priority sites (inside the deliverable window — relay pool up + admin known):

• Step 11 — discover self relay list failures (src/main.c:1420, src/main.c:1437).
• Step 14 — self-skill cache failures (the incident we hit):
  • subscribe failed (src/main.c:1481),
  • EOSE timeout (src/main.c:1496),
  • no skill events after EOSE (src/main.c:1521).
• Step 15 — Subscribe DMs failed (src/main.c:1552).

Earlier-window sites (relay/admin may not be ready — helper will self-skip and just log, but we still call it uniformly for completeness):

• Step 1/3/4/5/6/7/8/9 fatal aborts. The helper's guard clauses make these safe no-ops when delivery isn't possible.

Keep the existing fprintf(stderr, ...) and cleanup unchanged; the DM is purely additive and must never alter the exit code or cleanup ordering.

3. Message size / safety

• Use a fixed stack buffer (e.g. char msg[768];) like the success DM at src/main.c:1644.
• snprintf with truncation is acceptable; detail strings are short and controlled.

Out of Scope

• No blocking retry / wait-for-relay loop (explicitly declined).
• No auto-remediation of the step-14 self-skill cause (e.g. auto-publishing the default skill). Tracked separately if desired.
• No change to deploy_lt.sh post-deploy health checks (could be a follow-up: have the deploy script verify each service reaches READY after restart).

Verification

1. Build the static binary (build_static.sh) and confirm no warnings.
2. Local reproduction of step-14 failure: run an --nsec-only agent for an identity that has connected relays but no published skill events, confirm:
   • startup still aborts with exit code 1 (unchanged behavior),
   • the admin receives a DM naming step 14 and the "no skill events found after EOSE" detail.
3. Negative case: kill relay connectivity (or use an unreachable relay) and confirm the helper logs "no connected relay; cannot notify" and does not crash or hang.
4. Confirm the success-path startup DM at src/main.c:1663 is unaffected.
5. Deploy to VM410, restart didactyl.service / simon.service, and confirm the admin npub receives the failure DM while the skill events are still missing; then publish a skill and confirm normal READY + startup DM resumes.

---

Follow-up: Throttling + Descriptive Failure DM

Problems observed in production

1. DM flood. With Restart=on-failure / RestartSec=10, the step-14 failure recurs on every restart and the agent reaches the deliverable window each time, so the admin gets a fresh failure DM every ~25s indefinitely (confirmed in the live message box).
2. Insufficient detail. The current message names the step + detail, but not which relays were attempted, whether each connected, nor which event kinds were required.

Decisions (confirmed with user)

• Throttle semantics: send exactly once per failure type (keyed by step) until the agent next starts successfully. The marker is cleared on READY. No periodic reminders. A different failing step is a different "type" and notifies once.
• Descriptive content: include the attempted relay list with per-relay connect state, and the event kinds that step 14 needs (31123 / 31124 skill events; 10123 adoption list).

Part A -- Throttle / de-duplicate via on-disk marker

Marker file location (systemd-sandbox safe)

• The unit runs with WorkingDirectory=/home/<user> and (hardened variant) ReadWritePaths=/home/<user>, so only the agent home is writable.
• cfg.tools.shell.working_directory defaults to "." (see config.c), which resolves to the home dir at runtime.
• Use a hidden marker in the working directory: ./.didactyl_startup_failure.
  • For didactyl -> /home/didactyl/.didactyl_startup_failure
  • For simon -> /home/simon/.didactyl_startup_failure
  • Both inside the writable ReadWritePaths, so writes succeed under the sandbox.
• Resolve the directory the same way the local-file tools do: prefer cfg->tools.shell.working_directory when non-empty, else "." (mirror tool_local.c).

Marker contents

Plain text, one record, easy to parse and inspect manually:

• step=14
• label=Subscribe self-skill cache
• detail=no skill events found after EOSE
• last_sent_epoch=1781314692

(Only step is strictly needed for the de-dup key; the rest aids debugging.)

New helpers in src/main.c

Add small static helpers near notify_admin_startup_failure():

• startup_failure_marker_path(cfg, out, out_size) -- builds <workdir>/.didactyl_startup_failure.
• startup_failure_already_notified(cfg, step) -- returns 1 if the marker exists AND its step= matches the current step (suppress); 0 otherwise (allow). Missing/unreadable marker -> 0 (allow).
• startup_failure_marker_write(cfg, step, label, detail) -- writes/overwrites the marker after an attempted send.
• startup_failure_marker_clear(cfg) -- remove() the marker; ignore ENOENT.

Wiring

• In notify_admin_startup_failure(): after the existing guard clauses (config/admin/relay), check startup_failure_already_notified(cfg, step). If it returns 1, log a DEBUG_INFO suppression line and return without sending.
  • On a successful send, call startup_failure_marker_write(cfg, step, label, detail).
  • If the same step keeps failing, only the first restart in the episode sends a DM; subsequent restarts find the matching marker and stay silent.
  • If a different step fails later, the step= mismatch allows exactly one new DM and rewrites the marker with the new step.
• On the success path, clear the marker so the next failure episode notifies again. Add startup_failure_marker_clear(&cfg); next to the READY success DM at src/main.c:1663.

Edge cases

• Marker write failure (e.g. read-only FS): log DEBUG_WARN and continue. Worst case we fall back to current behavior (may re-send on next restart) -- acceptable, never fatal.
• Two services share the binary but use different home dirs, so markers never collide.

Part B -- Descriptive failure message

Enhance the message built in notify_admin_startup_failure() to append:

1. Attempted relays + per-relay state. Add a compact accessor to the nostr handler:
   • New nostr_handler_relay_state_summary(void) in nostr_handler.c / nostr_handler.h returning a caller-freed compact string like relay.laantungir.net=connected; relay.damus.io=connected; relay.primal.net=connecting.
   • Implement by reusing the existing loop over g_cfg->relays[] + nostr_relay_pool_get_relay_status() + relay_status_str() (patterns at src/nostr_handler.c:3183 and src/nostr_handler.c:419).
   • Preferred over embedding raw JSON from nostr_handler_relay_status_json() to keep the DM short.
2. Needed event kinds (step-specific). For step 14 append a fixed clause naming the required events: kind 31123/31124 (skills) and kind 10123 (adoption list). Implement as a small startup_step_needed_events(step) lookup in src/main.c that returns a clause for known steps (14, 11, ...) and empty for others.

Resulting message shape (example)

• Didactyl FAILED to start at 2026-06-13 10:38:13.
• Startup step [14] Subscribe self-skill cache failed: no skill events found after EOSE.
• Version v0.2.48. Connected relays: 3/3.
• Relays: relay.laantungir.net=connected; relay.damus.io=connected; relay.primal.net=connected.
• Needed events: kind 31123/31124 (skills) and kind 10123 (adoption list) for this agent's pubkey.
• The agent is NOT online and will not answer DMs until fixed.

Keep within the existing failure_dm buffer (bump to ~1536 if needed); snprintf truncation remains safe.

Implementation order

1. Add nostr_handler_relay_state_summary() to handler .c/.h.
2. Add marker helpers + startup_step_needed_events() to src/main.c.
3. Update notify_admin_startup_failure(): de-dup check, richer message, marker write on send.
4. Add startup_failure_marker_clear(&cfg) on the READY success path (src/main.c:1663).
5. Build (make), then build_static.sh, deploy, verify.

Verification (follow-up)

1. Throttle: reproduce step-14 failure; confirm exactly one DM on the first failing restart and no further DMs across subsequent restart-loop iterations (journal shows a suppression line).
2. Clear-on-success: publish a skill so startup reaches READY; confirm marker is removed and the normal startup DM is sent.
3. Re-arm: break startup again and confirm a single new DM is delivered.
4. Descriptive content: confirm the DM includes the per-relay state line and the needed-events clause.
5. Sandbox write: on VM410 confirm the marker is created at /home/didactyl/.didactyl_startup_failure (and /home/simon/...) without permission errors under the systemd sandbox.