didactyl/docs/CRASH_FIXES.md

Crash Fixes Reference

This document catalogues crashes that have been diagnosed and fixed in Didactyl. Use it as a reference when investigating future crashes — the symptoms, root causes, and investigation techniques described here may save significant debugging time.

---

1. Silent process exit on relay disconnect — SIGPIPE (v0.0.24)

Date: 2026-03-02
Version: v0.0.23 → fixed in v0.0.24
Severity: Critical — process terminates without any log output
Files changed: src/main.c

Symptoms

• The process exits cleanly to the shell prompt with no error message.
• The [didactyl] shutting down log line does not appear.
• No coredump is generated.
• The last log lines show all relays transitioning from connected → disconnected and then immediately disconnected → connected.
• The crash typically follows a period of network instability or DNS resolution failure — for example, an LLM HTTP request failing with curl=Could not resolve hostname.

Root cause

SIGPIPE was never handled. The default OS action for SIGPIPE is to terminate the process immediately — no signal handler runs, no cleanup occurs, no coredump is written.

The TLS write path in nostr_core_lib uses SSL_write(), which internally calls write() on the underlying socket. When a wss:// relay drops its TCP connection and the relay pool attempts to write to that socket, the kernel delivers SIGPIPE.

The plain-TCP path for ws:// connections was already protected by using MSG_NOSIGNAL on send() calls, but the TLS path had no equivalent protection.

How it was diagnosed

1. No crash message or coredump ruled out SIGSEGV, SIGABRT, and OOM.
2. No shutdown log ruled out a graceful exit via the signal handler.
3. dmesg and journalctl showed no OOM-kill or segfault entries.
4. Searching the entire codebase for SIGPIPE or SIG_IGN returned zero results.
5. The websocket TLS transport was confirmed to use SSL_write() without MSG_NOSIGNAL or any per-thread signal mask.
6. The timeline matched: relay disconnects occurred, the next poll cycle attempted a write to a dead TLS socket, and the process vanished.

Fix

Added signal(SIGPIPE, SIG_IGN) in main() alongside the existing SIGINT and SIGTERM handlers:

signal(SIGINT, signal_handler);
signal(SIGTERM, signal_handler);
signal(SIGPIPE, SIG_IGN);          /* ← added */

This causes SSL_write() and any other write to a broken pipe to return -1 with errno = EPIPE instead of killing the process. The relay pool and libcurl already handle write errors gracefully.

How to recognise this class of bug in the future

• Process disappears without any log output or coredump.
• Happens after network disruption or relay disconnects.
• ulimit -c may be 0, but even with unlimited core size, SIGPIPE does not produce a coredump by default.
• Any new network transport layer added to the project should be audited for SIGPIPE protection.

---

2. Use-after-free in execute_nostr_list_manage — SIGSEGV (v0.0.24)

Date: 2026-03-01 (coredump), fixed 2026-03-02
Version: v0.0.23 → fixed in v0.0.24
Severity: Critical — segmentation fault during tool execution
Files changed: src/tools.c

Symptoms

• SIGSEGV crash during execution of the nostr_list_manage tool.
• Coredump shows the crash at src/tools.c:29 inside json_error(), but with heavily corrupted stack frames — the real crash site is in execute_nostr_list_manage().
• The corrupted stack is characteristic of heap corruption from use-after-free.

Root cause

In execute_nostr_list_manage(), the action pointer was obtained from the args cJSON tree:

cJSON* action = cJSON_GetObjectItemCaseSensitive(args, "action");

Later, args was freed:

cJSON_Delete(args);          /* frees the entire tree including action */

But action->valuestring was still accessed afterwards:

cJSON_AddStringToObject(out, "action", action->valuestring);  /* dangling pointer */

After cJSON_Delete(args), the action pointer is dangling. Accessing action->valuestring reads freed heap memory, which may contain arbitrary data or may have been reallocated for another purpose.

How it was diagnosed

1. The coredump was extracted from systemd-coredump storage:

   zstd -d /var/lib/systemd/coredump/core.didactyl_static.*.zst -o /tmp/core.bin
   gdb ./didactyl_static_x86_64_debug /tmp/core.bin -batch -ex "bt"
   

2. The backtrace showed execute_nostr_list_manage with corrupted frames.
3. Code review of the function identified the cJSON_Delete(args) call occurring before the last use of action->valuestring.

Fix

Deferred cJSON_Delete(args) until after action->valuestring has been consumed by cJSON_AddStringToObject(). The free now occurs immediately after the last use:

cJSON_AddStringToObject(out, "action", action->valuestring);
cJSON_Delete(args);          /* ← moved here, after last use of action */

All early-return error paths before this point also received their own cJSON_Delete(args) call to prevent leaks.

How to recognise this class of bug in the future

• SIGSEGV with corrupted or nonsensical stack frames.
• Crash location reported by GDB does not match the actual buggy code — the corruption happened earlier.
• Any function that calls cJSON_Delete() on a parent object should be audited to ensure no child pointers are used afterwards.
• Pattern to watch for:

  cJSON* child = cJSON_GetObjectItemCaseSensitive(parent, "key");
  /* ... */
  cJSON_Delete(parent);
  /* ... */
  use(child->valuestring);   /* BUG: child is dangling */
  

---

3. DM subscription not receiving incoming kind 4 events — INVESTIGATION (v0.0.24)

Date: 2026-03-02 Version: v0.0.24 Severity: High — agent does not respond to incoming DMs Status: Under investigation — diagnostic logging added Files changed: src/nostr_handler.c

Symptoms

• The agent starts up normally, connects to relays, and can send kind 4 DMs.
• Incoming kind 4 messages posted to the same relays are never processed.
• No error messages appear in the log — the agent simply sits idle after sending its startup DM.
• The last log lines show successful outbound DM publishing but no inbound event processing.

Possible root causes (under investigation)

1. Events never reach on_event() callback — The relay pool library only calls on_event() when it receives an EVENT message matching the subscription ID. If the subscription was silently closed, errored, or not re-established after a relay reconnect, no events would be delivered. The relay could also be rejecting the subscription filter.

2. since filter timing mismatch — g_start_time is set to time(NULL) during nostr_handler_init() (early in startup), but the DM subscription is created much later (after admin context subscription, startup event reconciliation, and startup DM sending). If the sender's clock is behind the server's clock, their created_at timestamp would be before g_start_time and the relay would filter them out.

3. Sender tier filtering — If the sender's pubkey doesn't match the admin pubkey and isn't in the WoT contact list, the message is classified as DIDACTYL_SENDER_STRANGER and silently dropped (only a DEBUG_LOG at level 4).

4. Signature verification failure — If verify_signatures is enabled and the event has an invalid signature, it is dropped with only a WARN log.

5. Pool-level deduplication — The relay pool's is_event_seen() cache is shared across all subscriptions. If an event ID was somehow marked as seen by another subscription, it would be silently dropped.

Diagnostic logging added

TRACE-level (level 5 / --debug 5) logs were added at every decision point in the on_event() callback and the nostr_handler_subscribe_dms() setup:

Log message prefix	What it tells you
DEBUG on_event ENTRY	Callback was called — events are reaching didactyl
DEBUG on_event NULL guard	Event, config, or callback pointer is NULL
DEBUG on_event: missing required fields	Event JSON is malformed
DEBUG on_event: kind=N id=... from=...	Event kind, ID, and sender pubkey
DEBUG on_event: ignoring non-kind4	Event is not kind 4
DEBUG on_event: no p-tag found	Kind 4 event has no p tag
DEBUG on_event: p-tag mismatch	p tag doesn't match agent's pubkey
DEBUG on_event: sender=... tier=N	Sender tier classification
DEBUG on_eose called	EOSE received from relays
DEBUG DM subscription filter	Full subscription filter JSON
DEBUG DM subscription g_start_time=...	since timestamp and time delta
DEBUG DM subscription sub=...	Subscription pointer (confirms creation)

How to use the diagnostic logs

1. Build with ./build_static.sh --debug
2. Run with --debug 5 to enable TRACE output
3. Send a kind 4 DM to the agent
4. Check the output:
   • No on_event ENTRY lines → problem is at the relay/subscription level
   • on_event ENTRY appears but processing stops → the specific drop-point log identifies the exact filter that rejected the event
5. Check the DM subscription filter log to verify the since timestamp and #p tag are correct

How to recognise this class of bug in the future

• Agent can send but not receive — asymmetric connectivity.
• No error messages in the log — silent event filtering.
• The subscription filter (since, #p, kinds) may not match what the sender is actually publishing.
• Clock skew between sender and receiver can cause since filter mismatches.
• Relay reconnections may not automatically re-subscribe.

---

General debugging checklist

When investigating a crash in Didactyl, work through these steps:

1. Check for coredumps: ls /var/lib/systemd/coredump/ | grep didactyl
2. Check dmesg/journalctl: dmesg | grep -i 'didactyl\|oom\|segfault'
3. Check for the shutdown log line: If [didactyl] shutting down is missing, the process was killed by a signal that bypassed the handler.
4. Check ulimit: ulimit -c — if 0, coredumps are disabled.
5. Extract and analyse coredumps:

   zstd -d /var/lib/systemd/coredump/core.didactyl*.zst -o /tmp/core.bin
   gdb ./didactyl_static_x86_64_debug /tmp/core.bin -batch -ex "bt full"
   

6. Common silent killers:
   • SIGPIPE — process writes to a broken socket/pipe
   • SIGKILL — OOM killer or external kill
   • SIGBUS — memory-mapped file issues
7. Common crash causes:
   • Use-after-free on cJSON child pointers after parent deletion
   • Buffer overflows in fixed-size stack buffers
   • NULL pointer dereference on failed allocations
   • Re-entrancy in callbacks during internal nostr_relay_pool_poll() calls