didactyl/plans/tool_testing.md

Didactyl Tool Testing Plan

Overview

Testing agent-mediated tools is different from unit testing pure functions because the execution path spans multiple boundaries:

LLM schema interpretation → JSON argument generation → argument parsing → business logic → Nostr/network I/O → JSON response → LLM interpretation

This plan defines three testing layers, ordered by implementation priority.

---

Layer 1: Direct Tool Execution

Call tools_execute() directly with known JSON arguments and assert the JSON response structure.

Implementation

Add a --test-tool <name> <args_json> CLI flag to src/main.c that:

1. Initializes config and nostr_handler (for network-dependent tools)
2. Calls tools_execute(&ctx, name, args_json)
3. Prints the raw JSON result to stdout
4. Exits with 0 if success: true, 1 otherwise

Pure Computation Tools (no network needed)

Tool	Test Command	Expected
nostr_encode	--test-tool nostr_encode '{"type":"npub","hex":"<64-char-hex>"}'	success: true, uri starts with nostr:npub1
nostr_decode	--test-tool nostr_decode '{"uri":"npub1..."}'	success: true, pubkey is 64-char hex
nostr_encrypt	--test-tool nostr_encrypt '{"recipient_pubkey":"<hex>","plaintext":"hello"}'	success: true, ciphertext is base64
nostr_decrypt	--test-tool nostr_decrypt '{"sender_pubkey":"<hex>","ciphertext":"<from encrypt>"}'	success: true, plaintext is "hello"

Network-Dependent Tools (need relay or HTTP)

Tool	Test Command	Expected
nostr_nip05_lookup	--test-tool nostr_nip05_lookup '{"identifier":"_@laantungir.com"}'	success: true, pubkey returned
nostr_relay_info	--test-tool nostr_relay_info '{"relay_url":"wss://relay.damus.io"}'	success: true, info.basic.name present
nostr_relay_status	--test-tool nostr_relay_status '{}'	success: true, relay_count > 0
nostr_dm_send	--test-tool nostr_dm_send '{"recipient_pubkey":"<hex>","message":"test"}'	success: true
nostr_post	--test-tool nostr_post '{"kind":1,"content":"test"}'	success: true, event_id present
nostr_delete	--test-tool nostr_delete '{"event_ids":["<64-char-hex>"]}'	success: true
nostr_react	--test-tool nostr_react '{"event_id":"<hex>","event_pubkey":"<hex>"}'	success: true
nostr_profile_get	--test-tool nostr_profile_get '{"pubkey":"<hex>"}'	success: true, found: true/false
nostr_dm_send_nip17	--test-tool nostr_dm_send_nip17 '{"recipient_pubkey":"<hex>","message":"test"}'	success: true
nostr_list_manage	--test-tool nostr_list_manage '{"list_kind":10000,"action":"add","items":[["p","<hex>"]]}'	success: true

Error Case Tests

Each tool should also be tested with:

• Empty args: '{}' — should return descriptive error
• Missing required fields — should return specific error message
• Invalid hex strings — should reject gracefully
• Double-encoded JSON string args — should parse correctly (regression for the invalid arguments JSON bug)

---

Layer 2: Schema-Parse Fidelity

Validates that the OpenAI function schema matches what executors actually accept.

Implementation

A Python script (tests/validate_schemas.py) that:

1. Runs didactyl_static --dump-schemas (new flag) to get the tool schema JSON
2. For each tool in the schema:
   • Generates a minimal valid argument payload from required + properties
   • Runs didactyl_static --test-tool <name> '<generated_json>'
   • Asserts exit code 0 or expected network error
3. Reports mismatches between schema and executor expectations

What This Catches

• Schema says required: ["hex"] but executor checks for "pubkey" — mismatch
• Schema says type: "integer" but executor reads it as string
• Schema advertises parameters the executor ignores
• Missing required parameters in schema that executor demands

---

Layer 3: Agent Integration Testing (Live DM)

The most natural test — DM the running agent and verify it uses tools correctly.

Prerequisites

• Local relay running at ws://127.0.0.1:7777
• Didactyl running with valid config
• A separate Nostr client (or script) to send/receive DMs

Test Prompts

#	Tool	DM Prompt	Assert in Response
1	nostr_encode	"Encode this pubkey as npub: <64-char hex>"	Contains nostr:npub1
2	nostr_decode	"Decode this npub: npub1..."	Contains the hex pubkey
3	nostr_dm_send	"Send a DM to <your-own-pubkey> saying 'hello test'"	Confirms sent; you receive it
4	nostr_encrypt	"Encrypt 'secret message' for <pubkey>"	Contains base64 ciphertext
5	nostr_decrypt	"Decrypt this NIP-44 payload: <ciphertext from #4>"	Contains 'secret message'
6	nostr_nip05_lookup	"Look up _@laantungir.com"	Returns a pubkey
7	nostr_relay_info	"Get NIP-11 info for wss://relay.damus.io"	Returns relay name and supported NIPs
8	nostr_relay_status	"Show me relay connection status"	Lists connected relays with stats
9	nostr_react	"React with 🤙 to event <id> from <pubkey>"	Confirms kind 7 published
10	nostr_delete	"Delete event <id>"	Confirms kind 5 published
11	nostr_profile_get	"Look up the profile for <pubkey>"	Returns name/about/picture
12	nostr_post	"Post a kind 1 note saying 'tool test'"	Confirms published with event_id
13	nostr_list_manage	"Add <pubkey> to my mute list"	Confirms kind 10000 published
14	nostr_dm_send_nip17	"Send a private NIP-17 DM to <pubkey> saying 'gift wrap test'"	Confirms gift wrap sent
15	nostr_post_readme	"Publish the README to Nostr"	Confirms kind 30023 with d=readme.md

Verification Methods

• stdout logs: Watch [didactyl] executing tool call: <name> in terminal
• context.log: Full LLM conversation including tool calls and results
• Relay inspection: Query the local relay for published events
• DM receipt: For DM tools, verify the message arrives at the recipient

---

Implementation Priority

1. Immediate (no code changes): Run Layer 3 tests by DMing the live agent
2. Next sprint: Add --test-tool CLI flag for Layer 1
3. Later: Add --dump-schemas flag and Python schema validator for Layer 2
4. CI integration: Wrap Layer 1 pure-computation tests in a shell script that runs after build_static.sh

---

Local Relay Setup for Testing

A local relay like strfry or nostr-rs-relay at ws://127.0.0.1:7777 provides:

• All published events are observable and queryable
• No rate limits or content policies
• NIP-42 auth testing in isolation
• Gift-wrapped NIP-17 DMs stay in your test environment
• Database can be wiped between test runs