# Didactyl Tool Testing Plan

## Overview

Testing agent-mediated tools is different from unit testing pure functions because the execution path spans multiple boundaries:

```
LLM schema interpretation → JSON argument generation → argument parsing → business logic → Nostr/network I/O → JSON response → LLM interpretation
```

This plan defines three testing layers, ordered by implementation priority.

---

## Layer 1: Direct Tool Execution

Call `tools_execute()` directly with known JSON arguments and assert the JSON response structure.

### Implementation

Add a `--test-tool <name> <args_json>` CLI flag to `src/main.c` that:
1. Initializes config and nostr_handler (for network-dependent tools)
2. Calls `tools_execute(&ctx, name, args_json)`
3. Prints the raw JSON result to stdout
4. Exits with 0 if `success: true`, 1 otherwise

### Pure Computation Tools (no network needed)

| Tool | Test Command | Expected |
|------|-------------|----------|
| `nostr_encode` | `--test-tool nostr_encode '{"type":"npub","hex":"<64-char-hex>"}'` | `success: true`, uri starts with `nostr:npub1` |
| `nostr_decode` | `--test-tool nostr_decode '{"uri":"npub1..."}'` | `success: true`, pubkey is 64-char hex |
| `nostr_encrypt` | `--test-tool nostr_encrypt '{"recipient_pubkey":"<hex>","plaintext":"hello"}'` | `success: true`, ciphertext is base64 |
| `nostr_decrypt` | `--test-tool nostr_decrypt '{"sender_pubkey":"<hex>","ciphertext":"<from encrypt>"}'` | `success: true`, plaintext is "hello" |

### Network-Dependent Tools (need relay or HTTP)

| Tool | Test Command | Expected |
|------|-------------|----------|
| `nostr_nip05_lookup` | `--test-tool nostr_nip05_lookup '{"identifier":"_@laantungir.com"}'` | `success: true`, pubkey returned |
| `nostr_relay_info` | `--test-tool nostr_relay_info '{"relay_url":"wss://relay.damus.io"}'` | `success: true`, info.basic.name present |
| `nostr_relay_status` | `--test-tool nostr_relay_status '{}'` | `success: true`, relay_count > 0 |
| `nostr_dm_send` | `--test-tool nostr_dm_send '{"recipient_pubkey":"<hex>","message":"test"}'` | `success: true` |
| `nostr_post` | `--test-tool nostr_post '{"kind":1,"content":"test"}'` | `success: true`, event_id present |
| `nostr_delete` | `--test-tool nostr_delete '{"event_ids":["<64-char-hex>"]}'` | `success: true` |
| `nostr_react` | `--test-tool nostr_react '{"event_id":"<hex>","event_pubkey":"<hex>"}'` | `success: true` |
| `nostr_profile_get` | `--test-tool nostr_profile_get '{"pubkey":"<hex>"}'` | `success: true`, found: true/false |
| `nostr_dm_send_nip17` | `--test-tool nostr_dm_send_nip17 '{"recipient_pubkey":"<hex>","message":"test"}'` | `success: true` |
| `nostr_list_manage` | `--test-tool nostr_list_manage '{"list_kind":10000,"action":"add","items":[["p","<hex>"]]}'` | `success: true` |

### Error Case Tests

Each tool should also be tested with:
- Empty args: `'{}'` — should return descriptive error
- Missing required fields — should return specific error message
- Invalid hex strings — should reject gracefully
- Double-encoded JSON string args — should parse correctly (regression for the `invalid arguments JSON` bug)

---

## Layer 2: Schema-Parse Fidelity

Validates that the OpenAI function schema matches what executors actually accept.

### Implementation

A Python script (`tests/validate_schemas.py`) that:
1. Runs `didactyl_static --dump-schemas` (new flag) to get the tool schema JSON
2. For each tool in the schema:
   - Generates a minimal valid argument payload from `required` + `properties`
   - Runs `didactyl_static --test-tool <name> '<generated_json>'`
   - Asserts exit code 0 or expected network error
3. Reports mismatches between schema and executor expectations

### What This Catches

- Schema says `required: ["hex"]` but executor checks for `"pubkey"` — mismatch
- Schema says `type: "integer"` but executor reads it as string
- Schema advertises parameters the executor ignores
- Missing required parameters in schema that executor demands

---

## Layer 3: Agent Integration Testing (Live DM)

The most natural test — DM the running agent and verify it uses tools correctly.

### Prerequisites

- Local relay running at `ws://127.0.0.1:7777`
- Didactyl running with valid config
- A separate Nostr client (or script) to send/receive DMs

### Test Prompts

| # | Tool | DM Prompt | Assert in Response |
|---|------|-----------|-------------------|
| 1 | `nostr_encode` | "Encode this pubkey as npub: `<64-char hex>`" | Contains `nostr:npub1` |
| 2 | `nostr_decode` | "Decode this npub: `npub1...`" | Contains the hex pubkey |
| 3 | `nostr_dm_send` | "Send a DM to `<your-own-pubkey>` saying 'hello test'" | Confirms sent; you receive it |
| 4 | `nostr_encrypt` | "Encrypt 'secret message' for `<pubkey>`" | Contains base64 ciphertext |
| 5 | `nostr_decrypt` | "Decrypt this NIP-44 payload: `<ciphertext from #4>`" | Contains 'secret message' |
| 6 | `nostr_nip05_lookup` | "Look up `_@laantungir.com`" | Returns a pubkey |
| 7 | `nostr_relay_info` | "Get NIP-11 info for `wss://relay.damus.io`" | Returns relay name and supported NIPs |
| 8 | `nostr_relay_status` | "Show me relay connection status" | Lists connected relays with stats |
| 9 | `nostr_react` | "React with 🤙 to event `<id>` from `<pubkey>`" | Confirms kind 7 published |
| 10 | `nostr_delete` | "Delete event `<id>`" | Confirms kind 5 published |
| 11 | `nostr_profile_get` | "Look up the profile for `<pubkey>`" | Returns name/about/picture |
| 12 | `nostr_post` | "Post a kind 1 note saying 'tool test'" | Confirms published with event_id |
| 13 | `nostr_list_manage` | "Add `<pubkey>` to my mute list" | Confirms kind 10000 published |
| 14 | `nostr_dm_send_nip17` | "Send a private NIP-17 DM to `<pubkey>` saying 'gift wrap test'" | Confirms gift wrap sent |
| 15 | `nostr_post_readme` | "Publish the README to Nostr" | Confirms kind 30023 with d=readme.md |

### Verification Methods

- **stdout logs**: Watch `[didactyl] executing tool call: <name>` in terminal
- **context.log**: Full LLM conversation including tool calls and results
- **Relay inspection**: Query the local relay for published events
- **DM receipt**: For DM tools, verify the message arrives at the recipient

---

## Implementation Priority

1. **Immediate (no code changes)**: Run Layer 3 tests by DMing the live agent
2. **Next sprint**: Add `--test-tool` CLI flag for Layer 1
3. **Later**: Add `--dump-schemas` flag and Python schema validator for Layer 2
4. **CI integration**: Wrap Layer 1 pure-computation tests in a shell script that runs after `build_static.sh`

---

## Local Relay Setup for Testing

A local relay like `strfry` or `nostr-rs-relay` at `ws://127.0.0.1:7777` provides:
- All published events are observable and queryable
- No rate limits or content policies
- NIP-42 auth testing in isolation
- Gift-wrapped NIP-17 DMs stay in your test environment
- Database can be wiped between test runs