Files

Didactyl User 9a5beac80d v0.2.16 - Add Python test harness and fix increment script commit identity handling

2026-03-25 09:22:15 -04:00

22 KiB

Raw Blame History

Didactyl Automated Test Harness

Overview

An automated testing system that starts a Didactyl agent locally (debug build), converses with it via the HTTP API, exercises all tools, monitors logs in real-time, handles agent crashes/restarts, and produces a structured test results report.

Language: Python (stdlib only, no external dependencies)
Location: tests/
Phase 1: Scripted tests (no LLM driving the tester)
Phase 2 (future): LLM-driven test agent that generates prompts, evaluates responses, and adapts

Architecture

flowchart TB
    subgraph Test Harness - Python
        RUNNER[test_runner.py<br/>orchestrator]
        PROC[agent_process.py<br/>start/stop/restart]
        CLIENT[didactyl_client.py<br/>HTTP API wrapper]
        LOG[log_watcher.py<br/>tail debug.log]
        REPORT[reporter.py<br/>results output]
        
        RUNNER --> PROC
        RUNNER --> CLIENT
        RUNNER --> LOG
        RUNNER --> REPORT
    end
    
    subgraph Test Suites
        TH[test_health]
        TC[test_conversation]
        TI[test_tools_identity]
        TN[test_tools_nostr]
        TS[test_tools_skills]
        TSY[test_tools_system]
        TM[test_tools_memory]
        TCA[test_tools_cashu]
        TB[test_tools_blossom]
        TTO[test_timeouts]
        TE[test_errors]
        TR[test_restart]
    end
    
    RUNNER --> TH & TC & TI & TN & TS & TSY & TM & TCA & TB & TTO & TE & TR
    
    subgraph Didactyl Agent - debug build
        AGENT[didactyl process]
        API[HTTP API :8484]
        LOGFILE[debug.log]
    end
    
    CLIENT -- HTTP --> API
    PROC -- subprocess --> AGENT
    LOG -- tail --> LOGFILE

Design Decisions

Decision	Choice	Rationale
Language	Python	Best subprocess/HTTP/threading support; already used in project
Dependencies	stdlib only	No pip install needed; `urllib`, `subprocess`, `threading`, `json`
Test framework	Standalone runner	Self-contained, no pytest dependency
Relay strategy	Flexible genesis config	User provides their own test_genesis.jsonc
LLM for agent	Real model	User provides API key in test genesis config
Tool scope	All tools	Disposable test identity; full coverage
Entry point	`tests/run_tests.py`	Single script to run everything

Directory Structure

tests/
├── harness/
│   ├── __init__.py
│   ├── agent_process.py      # start/stop/restart didactyl subprocess
│   ├── didactyl_client.py    # HTTP API wrapper with timeouts
│   ├── log_watcher.py        # real-time log tail + marker system
│   ├── test_runner.py        # orchestrator
│   └── reporter.py           # results formatting (JSON + text)
├── suites/
│   ├── __init__.py
│   ├── test_health.py        # status, context endpoints
│   ├── test_conversation.py  # basic prompt/response
│   ├── test_tools_identity.py # identity and context tools
│   ├── test_tools_nostr.py   # nostr event, messaging, relay tools
│   ├── test_tools_skills.py  # skill and trigger tools
│   ├── test_tools_system.py  # system, local, model, config tools
│   ├── test_tools_memory.py  # task and memory tools
│   ├── test_tools_cashu.py   # cashu wallet tools
│   ├── test_tools_blossom.py # blossom tools
│   ├── test_timeouts.py      # response time assertions
│   ├── test_errors.py        # API error handling paths
│   └── test_restart.py       # crash recovery
├── configs/
│   └── test_genesis.jsonc    # example test config (user fills in secrets)
├── results/                  # test run output (gitignored)
├── run_tests.py              # entry point
├── test.sh                   # existing bash test (kept as-is)
├── blossom_tool_validation_test.c  # existing C test (kept as-is)
└── blossom_tool_validation_test    # existing compiled test (kept as-is)

Component Specifications

1. agent_process.py — Process Manager

Manages the didactyl subprocess lifecycle.

class AgentProcess:
    def __init__(self, binary_path, config_path, api_port=8484,
                 api_bind="127.0.0.1", debug_level=5, log_file=None):
        """Configure but don't start yet."""
    
    def start(self, timeout=30) -> bool:
        """
        Spawn didactyl as subprocess:
          ./didactyl --config <config_path> --debug <level> 
                     --api-port <port> --api-bind <bind>
        
        Set DIDACTYL_LOG_FILE env var to log_file path.
        Wait for GET /api/status to return 200 (poll with timeout).
        Returns True if agent started successfully.
        """
    
    def stop(self, timeout=10) -> bool:
        """
        Send SIGTERM, wait for clean exit.
        If still alive after timeout, send SIGKILL.
        Returns True if stopped cleanly.
        """
    
    def restart(self, timeout=30) -> bool:
        """stop() then start()."""
    
    def is_alive(self) -> bool:
        """Check process.poll() and optionally /api/status."""
    
    def pid(self) -> int | None:
        """Return PID if running."""
    
    def return_code(self) -> int | None:
        """Return exit code if stopped."""

Key details:

Uses subprocess.Popen with stdout=PIPE, stderr=PIPE
Captures stdout/stderr for crash diagnostics
The health check polls GET /api/status every 500ms until success or timeout
Sets DIDACTYL_LOG_FILE to a test-run-specific path like tests/results/<timestamp>/agent_debug.log

2. didactyl_client.py — HTTP API Client

Thin wrapper around the Didactyl HTTP API using only urllib.

class DidactylClient:
    def __init__(self, base_url="https://127.0.0.1:8484", timeout=60,
                 verify_tls=False):
        """Configure base URL and default timeout."""
    
    def status(self) -> dict:
        """GET /api/status"""
    
    def prompt(self, message: str, max_turns: int = 4,
               model: str = None) -> dict:
        """POST /api/prompt/agent — full agent context conversation"""
    
    def prompt_raw(self, messages: list, max_turns: int = 4,
                   model: str = None) -> dict:
        """POST /api/prompt/run — raw messages, no auto-context"""
    
    def prompt_simple(self, system: str, user: str,
                      model: str = None) -> dict:
        """POST /api/prompt/run-simple — no tools"""
    
    def context_current(self) -> dict:
        """GET /api/context/current"""
    
    def context_parts(self) -> dict:
        """GET /api/context/parts"""
    
    def fire_webhook(self, d_tag: str, payload: dict = None) -> dict:
        """POST /api/trigger/<d_tag>"""

Key details:

All methods return parsed JSON dict
Raises TimeoutError if response exceeds timeout
Raises ConnectionError if agent is unreachable
Raises APIError(status_code, body) for non-2xx responses
Uses ssl._create_unverified_context() for local TLS (same as chat CLI)

3. log_watcher.py — Real-time Log Monitor

Background thread that tails the agent's debug log file.

class LogWatcher:
    def __init__(self, log_path: str):
        """Configure log file path."""
    
    def start(self):
        """
        Start background thread.
        Open file, seek to end, poll for new lines.
        Store all lines in memory with timestamps.
        """
    
    def stop(self):
        """Stop background thread."""
    
    def set_marker(self, name: str):
        """Record current line count as a named marker."""
    
    def get_lines_since(self, marker: str) -> list[str]:
        """Return all lines captured since the named marker."""
    
    def get_all_lines(self) -> list[str]:
        """Return all captured lines."""
    
    def search(self, pattern: str, since_marker: str = None) -> list[str]:
        """Regex search through captured lines."""
    
    def has_errors(self, since_marker: str = None) -> bool:
        """Check for [ERROR] lines since marker."""
    
    def has_warnings(self, since_marker: str = None) -> bool:
        """Check for [WARN] lines since marker."""
    
    def error_lines(self, since_marker: str = None) -> list[str]:
        """Return all ERROR lines since marker."""

Key details:

Uses threading.Thread(daemon=True) for background polling
Polls file every 100ms for new content
Handles file rotation (agent restart creates new file)
Thread-safe access to captured lines via threading.Lock

4. test_runner.py — Orchestrator

class TestResult:
    name: str
    suite: str
    status: str  # "pass", "fail", "error", "skip", "timeout"
    message: str
    duration_seconds: float
    agent_errors: list[str]  # ERROR lines from log during this test
    details: dict  # arbitrary test-specific data

class TestCase:
    name: str
    description: str
    requires_restart: bool = False
    
    def run(self, client: DidactylClient, log: LogWatcher) -> TestResult:
        """Execute the test and return result."""

class TestRunner:
    def __init__(self, agent: AgentProcess, client: DidactylClient,
                 log: LogWatcher):
        """Configure with harness components."""
    
    def discover_suites(self, suites_dir: str) -> list:
        """Import all test_*.py modules from suites directory."""
    
    def run_all(self, suites: list = None) -> list[TestResult]:
        """
        For each suite:
          1. If test requires restart, restart agent
          2. Set log marker for this test
          3. Run test with timeout wrapper
          4. Capture result + any agent errors from log
          5. If agent crashed, restart and record error
          6. Collect all results
        Return list of TestResult.
        """
    
    def run_suite(self, suite_name: str) -> list[TestResult]:
        """Run a single named suite."""

Key details:

Each test gets a fresh log marker so errors can be correlated
If client.prompt() raises TimeoutError, the test is marked "timeout" and the agent is restarted
If agent.is_alive() returns False mid-suite, the agent is restarted and remaining tests continue
Supports filtering by suite name or test name via CLI args

5. reporter.py — Results Output

class Reporter:
    def __init__(self, results: list[TestResult], output_dir: str):
        """Configure with results and output directory."""
    
    def print_summary(self):
        """Print pass/fail/error/skip/timeout counts to stdout."""
    
    def print_details(self):
        """Print each test result with details."""
    
    def write_json(self, path: str):
        """Write full results as JSON for programmatic consumption."""
    
    def write_text(self, path: str):
        """Write human-readable report."""

Output format example:

== Didactyl Test Results ==
Run: 2026-03-25T09:50:00Z
Agent: v0.0.26
Model: claude-haiku-4.5

Pass:    42
Fail:     3
Error:    1
Timeout:  2
Skip:     0
Total:   48

-- Failures --
[FAIL] test_tools_nostr::nostr_post_kind1
  Expected tool_calls to contain 'nostr_post', got: ['nostr_query']
  Agent errors during test: 0

[FAIL] test_conversation::multi_turn
  Agent returned empty final_response
  Agent errors during test: 2
    [ERROR] [llm.c:234] HTTP 429 rate limited
    [ERROR] [llm.c:240] LLM call failed after 3 retries

[TIMEOUT] test_tools_skills::skill_create
  No response within 60s
  Agent was restarted

Test Suite Specifications

test_health.py

Test	Prompt/Action	Assertions
`status_returns_200`	`GET /api/status`	HTTP 200, `success=true`
`status_has_fields`	`GET /api/status`	Has `name`, `version`, `pubkey`, `relay_count`
`context_current_returns_messages`	`GET /api/context/current`	Has `messages` array, `total_chars > 0`
`context_parts_has_system_prompt`	`GET /api/context/parts`	Has part named `system_prompt`

test_conversation.py

Test	Prompt	Assertions
`simple_greeting`	"Hello, what is your name?"	`final_response` is non-empty string
`agent_responds_about_itself`	"What are you? Describe yourself briefly."	`final_response` mentions agent/Didactyl/Nostr
`empty_message_handling`	"" (empty string)	Returns error or handles gracefully
`very_long_message`	10000 char string	Returns response or graceful error, no crash

test_tools_identity.py

Test	Prompt	Expected Tool	Assertions
`get_pubkey`	"What is your public key in hex?"	`nostr_pubkey` or `my_pubkey`	Tool called, result has hex pubkey
`get_npub`	"What is your npub?"	`nostr_npub` or `my_npub`	Tool called, result has npub1...
`agent_identity`	"Tell me about your identity"	`agent_identity`	Tool called, success
`agent_version`	"What version are you?"	`agent_version`	Tool called, result has version string
`admin_identity`	"Who is your administrator?"	`admin_identity`	Tool called, success

test_tools_nostr.py

Test	Prompt	Expected Tool	Assertions
`nostr_post_kind1`	"Post a test note saying 'Automated test post'"	`nostr_post`	Tool called with kind=1, success, event_id returned
`nostr_query_recent`	"Query the 3 most recent kind 1 notes from any author"	`nostr_query`	Tool called, returns events array
`nostr_my_events`	"List your recent events"	`nostr_my_events`	Tool called, success
`nostr_relay_status`	"What is the status of your relay connections?"	`nostr_relay_status`	Tool called, returns relay info
`nostr_dm_send`	"Send a test DM to yourself"	`nostr_dm_send`	Tool called, success
`nostr_encode_npub`	"Encode your pubkey as an npub"	`nostr_encode`	Tool called, returns npub
`nostr_profile_get`	"Look up your own Nostr profile"	`nostr_profile_get`	Tool called, returns profile

test_tools_skills.py

Test	Prompt	Expected Tool	Assertions
`skill_list`	"List your available skills"	`skill_list`	Tool called, returns skills array
`trigger_list`	"List your active triggers"	`trigger_list`	Tool called, success
`skill_create_and_remove`	"Create a test skill called 'test-harness-probe' with content 'Test skill' then remove it"	`skill_create`, `skill_remove`	Both tools called, success

test_tools_system.py

Test	Prompt	Expected Tool	Assertions
`tool_list`	"List all your available tools"	`tool_list`	Tool called, returns tools array
`model_get`	"What model are you currently using?"	`model_get`	Tool called, returns model info
`model_list`	"List available models"	`model_list`	Tool called, success
`local_http_fetch`	"Fetch https://httpbin.org/get"	`local_http_fetch`	Tool called, returns HTTP response
`config_store_recall`	"Store a test config with d_tag 'test_harness_probe' containing 'hello', then recall it"	`config_store`, `config_recall`	Both called, recalled value matches

test_tools_memory.py

Test	Prompt	Expected Tool	Assertions
`task_list`	"Show me your current task list"	`task_list` or `task_manage`	Tool called, success
`task_manage_add_remove`	"Add a task 'test harness probe task' then remove it"	`task_manage`	Tool called with add then remove
`memory_save_recall`	"Save 'test harness probe' to memory, then recall your memory"	`memory_save`, `memory_recall`	Both called, recalled contains probe text

test_tools_cashu.py

Test	Prompt	Expected Tool	Assertions
`wallet_balance`	"Check your cashu wallet balance"	`cashu_wallet_balance`	Tool called, success (even if empty)

Note: Most cashu tools require a configured mint and funded wallet. Phase 1 tests only the read-only balance check. Full cashu testing requires a test mint setup.

test_tools_blossom.py

Test	Prompt	Expected Tool	Assertions
`blossom_list`	"List your blossom blobs"	`blossom_list`	Tool called, success (even if empty)

Note: Full blossom testing requires a configured blossom server. Phase 1 tests only the list operation.

test_timeouts.py

Test	Action	Assertions
`response_within_timeout`	Send simple prompt, measure time	Response received within 60s
`status_responds_fast`	`GET /api/status`, measure time	Response within 2s
`context_responds_fast`	`GET /api/context/current`, measure time	Response within 5s

test_errors.py

Test	Action	Assertions
`invalid_json_body`	POST malformed JSON to `/api/prompt/agent`	Returns 400, `success=false`
`missing_message_field`	POST `{}` to `/api/prompt/agent`	Returns 400, `success=false`
`unknown_endpoint`	GET `/api/nonexistent`	Returns 404
`webhook_nonexistent_dtag`	POST to `/api/trigger/nonexistent-dtag`	Returns 404

test_restart.py

Test	Action	Assertions
`clean_restart`	Stop agent, start agent	Agent comes back, `/api/status` works
`status_after_restart`	Restart, then `GET /api/status`	Same pubkey, version as before restart
`conversation_after_restart`	Restart, then send prompt	Agent responds normally

Entry Point: run_tests.py

Usage:
  python tests/run_tests.py [options]

Options:
  --config PATH       Path to test genesis.jsonc (default: tests/configs/test_genesis.jsonc)
  --binary PATH       Path to didactyl binary (default: ./didactyl)
  --suite NAME        Run only this suite (can repeat)
  --test NAME         Run only this test (can repeat)
  --api-port PORT     API port (default: 8485)
  --timeout SECS      Default response timeout (default: 60)
  --debug-level N     Agent debug level 0-5 (default: 5)
  --output-dir PATH   Results output directory (default: tests/results/<timestamp>)
  --verbose           Print each test result as it runs
  --no-restart        Don't auto-restart agent on crash (fail remaining tests)

Test Genesis Config

tests/configs/test_genesis.jsonc — an example config the user fills in:

{
  // TEST CONFIGURATION — fill in your test identity secrets
  
  "key": {
    "nsec": "nsec1REPLACE_WITH_DISPOSABLE_TEST_NSEC"
  },
  
  "admin": {
    "pubkey": "npub1REPLACE_WITH_TEST_ADMIN_PUBKEY"
  },
  
  "dm_protocol": "nip04",
  
  "llm": {
    "provider": "openai",
    "api_key": "sk-REPLACE_WITH_API_KEY",
    "model": "claude-haiku-4.5",
    "base_url": "https://api.anthropic.com/v1",
    "max_tokens": 512,
    "temperature": 0.3
  },
  
  "api": {
    "enabled": true,
    "port": 8485,
    "bind_address": "127.0.0.1"
  },
  
  "startup_events": [
    {
      "kind": 0,
      "content_fields": {
        "name": "Didactyl Test Agent",
        "about": "Automated test instance"
      },
      "tags": []
    },
    {
      "kind": 10002,
      "content": "",
      "tags": [
        ["r", "wss://relay.damus.io"],
        ["r", "wss://relay.primal.net"]
      ]
    },
    {
      "kind": 31124,
      "content": "# Test Agent\n\nYou are a test agent. Respond to all requests. Use tools when asked.\n\n{{my_kind0_profile}}\n\nYour npub: {{my_npub}}",
      "tags": [
        ["d", "identity_and_rules"],
        ["app", "didactyl"],
        ["scope", "private"],
        ["trigger", "dm"],
        ["filter", "{\"from\":\"admin\"}"]
      ]
    }
  ]
}

Execution Flow

flowchart TD
    START[run_tests.py] --> PARSE[Parse CLI args]
    PARSE --> BUILD[Verify didactyl binary exists]
    BUILD --> CONFIG[Load test genesis config]
    CONFIG --> MKDIR[Create results output dir]
    MKDIR --> LOG_START[Start LogWatcher]
    LOG_START --> AGENT_START[Start AgentProcess]
    AGENT_START --> HEALTH[Wait for /api/status 200]
    
    HEALTH -->|timeout| FAIL_STARTUP[Report startup failure and exit]
    HEALTH -->|success| DISCOVER[Discover test suites]
    
    DISCOVER --> LOOP{Next test?}
    
    LOOP -->|yes| CHECK_ALIVE{Agent alive?}
    CHECK_ALIVE -->|no| RESTART_MID[Restart agent]
    RESTART_MID --> CHECK_ALIVE
    CHECK_ALIVE -->|yes| MARKER[Set log marker]
    MARKER --> RUN_TEST[Run test with timeout]
    
    RUN_TEST -->|pass| RECORD_PASS[Record PASS]
    RUN_TEST -->|fail| RECORD_FAIL[Record FAIL]
    RUN_TEST -->|timeout| RESTART_TIMEOUT[Restart agent]
    RESTART_TIMEOUT --> RECORD_TIMEOUT[Record TIMEOUT]
    RUN_TEST -->|error| RECORD_ERROR[Record ERROR]
    
    RECORD_PASS & RECORD_FAIL & RECORD_TIMEOUT & RECORD_ERROR --> COLLECT_LOGS[Collect agent errors since marker]
    COLLECT_LOGS --> LOOP
    
    LOOP -->|no more| STOP_AGENT[Stop agent]
    STOP_AGENT --> STOP_LOG[Stop LogWatcher]
    STOP_LOG --> REPORT[Generate report]
    REPORT --> EXIT[Exit with code 0 if all pass else 1]

Implementation Order

Harness infrastructure (agent_process, didactyl_client, log_watcher)
Test runner + reporter (orchestration layer)
test_health.py (validates the harness itself works)
test_conversation.py (validates basic agent interaction)
test_errors.py (validates error handling)
test_timeouts.py (validates timeout detection)
test_restart.py (validates process management)
Tool test suites (one at a time, identity → nostr → skills → system → memory → cashu → blossom)
Entry point + config (run_tests.py, test_genesis.jsonc)
End-to-end validation against a running agent

Future: Phase 2 — LLM-Driven Testing

The architecture supports this naturally. In Phase 2:

Add an LLMTestAgent class that wraps an LLM API call
The LLM receives: tool documentation, test objectives, previous results
It generates test prompts, evaluates responses, decides next actions
The TestCase.run() method delegates to the LLM agent instead of scripted logic
The LLM can also analyze agent logs for anomalies
Potentially: the LLM can generate code patches for bugs it finds

No architectural changes needed — just a new test suite type that uses LLM reasoning instead of scripted assertions.

22 KiB Raw Blame History

Didactyl Automated Test Harness

Overview

Architecture

Design Decisions

Directory Structure

Component Specifications

1. agent_process.py — Process Manager

2. didactyl_client.py — HTTP API Client

3. log_watcher.py — Real-time Log Monitor

4. test_runner.py — Orchestrator

5. reporter.py — Results Output

Test Suite Specifications

test_health.py

test_conversation.py

test_tools_identity.py

test_tools_nostr.py

test_tools_skills.py

test_tools_system.py

test_tools_memory.py

test_tools_cashu.py

test_tools_blossom.py

test_timeouts.py

test_errors.py

test_restart.py

Entry Point: run_tests.py

Test Genesis Config

Execution Flow

Implementation Order

Future: Phase 2 — LLM-Driven Testing

22 KiB

Raw Blame History