didactyl/plans/prompt_templates_coding.md

Prompt Template System — Coding Plan

Design doc: plans/prompt_templates.md

Overview

Replace the hardcoded context assembly in src/agent.c with a template-driven system. The template lives inside the soul event content (kind 31120), delimited by ---template---. If no template marker is found, fall back to the current hardcoded assembly for backward compatibility.

---

Phase 1: New Files — Template Parser & Builder

Step 1.1: Create src/prompt_template.h

Header with data structures and public API.

#ifndef DIDACTYL_PROMPT_TEMPLATE_H
#define DIDACTYL_PROMPT_TEMPLATE_H

#include "cjson/cJSON.h"
#include "config.h"

#define PROMPT_TEMPLATE_MAX_SECTIONS 32
#define PROMPT_TEMPLATE_MAX_NAME_LEN 64
#define PROMPT_TEMPLATE_MAX_ROLE_LEN 16
#define PROMPT_TEMPLATE_MARKER "---template---"

typedef struct {
    char name[PROMPT_TEMPLATE_MAX_NAME_LEN];
    char role[PROMPT_TEMPLATE_MAX_ROLE_LEN];   // system, user, assistant, expand
    char* content_template;                     // content with {{var}} placeholders, or NULL
    int limit;                                  // for expand sections, 0 = default
} prompt_template_section_t;

typedef struct {
    char* personality;                          // everything above ---template---
    prompt_template_section_t sections[PROMPT_TEMPLATE_MAX_SECTIONS];
    int section_count;
} prompt_template_t;

// Variable resolver callback: given a variable name, return a malloc'd string or NULL.
// The caller frees the returned string.
typedef char* (*prompt_var_resolver_fn)(const char* var_name, void* user_data);

// Parse soul content into a template. Returns 0 on success, -1 if no template found.
// On success, caller must call prompt_template_free() when done.
// On -1 (no template), out_template is zeroed — caller should use hardcoded fallback.
int prompt_template_parse(const char* soul_content, prompt_template_t* out_template);

// Build a cJSON messages array from a parsed template.
// resolver_fn is called for each {{variable}} encountered.
// dm_history_messages is a cJSON array of user/assistant messages for "expand" sections.
// Returns a new cJSON array (caller owns it), or NULL on error.
cJSON* prompt_template_build_messages(
    const prompt_template_t* tmpl,
    prompt_var_resolver_fn resolver_fn,
    void* resolver_user_data,
    cJSON* dm_history_messages
);

// Free internals of a parsed template (does not free the struct itself).
void prompt_template_free(prompt_template_t* tmpl);

// Get the section name for a message index (for logging/API).
// Returns the section name string or NULL if idx is out of range.
const char* prompt_template_section_name_at(const prompt_template_t* tmpl, int section_idx);

#endif

Step 1.2: Create src/prompt_template.c

Implementation file with three main components:

1.2a: Template Parser — prompt_template_parse()

Logic:

1. Search soul_content for the string "\n---template---\n" (with newlines on both sides, or at start/end of string).
2. If not found, return -1 (no template).
3. Split: everything before the marker → tmpl->personality (strdup'd).
4. Everything after the marker → parse as template sections.
5. Template section format is line-oriented YAML-like:

- section: admin_identity
  role: system
  limit: 0
  content: |
    This is your administrator! Admin pubkey: {{admin_pubkey}}

Parsing rules:

• Lines starting with - section: begin a new section.
• role: sets the role (default: system).
• limit: sets the limit integer (default: 0).
• content: | starts a multi-line content block. All subsequent lines indented by 4+ spaces (or until the next - section: line) are the content template.
• {{variable_name}} placeholders in content are left as-is during parsing; they are resolved at build time.

Edge cases:

• Trim leading/trailing whitespace from section names and roles.
• If content: is a single line (not |), treat the rest of the line as the content.
• Cap at PROMPT_TEMPLATE_MAX_SECTIONS.

1.2b: Variable Resolver — prompt_template_build_messages()

Logic:

1. Create a new cJSON array.
2. First, append the personality as a system message (role=system, content=personality).
3. For each section in order:
   • If role is "expand": insert the dm_history_messages array items here, limited by section.limit (take last N if limit > 0).
   • Otherwise: resolve {{var}} placeholders in content_template by calling resolver_fn(var_name, user_data). Build the resolved string. Append as a message with the configured role.
4. Return the array.

Variable resolution:

• Scan content_template for {{ ... }} pairs.
• Extract the variable name (trimmed).
• Call resolver_fn(name, user_data).
• If resolver returns NULL, substitute empty string.
• If resolver returns a string, substitute it and free the returned string.
• Build the final resolved content by concatenating literal segments and resolved values.

1.2c: Cleanup — prompt_template_free()

• Free tmpl->personality.
• For each section, free content_template.
• Zero the struct.

---

Phase 2: Variable Resolver in src/agent.c

Step 2.1: Create a resolver function

Add a static function in src/agent.c:

static char* agent_resolve_template_var(const char* var_name, void* user_data);

This function maps variable names to data sources:

Variable Name	Source	Implementation
admin_pubkey	g_cfg->admin.pubkey	strdup(g_cfg->admin.pubkey)
admin_kind0_json	nostr_handler_get_admin_kind0_context()	Already returns malloc'd string
admin_kind10002_json	nostr_handler_get_admin_kind10002_context()	Already returns malloc'd string
startup_events_json	Serialize startup events	Reuse logic from current append_startup_events_context()
adopted_skills_content	Build skills string	Reuse logic from current append_adopted_skills_context()
admin_notes_content	nostr_handler_get_admin_kind1_notes_context()	Already returns malloc'd string
agent_pubkey	g_cfg->keys.public_key_hex	strdup(g_cfg->keys.public_key_hex)

The user_data parameter is unused (NULL) since the resolver accesses globals.

Step 2.2: Extract helper functions from existing code

Refactor the following existing static functions to return malloc'd strings instead of appending directly to a cJSON array:

• Extract startup events serialization from append_startup_events_context() (lines 383-434) into a new static char* build_startup_events_string(void).
• Extract adopted skills content from append_adopted_skills_context() (lines ~744-940) into a new static char* build_adopted_skills_string(void).

These helpers are called by the resolver function.

---

Phase 3: Wire Template into Agent

Step 3.1: Parse template at init time

In agent_init() (or when g_system_context is set), after the soul content is available:

static prompt_template_t g_prompt_template;
static int g_has_template = 0;

After g_system_context is assigned, call:

g_has_template = (prompt_template_parse(g_system_context, &g_prompt_template) == 0);

If g_has_template is true, g_prompt_template.personality replaces g_system_context for the system prompt message.

Step 3.2: Modify agent_build_admin_messages_json()

Current location: src/agent.c:1092

Current signature (unchanged):

int agent_build_admin_messages_json(const char* current_user_message, char** out_messages_json);

New logic:

if (g_has_template) {
    // Build DM history as a cJSON array
    cJSON* dm_history = build_dm_history_array(current_user_message);
    
    // Build messages from template
    cJSON* messages = prompt_template_build_messages(
        &g_prompt_template,
        agent_resolve_template_var,
        NULL,
        dm_history
    );
    
    cJSON_Delete(dm_history);
    
    if (!messages) {
        return -1;
    }
    
    char* json = cJSON_PrintUnformatted(messages);
    cJSON_Delete(messages);
    *out_messages_json = json;
    return json ? 0 : -1;
} else {
    // Existing hardcoded assembly (current code, unchanged)
    ...
}

Step 3.3: Extract DM history builder

Extract the DM history logic from append_recent_admin_dm_history() into a function that returns a cJSON array of user/assistant messages:

static cJSON* build_dm_history_array(const char* current_user_message);

This is used by the template builder for role: expand sections.

---

Phase 4: Context Log Formatting

Step 4.1: Update format_context_payload_for_log()

Current location: src/agent.c:245

When g_has_template is true, use section names from the template instead of detect_context_section():

• Message 0 is always system_prompt (the personality).
• Messages 1..N map to template sections by index.
• For expand sections, multiple messages share the same section name.

Change the log header from:

Message 01 | role=system | section=system_prompt

To:

Section: system_prompt | role=system

Step 4.2: Update classify_part_name() in src/http_api.c

Current location: src/http_api.c:113

Add a new exported function from src/agent.h:

const char* agent_get_section_name_for_message(int message_index);

This returns the template section name if a template is active, or falls back to the existing content-prefix detection.

In classify_part_name(), call this function first. If it returns non-NULL, use it. Otherwise fall back to the existing strncmp chain.

---

Phase 5: Build System Updates

Step 5.1: Update Makefile

Add $(SRC_DIR)/prompt_template.c to the SRCS list (after trigger_manager.c, before http_api.c).

Step 5.2: Update Dockerfile.alpine-musl

Add src/prompt_template.c to the gcc command line (after src/trigger_manager.c, before src/http_api.c).

---

Phase 6: Default Template in Soul

Step 6.1: Update config.json.example

The startup events should include a soul event (kind 31120) with a ---template--- section that matches the current hardcoded behavior:

# Didactyl Agent

You are Didactyl, a sovereign AI agent living on Nostr.
...existing soul content...

---template---

- section: admin_identity
  role: system
  content: |
    This is your administrator! Admin pubkey (hex): {{admin_pubkey}}

- section: admin_profile
  role: system
  content: |
    Administrator profile (JSON): {{admin_kind0_json}}

- section: admin_relay_list
  role: system
  content: |
    Administrator relay list (JSON): {{admin_kind10002_json}}

- section: startup_events
  role: system
  content: |
    Startup events memory (kinds/content/tags): {{startup_events_json}}

- section: adopted_skills
  role: system
  content: |
    {{adopted_skills_content}}

- section: dm_history
  role: expand
  limit: 12

- section: admin_notes
  role: system
  limit: 10
  content: |
    Administrator recent public notes: {{admin_notes_content}}

---

Phase 7: Testing

Step 7.1: Backward compatibility test

1. Build with make.
2. Run with existing config (no ---template--- in soul).
3. Send a DM and verify context.log output matches pre-change format.
4. Verify /api/context/current returns expected structure.

Step 7.2: Template test

1. Edit the soul event content to include ---template--- section.
2. Restart agent.
3. Send a DM and verify context.log shows section-named headers.
4. Verify /api/context/current returns section names from template.
5. Verify the LLM receives the correct messages in the correct order.

Step 7.3: Section reordering test

1. Move admin_notes section above adopted_skills in the template.
2. Restart and verify the order changes in context.log.

Step 7.4: Limit test

1. Set dm_history limit to 4 (instead of 12).
2. Verify only 4 DM history turns appear in context.

---

File Change Summary

File	Action	Description
src/prompt_template.h	NEW	Data structures and API
src/prompt_template.c	NEW	Parser, variable resolver, context builder
src/agent.c	MODIFY	Add template globals, resolver function, wire into agent_build_admin_messages_json(), update log formatter
src/agent.h	MODIFY	Add agent_get_section_name_for_message() export
src/http_api.c	MODIFY	Update classify_part_name() to use template section names
Makefile	MODIFY	Add prompt_template.c to SRCS
Dockerfile.alpine-musl	MODIFY	Add prompt_template.c to gcc command
config.json.example	MODIFY	Update soul event to include template section

---

Implementation Order

1. src/prompt_template.h — data structures and API declarations
2. src/prompt_template.c — parser, builder, free
3. Makefile + Dockerfile.alpine-musl — add new source file
4. Build and verify compilation
5. src/agent.c — extract helper functions (build_startup_events_string, build_adopted_skills_string, build_dm_history_array)
6. src/agent.c — add resolver function and template globals
7. src/agent.c — wire template into agent_build_admin_messages_json()
8. src/agent.c — update format_context_payload_for_log()
9. src/agent.h + src/http_api.c — section name API for classify_part_name
10. Build and test backward compatibility (no template in soul)
11. config.json.example — add template to soul event
12. Test with template soul
13. Test section reordering and limit changes