Files
sovereign_browser/plans/agent-tools.md
Laan Tungir 372339aa35 v0.0.12 - MCP server: 100 tools, Streamable HTTP transport, favicons, refresh button, webview sizing fix
- MCP protocol compliance: ping, resources/list, prompts/list, logging/setLevel, 202 notifications
- Screenshot tool: WebKit snapshot to PNG to base64 MCP image content
- Session management: Mcp-Session-Id header, SSE response format, GET/DELETE endpoints
- Phase 3: 70 new tools across 8 batches (extended interaction, get/state, find elements, wait/batch, cookies/storage, mouse/clipboard/settings, frames/dialogs/debug, complex)
- Favicon support: enabled WebKitFaviconDatabase, notify::favicon signal handler
- Tab UI: favicons in tab labels, evenly distributed tabs, immediate title from URL host
- Refresh button: left-click reload, right-click hard reload menu (bypass cache, clear site data)
- Webview sizing fix: gtk_widget_set_vexpand/hexpand to prevent 1px height blank page
- Updated Roo Code MCP config (100 tools in alwaysAllow)
- Updated plans: mcp-server.md, agent-tools.md, phase3-tools.md
2026-07-12 09:20:14 -04:00

30 KiB

Agent Tools Implementation Plan

Overview

Make sovereign_browser agentically capable by embedding a WebSocket server in the browser process. External AI agents (LLM tools, scripts, other processes) connect to ws://localhost:PORT and send JSON tool commands to navigate, inspect, and interact with web pages.

The design is inspired by agent-browser, which uses a snapshot + ref workflow: take a snapshot (accessibility tree with element refs), then interact by ref (click @e2, fill @e3). This is the optimal pattern for LLM-driven browser automation.

Architecture

flowchart TB
    subgraph AgentProcess
        Agent[AI Agent / Script / CLI]
    end

    subgraph BrowserProcess
        WSServer[WebSocket Server — SoupServer]
        ToolDispatch[Tool Dispatcher]
        Snapshot[Snapshot Engine]
        Tools[Tool Implementations]
        TabMgr[Tab Manager]
    end

    subgraph WebKitGTK
        Webview[WebKitWebView — active tab]
        JSEval[webkit_web_view_evaluate_javascript]
    end

    Agent -->|ws://localhost:PORT JSON| WSServer
    WSServer --> ToolDispatch
    ToolDispatch --> Tools
    Tools --> Snapshot
    Tools --> TabMgr
    Tools -->|JS eval| JSEval
    JSEval --> Webview
    Snapshot -->|inject + extract| JSEval
    TabMgr --> Webview

Key design decisions

  1. Agent tools must use the same code path as the GUI — Agent tools should call the same C functions the GUI calls, not a parallel implementation. This ensures that any error a user would see, the agent also sees. Debugging WebKit internals is not our concern; our C code is. If the agent and GUI take different paths, bugs can hide from the agent. The login tools are the one exception (backend-only, bypassing the GTK dialog) — all browser tools go through the same WebKitWebView and tab_manager APIs.

  2. WebSocket server via libsoupSoupServer with soup_server_add_websocket_handler() provides a production-ready WebSocket server. libsoup is already linked (WebKitGTK dependency). No new dependencies, no hand-rolling the WebSocket protocol.

  3. JSON protocol via cJSON — cJSON is already vendored in nostr_core_lib/cjson/. All tool requests and responses are JSON.

  4. Snapshot + ref pattern — The core workflow:

    • snapshot injects a JS script that walks the accessibility tree, assigns sequential refs (e1, e2, ...) to interactive elements, and returns a text representation of the tree.
    • click @e2 looks up the ref in a per-tab ref map, finds the DOM element, and clicks it via element.click().
    • Refs are per-tab and persist until the next snapshot call.
  5. Per-tab ref storage — Each tab_info_t gets a ref map (hash table from ref string → element selector or DOM node reference). Since JS execution is per-webview, refs are scoped to the tab's webview.

  6. Async JS executionwebkit_web_view_evaluate_javascript() is async. For tools that need the return value (snapshot, get_text, eval), we use a callback-based approach: the tool injects JS that stores the result in a known global variable, then reads it back. Alternatively, we can use a Promise-based approach with a polling mechanism.

  7. Simple HTTP fallback — The WebSocket server also handles plain HTTP GET requests to http://localhost:PORT/ returning a status JSON (server info, connected clients, tab count). This makes it easy to discover the server with curl without a WebSocket client.

WebSocket protocol

Request format

{
  "id": 1,
  "tool": "snapshot",
  "params": {
    "interactive": true,
    "compact": true
  }
}

Response format

{
  "id": 1,
  "success": true,
  "data": {
    "snapshot": "- heading \"Example Domain\" [ref=e1] [level=1]\n- link \"More information...\" [ref=e2]",
    "refs": {
      "e1": {"role": "heading", "name": "Example Domain", "level": 1},
      "e2": {"role": "link", "name": "More information...", "href": "https://example.com"}
    }
  }
}

Error response

{
  "id": 1,
  "success": false,
  "error": {
    "code": "ELEMENT_NOT_FOUND",
    "message": "No element with ref @e99"
  }
}

Push events (server → client)

{
  "type": "event",
  "event": "load",
  "data": {"url": "https://example.com", "title": "Example Domain", "tab": 0}
}
{
  "type": "event",
  "event": "tab_changed",
  "data": {"tab": 1, "url": "https://example.com", "title": "Example"}
}

Tool catalog — full comparison with agent-browser

The table below lists every agent-browser CLI command and whether we plan to implement it. "Phase 1" tools are in the initial implementation; "Phase 2" tools are deferred but designed for; "N/A" tools don't apply to our architecture (CDP-specific, cloud providers, etc.).

Core commands

agent-browser command Our tool Phase Notes
open <url> open 1 Navigate active tab
click <sel> click 1 By ref or CSS selector
dblclick <sel> dblclick 2 Double-click
focus <sel> focus 1 Focus element
type <sel> <text> type 1 Type into element
fill <sel> <text> fill 1 Clear and fill
press <key> press 1 Press keyboard key
keyboard type <text> keyboard_type 2 Type with real keystrokes
keyboard inserttext <text> insert_text 2 Insert text without key events
keydown <key> keydown 2 Hold key down
keyup <key> keyup 2 Release key
hover <sel> hover 1 Hover element
select <sel> <val> select 2 Select dropdown option
check <sel> check 2 Check checkbox
uncheck <sel> uncheck 2 Uncheck checkbox
scroll <dir> [px] scroll 1 Scroll page
scrollintoview <sel> scroll_into_view 2 Scroll element into view
drag <src> <tgt> drag 2 Drag and drop
upload <sel> <files> upload 2 Upload files
screenshot [path] screenshot 1 Take screenshot
screenshot --annotate screenshot_annotated 2 Annotated with element labels
pdf <path> pdf 2 Save as PDF
snapshot snapshot 1 Accessibility tree with refs
eval <js> eval 1 Run JavaScript
connect <port> N/A CDP-specific, we are the server
stream enable stream_enable 2 WebSocket viewport streaming
stream status stream_status 2 Streaming state
stream disable stream_disable 2 Stop streaming
close close 1 Close active tab
close --all close_all 2 Close all tabs

Get info commands

agent-browser command Our tool Phase Notes
get text <sel> get_text 1 Get text content
get html <sel> get_html 1 Get innerHTML
get value <sel> get_value 2 Get input value
get attr <sel> <attr> get_attr 1 Get attribute
get title get_title 1 Get page title
get url get_url 1 Get current URL
get cdp-url N/A CDP-specific
get count <sel> get_count 2 Count matching elements
get box <sel> get_box 2 Get bounding box
get styles <sel> get_styles 2 Get computed styles

Check state commands

agent-browser command Our tool Phase Notes
is visible <sel> is_visible 2 Check if visible
is enabled <sel> is_enabled 2 Check if enabled
is checked <sel> is_checked 2 Check if checked

Find elements (semantic locators)

agent-browser command Our tool Phase Notes
find role <role> ... find_role 2 By ARIA role
find text <text> ... find_text 2 By text content
find label <label> ... find_label 2 By label
find placeholder <ph> ... find_placeholder 2 By placeholder
find alt <text> ... find_alt 2 By alt text
find title <text> ... find_title 2 By title attr
find testid <id> ... find_testid 2 By data-testid
find first <sel> ... find_first 2 First match
find last <sel> ... find_last 2 Last match
find nth <n> <sel> ... find_nth 2 Nth match

Wait commands

agent-browser command Our tool Phase Notes
wait <selector> wait_for 1 Wait for element visible
wait <ms> wait 1 Wait for time
wait --text "..." wait_for_text 2 Wait for text to appear
wait --url "..." wait_for_url 2 Wait for URL pattern
wait --load networkidle wait_for_load 2 Wait for load state
wait --fn "..." wait_for_fn 2 Wait for JS condition

Batch execution

agent-browser command Our tool Phase Notes
batch --json batch 2 Execute multiple commands

Clipboard

agent-browser command Our tool Phase Notes
clipboard read clipboard_read 2 Read clipboard
clipboard write "..." clipboard_write 2 Write clipboard
clipboard copy clipboard_copy 2 Copy selection
clipboard paste clipboard_paste 2 Paste from clipboard

Mouse control

agent-browser command Our tool Phase Notes
mouse move <x> <y> mouse_move 2 Move mouse
mouse down [button] mouse_down 2 Press button
mouse up [button] mouse_up 2 Release button
mouse wheel <dy> [dx] mouse_wheel 2 Scroll wheel

Browser settings

agent-browser command Our tool Phase Notes
set viewport <w> <h> set_viewport 2 Set viewport size
set device <name> set_device N/A Device emulation is CDP-specific
set geo <lat> <lng> set_geo N/A Geolocation emulation
set offline [on|off] set_offline 2 Toggle offline
set headers <json> set_headers 2 Extra HTTP headers
set credentials <u> <p> set_credentials 2 HTTP basic auth
set media [dark|light] set_media 2 Emulate color scheme

Cookies and storage

agent-browser command Our tool Phase Notes
cookies cookies_get 2 Get all cookies
cookies set <name> <val> cookies_set 2 Set cookie
cookies clear cookies_clear 2 Clear cookies
storage local storage_local_get 2 Get localStorage
storage local <key> storage_local_get_key 2 Get specific key
storage local set <k> <v> storage_local_set 2 Set value
storage local clear storage_local_clear 2 Clear all
storage session storage_session_* 2 Same for sessionStorage

Network

agent-browser command Our tool Phase Notes
network route <url> network_route N/A Request interception is CDP-specific
network unroute network_unroute N/A CDP-specific
network requests network_requests N/A CDP-specific
network request <id> network_request_detail N/A CDP-specific
network har start har_start N/A CDP-specific
network har stop har_stop N/A CDP-specific

Tabs and windows

agent-browser command Our tool Phase Notes
tab tab_list 1 List tabs
tab new [url] tab_new 1 New tab
tab <n> tab_switch 1 Switch to tab n
tab close [n] tab_close 1 Close tab
window new window_new N/A Multi-window is future roadmap

Frames

agent-browser command Our tool Phase Notes
frame <sel> frame_switch 2 Switch to iframe
frame main frame_main 2 Back to main frame

Dialogs

agent-browser command Our tool Phase Notes
dialog accept [text] dialog_accept 2 Accept dialog
dialog dismiss dialog_dismiss 2 Dismiss dialog
dialog status dialog_status 2 Check if dialog open

Diff

agent-browser command Our tool Phase Notes
diff snapshot diff_snapshot N/A Future, after snapshot is stable
diff screenshot diff_screenshot N/A Future
diff url diff_url N/A Future

Debug

agent-browser command Our tool Phase Notes
trace start/stop N/A Chrome trace, not available in WebKitGTK
profiler start/stop N/A Chrome DevTools profiler
console console 2 View console messages
errors errors 2 View page errors
highlight <sel> highlight 2 Highlight element
inspect N/A Opens Chrome DevTools
state save/load/list state_save/load/list 2 Save/restore auth state

Navigation

agent-browser command Our tool Phase Notes
back back 1 Go back
forward forward 1 Go forward
reload reload 1 Reload page

Setup

agent-browser command Our tool Phase Notes
install N/A We bundle WebKitGTK, no install needed
upgrade N/A Package manager handles updates

Authentication

agent-browser command Our tool Phase Notes
auth save N/A We use Nostr identity, not credential vault
auth login N/A Nostr login is our native flow

Sessions

agent-browser command Our tool Phase Notes
--session <name> N/A We have one browser process, tabs instead
--session-name <name> N/A Session persistence is our session.c
--profile <path> N/A WebKitGTK handles profile data
--state <path> state_save/load 2 Could implement via WebKitGTK data manager

Login tools (sovereign_browser-specific, not in agent-browser)

These tools are unique to sovereign_browser. They allow an agent to handle the Nostr login flow programmatically — essential because the browser shows a modal login dialog on startup that blocks all other tools until the user (or agent) signs in.

The agent server starts before the login dialog, so the agent can authenticate without any human interaction. The login tools call the same nostr_core_lib functions that the GTK login dialog uses, bypassing the dialog entirely.

Tool Params Description
login_status {} Check if logged in; return method, pubkey, readonly
login {"method": "local", "nsec": "nsec1..."} Sign in with local key
login {"method": "seed", "mnemonic": "word1 word2 ...", "passphrase": "..."} Sign in with BIP-39 seed phrase
login {"method": "readonly", "npub": "npub1..."} Sign in read-only with npub
login {"method": "nip46", "bunker_url": "bunker://..."} Sign in with NIP-46 remote signer
login {"method": "nsigner", "transport": "qrexec", "device": "nostr_signer", "service": "qubes.NsignerRpc", "index": 0} Sign in with n_signer hardware
logout {} Log out and clear signer
switch_identity {"method": "local", "nsec": "nsec1..."} Switch to a new identity (same params as login)

Login response:

{
  "id": 1,
  "success": true,
  "data": {
    "method": "local",
    "pubkey": "a1b2c3...",
    "npub": "npub1...",
    "readonly": false
  }
}

Architecture change: The agent server must start before the login dialog. In main.c, the startup sequence becomes:

  1. gtk_init() + load settings/history
  2. Create the main window (but don't show it yet)
  3. Start the agent server (so agents can connect)
  4. If agent_server_enabled and an agent connects within a timeout, wait for the agent to call login
  5. Otherwise (no agent, or timeout), show the GTK login dialog as before
  6. After login (via agent or dialog), create tabs and show the window

This means the agent server has two states:

  • Pre-login: Only login_status, login, logout tools are available
  • Post-login: All browser tools become available

Summary

Category agent-browser commands Our Phase 1 Our Phase 2 Our Phase 3 N/A
Login (unique) 0 (8 new) 8 0 0 0
Core 29 10 0 14 5
Get info 9 4 0 4 1
Check state 3 0 0 3 0
Find elements 10 0 0 10 0
Wait 6 2 0 4 0
Batch 1 0 0 1 0
Clipboard 4 0 0 4 0
Mouse 4 0 0 4 0
Settings 7 0 0 4 3
Cookies/storage 8 0 0 11 0
Network 6 0 0 0 6
Tabs/windows 5 4 0 0 1
Frames 2 0 0 2 0
Dialogs 3 0 0 3 0
Diff 3 0 0 0 3
Debug 8 0 0 4 4
Navigation 3 3 0 0 0
Setup 2 0 0 0 2
Auth 2 0 0 0 2
Sessions 4 0 0 2 2
Total 119 + 8 30 0 70 29

Phase 1 (30 tools): Login tools (8) + the minimal set for agent-driven browser automation (22) — login, navigate, snapshot, inspect, interact, and manage tabs, plus screenshot. This is enough for an LLM to log in, browse, read, fill forms, click buttons, and multi-tab. The agent server starts before login so the agent can authenticate without human interaction.

Phase 2 (0 tools): Reserved for refinements to Phase 1 tools based on real agent usage — better snapshot formatting, error handling, edge cases.

Phase 3 (70 tools): Everything else that has a WebKitGTK equivalent. The original plan counted 67, but implementation added 3 extra storage_session_* tools (the plan listed storage as 8 local-only tools; we implemented 11 by adding storage_session_get, storage_session_get_key, storage_session_set, and storage_session_clear). Deferred until Phase 1 is stable and tested with real agents.

N/A (29 tools): CDP-specific features (network interception, Chrome DevTools profiler, trace), cloud provider integrations, and features that don't map to our architecture (device emulation, multi-window). Some could be revisited if WebKitGTK adds equivalent APIs.

Snapshot engine

The snapshot is the heart of the agent system. It injects a JavaScript script into the active tab's webview that:

  1. Walks the DOM accessibility tree (using TreeWalker or recursive querySelectorAll)
  2. For each interactive element (links, buttons, inputs, selects, textareas, elements with role/aria-label), assigns a sequential ref (e1, e2, ...)
  3. Stores a mapping from ref → unique CSS selector (for later re-querying)
  4. Returns a text representation of the tree:
- heading "Example Domain" [ref=e1] [level=1]
- paragraph "This domain is for use in..."
- link "More information..." [ref=e2]
- textbox "Search" [ref=e3]
- button "Submit" [ref=e4]

Ref storage

The ref map is stored as a JSON string in a hidden window.__agentRefs global on the page. When a tool like click @e2 is called:

  1. The tool injects JS: var el = document.querySelector(window.__agentRefs['e2']); el.click();
  2. If the element is gone (page navigated), the tool returns an error suggesting a re-snapshot.

Interactive-only mode

When interactive: true, only elements that are focusable or have an ARIA role are included. This dramatically reduces the snapshot size for LLM context windows.

New files

src/agent_server.h / src/agent_server.c

/*
 * WebSocket server for agent tool commands.
 * Uses libsoup SoupServer with a websocket handler at /agent.
 */

void agent_server_start(int port);
void agent_server_stop(void);
int  agent_server_get_port(void);
gboolean agent_server_is_running(void);

/* Broadcast a push event to all connected clients. */
void agent_server_emit_event(const char *event_name, cJSON *data);

src/agent_tools.h / src/agent_tools.c

/*
 * Tool dispatch: receives a JSON request, executes the tool,
 * returns a JSON response.
 */

/* Process a tool request JSON and return response JSON.
 * The caller frees the returned cJSON*. */
cJSON *agent_tools_dispatch(cJSON *request);

src/agent_snapshot.h / src/agent_snapshot.c

/*
 * Accessibility tree snapshot via JS injection.
 * Assigns refs to interactive elements and returns a text tree.
 */

/* The JS script injected for snapshot. Exposed for testing. */
extern const char *AGENT_SNAPSHOT_JS;

/* Parse a snapshot result JSON from the JS execution. */
cJSON *agent_snapshot_parse(const char *js_result);

New files (login-specific)

src/agent_login.h / src/agent_login.c

/*
 * Agent-driven Nostr login — calls nostr_core_lib directly,
 * bypassing the GTK login dialog. Used by the agent server
 * so an AI agent can authenticate without human interaction.
 */

/* Check current login state. Returns JSON response. */
cJSON *agent_login_status(void);

/* Perform login with the given method and params.
 * params is a cJSON object with "method" and method-specific fields.
 * Returns JSON response with pubkey/npub on success. */
cJSON *agent_login(cJSON *params);

/* Log out — clear signer, reset state. Returns JSON response. */
cJSON *agent_logout(void);

/* Switch identity — same as login but frees the old signer first. */
cJSON *agent_switch_identity(cJSON *params);

/* Returns TRUE if logged in (signer or readonly loaded). */
gboolean agent_is_logged_in(void);

Modified files

src/settings.h / src/settings.c

Add settings:

  • agent_server_enabled (bool, default: true)
  • agent_server_port (int, default: 17777)
  • agent_server_allowed_origins (string, default: "*" for localhost)
  • agent_login_timeout_ms (int, default: 30000) — how long to wait for an agent to log in before falling back to the GTK dialog

src/tab_manager.h / src/tab_manager.c

  • Add GHashTable *ref_map to tab_info_t for per-tab element refs
  • Emit events to agent_server on load, title change, tab switch

src/main.c

Major change to startup sequence:

  • Call agent_server_start() before do_login() (not after)
  • After starting the server, wait for either:
    • An agent to call login within agent_login_timeout_ms, OR
    • The timeout to expire, then show the GTK login dialog
  • After login (via agent or dialog), proceed to create tabs and show window
  • Call agent_server_stop() in on_window_destroy()

Makefile

Add src/agent_server.c src/agent_tools.c src/agent_snapshot.c src/agent_login.c to SRC. Add libsoup-3.0 to pkg-config if not already included.

Implementation order

Phase 1 — Login + core browser automation

  1. settings — add agent server settings (port, enabled, allowed origins)
  2. agent_server — WebSocket server using libsoup SoupServer, JSON protocol
  3. agent_login — login tools (login_status, login, logout, switch_identity) that call nostr_core_lib directly, bypassing the GTK dialog
  4. main.c integration (pre-login) — start agent server before login dialog, support agent-driven login with fallback to GTK dialog
  5. agent_snapshot — the JS injection script + ref assignment
  6. agent_tools — tool dispatch + core tools (open, snapshot, click, fill, type, eval, get_text, get_html, get_attr, get_url, get_title)
  7. main.c integration (post-login) — wire browser tools to tab_manager, emit events (load, tab_changed)
  8. Navigation tools — back, forward, reload, scroll, press, wait, wait_for
  9. Tab tools — tab_list, tab_new, tab_switch, tab_close
  10. Interaction tools — hover, focus, close
  11. Screenshot tool — WebKitGTK snapshot API
  12. UI status indicator — show server port in toolbar
  13. Makefile + docs — build and documentation

Phase 2 — Refinements

  1. Polish Phase 1 tools based on real agent usage
  2. Better snapshot formatting and error messages
  3. Edge case handling (page navigation during interaction, stale refs)

Phase 3 — Extended tool catalog (COMPLETE)

  1. All remaining tools from the comparison table (70 tools — 67 planned
    • 3 extra storage_session_* tools)
  2. Find elements, check state, clipboard, mouse control, cookies/storage, frames, dialogs, console/errors, batch, state save/load, etc.
  3. Implemented in 8 batches (see plans/phase3-tools.md):
    • Batch 1: Extended interaction (11 tools)
    • Batch 2: Get info + check state (7 tools)
    • Batch 3: Find elements (10 tools)
    • Batch 4: Wait + batch (5 tools)
    • Batch 5: Cookies + storage (11 tools)
    • Batch 6: Mouse + clipboard + settings (13 tools)
    • Batch 7: Frames + dialogs + debug (10 tools)
    • Batch 8: Complex tools (3 tools)

Testing the tools

The agent server starts before login, so the first thing an agent does is authenticate. Once logged in, all browser tools become available.

Login flow

# Using websocat (install: cargo install websocat)

# Check login status (before logging in)
echo '{"id":1,"tool":"login_status","params":{}}' | websocat ws://localhost:17777/agent

# Login with local key (nsec)
echo '{"id":2,"tool":"login","params":{"method":"local","nsec":"nsec1..."}}' | websocat ws://localhost:17777/agent

# Login read-only with npub
echo '{"id":3,"tool":"login","params":{"method":"readonly","npub":"npub1..."}}' | websocat ws://localhost:17777/agent

# Login with seed phrase
echo '{"id":4,"tool":"login","params":{"method":"seed","mnemonic":"abandon abandon abandon ... abandon about"}}' | websocat ws://localhost:17777/agent

# Login with NIP-46 remote signer
echo '{"id":5,"tool":"login","params":{"method":"nip46","bunker_url":"bunker://..."}}' | websocat ws://localhost:17777/agent

# Login with n_signer via Other Qube (qrexec)
echo '{"id":6,"tool":"login","params":{"method":"nsigner","transport":"qrexec","device":"nostr_signer","service":"qubes.NsignerRpc","index":0}}' | websocat ws://localhost:17777/agent

Browser automation flow (after login)

# Open a page
echo '{"id":10,"tool":"open","params":{"url":"https://example.com"}}' | websocat ws://localhost:17777/agent

# Snapshot — get accessibility tree with refs
echo '{"id":11,"tool":"snapshot","params":{"interactive":true}}' | websocat ws://localhost:17777/agent

# Click by ref
echo '{"id":12,"tool":"click","params":{"ref":"@e2"}}' | websocat ws://localhost:17777/agent

# Fill an input
echo '{"id":13,"tool":"fill","params":{"ref":"@e3","value":"test@example.com"}}' | websocat ws://localhost:17777/agent

# Get text
echo '{"id":14,"tool":"get_text","params":{"ref":"@e1"}}' | websocat ws://localhost:17777/agent

# List tabs
echo '{"id":15,"tool":"tab_list","params":{}}' | websocat ws://localhost:17777/agent

# Open new tab
echo '{"id":16,"tool":"tab_new","params":{"url":"https://example.org"}}' | websocat ws://localhost:17777/agent

Python example (persistent connection)

import websocket, json

ws = websocket.create_connection("ws://localhost:17777/agent")

# Login first
ws.send(json.dumps({"id": 1, "tool": "login", "params": {
    "method": "readonly", "npub": "npub1..."
}}))
print(json.loads(ws.recv()))

# Then browse
ws.send(json.dumps({"id": 2, "tool": "open", "params": {
    "url": "https://example.com"
}}))
print(json.loads(ws.recv()))

ws.send(json.dumps({"id": 3, "tool": "snapshot", "params": {
    "interactive": True
}}))
print(json.loads(ws.recv()))

Future: Nostr relay integration

Since the WebSocket server is already in the browser process, a future enhancement could expose Nostr relay connections through the same server. An agent could subscribe to Nostr events and react to them — e.g., a remote agent controlling the browser via Nostr DMs. This is a natural extension of the "sovereign identity" thesis: your Nostr identity controls your browser, not just your signing.

Implementation Status

All phases complete. 100 tools total.

Phase Tools Status
Phase 1 — Login + core 30 Complete
Phase 2 — Refinements 0 N/A (folded into Phase 1/3)
Phase 3 — Extended catalog 70 Complete (8 batches)
Total 100

Phase 3 batches (all complete)

Batch Category Tools Count
1 Extended interaction dblclick, select, check, uncheck, scroll_into_view, keyboard_type, insert_text, keydown, keyup, drag, close_all 11
2 Get info + check state get_value, get_count, get_box, get_styles, is_visible, is_enabled, is_checked 7
3 Find elements find_role, find_text, find_label, find_placeholder, find_alt, find_title, find_testid, find_first, find_last, find_nth 10
4 Wait + batch wait_for_text, wait_for_url, wait_for_load, wait_for_fn, batch 5
5 Cookies + storage cookies_get, cookies_set, cookies_clear, storage_local_get, storage_local_get_key, storage_local_set, storage_local_clear, storage_session_get, storage_session_get_key, storage_session_set, storage_session_clear 11
6 Mouse + clipboard + settings mouse_move, mouse_down, mouse_up, mouse_wheel, clipboard_read, clipboard_write, clipboard_copy, clipboard_paste, set_viewport, set_offline, set_headers, set_credentials, set_media 13
7 Frames + dialogs + debug frame_switch, frame_main, dialog_accept, dialog_dismiss, dialog_status, console, errors, highlight, state_save, state_load 10
8 Complex tools upload, pdf, screenshot_annotated 3
Total 70

Verification

  • make clean && make — builds cleanly (only pre-existing warnings in login_dialog.c and agent_mcp.c).
  • tools/list returns exactly 100 tools.
  • initialize / ping / login_status work without login.
  • Browser tools (dblclick, etc.) correctly return NOT_LOGGED_IN before login.
  • batch dispatches without login; sub-commands enforce login individually.
  • Roo Code MCP config (mcp_settings.json) alwaysAllow updated with all 100 tool names.