- MCP protocol compliance: ping, resources/list, prompts/list, logging/setLevel, 202 notifications - Screenshot tool: WebKit snapshot to PNG to base64 MCP image content - Session management: Mcp-Session-Id header, SSE response format, GET/DELETE endpoints - Phase 3: 70 new tools across 8 batches (extended interaction, get/state, find elements, wait/batch, cookies/storage, mouse/clipboard/settings, frames/dialogs/debug, complex) - Favicon support: enabled WebKitFaviconDatabase, notify::favicon signal handler - Tab UI: favicons in tab labels, evenly distributed tabs, immediate title from URL host - Refresh button: left-click reload, right-click hard reload menu (bypass cache, clear site data) - Webview sizing fix: gtk_widget_set_vexpand/hexpand to prevent 1px height blank page - Updated Roo Code MCP config (100 tools in alwaysAllow) - Updated plans: mcp-server.md, agent-tools.md, phase3-tools.md
30 KiB
Agent Tools Implementation Plan
Overview
Make sovereign_browser agentically capable by embedding a WebSocket server
in the browser process. External AI agents (LLM tools, scripts, other
processes) connect to ws://localhost:PORT and send JSON tool commands to
navigate, inspect, and interact with web pages.
The design is inspired by agent-browser,
which uses a snapshot + ref workflow: take a snapshot (accessibility tree
with element refs), then interact by ref (click @e2, fill @e3). This is
the optimal pattern for LLM-driven browser automation.
Architecture
flowchart TB
subgraph AgentProcess
Agent[AI Agent / Script / CLI]
end
subgraph BrowserProcess
WSServer[WebSocket Server — SoupServer]
ToolDispatch[Tool Dispatcher]
Snapshot[Snapshot Engine]
Tools[Tool Implementations]
TabMgr[Tab Manager]
end
subgraph WebKitGTK
Webview[WebKitWebView — active tab]
JSEval[webkit_web_view_evaluate_javascript]
end
Agent -->|ws://localhost:PORT JSON| WSServer
WSServer --> ToolDispatch
ToolDispatch --> Tools
Tools --> Snapshot
Tools --> TabMgr
Tools -->|JS eval| JSEval
JSEval --> Webview
Snapshot -->|inject + extract| JSEval
TabMgr --> Webview
Key design decisions
-
Agent tools must use the same code path as the GUI — Agent tools should call the same C functions the GUI calls, not a parallel implementation. This ensures that any error a user would see, the agent also sees. Debugging WebKit internals is not our concern; our C code is. If the agent and GUI take different paths, bugs can hide from the agent. The login tools are the one exception (backend-only, bypassing the GTK dialog) — all browser tools go through the same
WebKitWebViewandtab_managerAPIs. -
WebSocket server via libsoup —
SoupServerwithsoup_server_add_websocket_handler()provides a production-ready WebSocket server. libsoup is already linked (WebKitGTK dependency). No new dependencies, no hand-rolling the WebSocket protocol. -
JSON protocol via cJSON — cJSON is already vendored in
nostr_core_lib/cjson/. All tool requests and responses are JSON. -
Snapshot + ref pattern — The core workflow:
snapshotinjects a JS script that walks the accessibility tree, assigns sequential refs (e1,e2, ...) to interactive elements, and returns a text representation of the tree.click @e2looks up the ref in a per-tab ref map, finds the DOM element, and clicks it viaelement.click().- Refs are per-tab and persist until the next
snapshotcall.
-
Per-tab ref storage — Each
tab_info_tgets a ref map (hash table from ref string → element selector or DOM node reference). Since JS execution is per-webview, refs are scoped to the tab's webview. -
Async JS execution —
webkit_web_view_evaluate_javascript()is async. For tools that need the return value (snapshot, get_text, eval), we use a callback-based approach: the tool injects JS that stores the result in a known global variable, then reads it back. Alternatively, we can use a Promise-based approach with a polling mechanism. -
Simple HTTP fallback — The WebSocket server also handles plain HTTP GET requests to
http://localhost:PORT/returning a status JSON (server info, connected clients, tab count). This makes it easy to discover the server withcurlwithout a WebSocket client.
WebSocket protocol
Request format
{
"id": 1,
"tool": "snapshot",
"params": {
"interactive": true,
"compact": true
}
}
Response format
{
"id": 1,
"success": true,
"data": {
"snapshot": "- heading \"Example Domain\" [ref=e1] [level=1]\n- link \"More information...\" [ref=e2]",
"refs": {
"e1": {"role": "heading", "name": "Example Domain", "level": 1},
"e2": {"role": "link", "name": "More information...", "href": "https://example.com"}
}
}
}
Error response
{
"id": 1,
"success": false,
"error": {
"code": "ELEMENT_NOT_FOUND",
"message": "No element with ref @e99"
}
}
Push events (server → client)
{
"type": "event",
"event": "load",
"data": {"url": "https://example.com", "title": "Example Domain", "tab": 0}
}
{
"type": "event",
"event": "tab_changed",
"data": {"tab": 1, "url": "https://example.com", "title": "Example"}
}
Tool catalog — full comparison with agent-browser
The table below lists every agent-browser CLI command and whether we plan to implement it. "Phase 1" tools are in the initial implementation; "Phase 2" tools are deferred but designed for; "N/A" tools don't apply to our architecture (CDP-specific, cloud providers, etc.).
Core commands
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
open <url> |
open |
1 | Navigate active tab |
click <sel> |
click |
1 | By ref or CSS selector |
dblclick <sel> |
dblclick |
2 | Double-click |
focus <sel> |
focus |
1 | Focus element |
type <sel> <text> |
type |
1 | Type into element |
fill <sel> <text> |
fill |
1 | Clear and fill |
press <key> |
press |
1 | Press keyboard key |
keyboard type <text> |
keyboard_type |
2 | Type with real keystrokes |
keyboard inserttext <text> |
insert_text |
2 | Insert text without key events |
keydown <key> |
keydown |
2 | Hold key down |
keyup <key> |
keyup |
2 | Release key |
hover <sel> |
hover |
1 | Hover element |
select <sel> <val> |
select |
2 | Select dropdown option |
check <sel> |
check |
2 | Check checkbox |
uncheck <sel> |
uncheck |
2 | Uncheck checkbox |
scroll <dir> [px] |
scroll |
1 | Scroll page |
scrollintoview <sel> |
scroll_into_view |
2 | Scroll element into view |
drag <src> <tgt> |
drag |
2 | Drag and drop |
upload <sel> <files> |
upload |
2 | Upload files |
screenshot [path] |
screenshot |
1 | Take screenshot |
screenshot --annotate |
screenshot_annotated |
2 | Annotated with element labels |
pdf <path> |
pdf |
2 | Save as PDF |
snapshot |
snapshot |
1 | Accessibility tree with refs |
eval <js> |
eval |
1 | Run JavaScript |
connect <port> |
N/A | — | CDP-specific, we are the server |
stream enable |
stream_enable |
2 | WebSocket viewport streaming |
stream status |
stream_status |
2 | Streaming state |
stream disable |
stream_disable |
2 | Stop streaming |
close |
close |
1 | Close active tab |
close --all |
close_all |
2 | Close all tabs |
Get info commands
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
get text <sel> |
get_text |
1 | Get text content |
get html <sel> |
get_html |
1 | Get innerHTML |
get value <sel> |
get_value |
2 | Get input value |
get attr <sel> <attr> |
get_attr |
1 | Get attribute |
get title |
get_title |
1 | Get page title |
get url |
get_url |
1 | Get current URL |
get cdp-url |
N/A | — | CDP-specific |
get count <sel> |
get_count |
2 | Count matching elements |
get box <sel> |
get_box |
2 | Get bounding box |
get styles <sel> |
get_styles |
2 | Get computed styles |
Check state commands
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
is visible <sel> |
is_visible |
2 | Check if visible |
is enabled <sel> |
is_enabled |
2 | Check if enabled |
is checked <sel> |
is_checked |
2 | Check if checked |
Find elements (semantic locators)
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
find role <role> ... |
find_role |
2 | By ARIA role |
find text <text> ... |
find_text |
2 | By text content |
find label <label> ... |
find_label |
2 | By label |
find placeholder <ph> ... |
find_placeholder |
2 | By placeholder |
find alt <text> ... |
find_alt |
2 | By alt text |
find title <text> ... |
find_title |
2 | By title attr |
find testid <id> ... |
find_testid |
2 | By data-testid |
find first <sel> ... |
find_first |
2 | First match |
find last <sel> ... |
find_last |
2 | Last match |
find nth <n> <sel> ... |
find_nth |
2 | Nth match |
Wait commands
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
wait <selector> |
wait_for |
1 | Wait for element visible |
wait <ms> |
wait |
1 | Wait for time |
wait --text "..." |
wait_for_text |
2 | Wait for text to appear |
wait --url "..." |
wait_for_url |
2 | Wait for URL pattern |
wait --load networkidle |
wait_for_load |
2 | Wait for load state |
wait --fn "..." |
wait_for_fn |
2 | Wait for JS condition |
Batch execution
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
batch --json |
batch |
2 | Execute multiple commands |
Clipboard
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
clipboard read |
clipboard_read |
2 | Read clipboard |
clipboard write "..." |
clipboard_write |
2 | Write clipboard |
clipboard copy |
clipboard_copy |
2 | Copy selection |
clipboard paste |
clipboard_paste |
2 | Paste from clipboard |
Mouse control
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
mouse move <x> <y> |
mouse_move |
2 | Move mouse |
mouse down [button] |
mouse_down |
2 | Press button |
mouse up [button] |
mouse_up |
2 | Release button |
mouse wheel <dy> [dx] |
mouse_wheel |
2 | Scroll wheel |
Browser settings
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
set viewport <w> <h> |
set_viewport |
2 | Set viewport size |
set device <name> |
set_device |
N/A | Device emulation is CDP-specific |
set geo <lat> <lng> |
set_geo |
N/A | Geolocation emulation |
set offline [on|off] |
set_offline |
2 | Toggle offline |
set headers <json> |
set_headers |
2 | Extra HTTP headers |
set credentials <u> <p> |
set_credentials |
2 | HTTP basic auth |
set media [dark|light] |
set_media |
2 | Emulate color scheme |
Cookies and storage
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
cookies |
cookies_get |
2 | Get all cookies |
cookies set <name> <val> |
cookies_set |
2 | Set cookie |
cookies clear |
cookies_clear |
2 | Clear cookies |
storage local |
storage_local_get |
2 | Get localStorage |
storage local <key> |
storage_local_get_key |
2 | Get specific key |
storage local set <k> <v> |
storage_local_set |
2 | Set value |
storage local clear |
storage_local_clear |
2 | Clear all |
storage session |
storage_session_* |
2 | Same for sessionStorage |
Network
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
network route <url> |
network_route |
N/A | Request interception is CDP-specific |
network unroute |
network_unroute |
N/A | CDP-specific |
network requests |
network_requests |
N/A | CDP-specific |
network request <id> |
network_request_detail |
N/A | CDP-specific |
network har start |
har_start |
N/A | CDP-specific |
network har stop |
har_stop |
N/A | CDP-specific |
Tabs and windows
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
tab |
tab_list |
1 | List tabs |
tab new [url] |
tab_new |
1 | New tab |
tab <n> |
tab_switch |
1 | Switch to tab n |
tab close [n] |
tab_close |
1 | Close tab |
window new |
window_new |
N/A | Multi-window is future roadmap |
Frames
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
frame <sel> |
frame_switch |
2 | Switch to iframe |
frame main |
frame_main |
2 | Back to main frame |
Dialogs
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
dialog accept [text] |
dialog_accept |
2 | Accept dialog |
dialog dismiss |
dialog_dismiss |
2 | Dismiss dialog |
dialog status |
dialog_status |
2 | Check if dialog open |
Diff
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
diff snapshot |
diff_snapshot |
N/A | Future, after snapshot is stable |
diff screenshot |
diff_screenshot |
N/A | Future |
diff url |
diff_url |
N/A | Future |
Debug
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
trace start/stop |
N/A | — | Chrome trace, not available in WebKitGTK |
profiler start/stop |
N/A | — | Chrome DevTools profiler |
console |
console |
2 | View console messages |
errors |
errors |
2 | View page errors |
highlight <sel> |
highlight |
2 | Highlight element |
inspect |
N/A | — | Opens Chrome DevTools |
state save/load/list |
state_save/load/list |
2 | Save/restore auth state |
Navigation
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
back |
back |
1 | Go back |
forward |
forward |
1 | Go forward |
reload |
reload |
1 | Reload page |
Setup
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
install |
N/A | — | We bundle WebKitGTK, no install needed |
upgrade |
N/A | — | Package manager handles updates |
Authentication
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
auth save |
N/A | — | We use Nostr identity, not credential vault |
auth login |
N/A | — | Nostr login is our native flow |
Sessions
| agent-browser command | Our tool | Phase | Notes |
|---|---|---|---|
--session <name> |
N/A | — | We have one browser process, tabs instead |
--session-name <name> |
N/A | — | Session persistence is our session.c |
--profile <path> |
N/A | — | WebKitGTK handles profile data |
--state <path> |
state_save/load |
2 | Could implement via WebKitGTK data manager |
Login tools (sovereign_browser-specific, not in agent-browser)
These tools are unique to sovereign_browser. They allow an agent to handle the Nostr login flow programmatically — essential because the browser shows a modal login dialog on startup that blocks all other tools until the user (or agent) signs in.
The agent server starts before the login dialog, so the agent can
authenticate without any human interaction. The login tools call the same
nostr_core_lib functions that the GTK login dialog uses, bypassing the
dialog entirely.
| Tool | Params | Description |
|---|---|---|
login_status |
{} |
Check if logged in; return method, pubkey, readonly |
login |
{"method": "local", "nsec": "nsec1..."} |
Sign in with local key |
login |
{"method": "seed", "mnemonic": "word1 word2 ...", "passphrase": "..."} |
Sign in with BIP-39 seed phrase |
login |
{"method": "readonly", "npub": "npub1..."} |
Sign in read-only with npub |
login |
{"method": "nip46", "bunker_url": "bunker://..."} |
Sign in with NIP-46 remote signer |
login |
{"method": "nsigner", "transport": "qrexec", "device": "nostr_signer", "service": "qubes.NsignerRpc", "index": 0} |
Sign in with n_signer hardware |
logout |
{} |
Log out and clear signer |
switch_identity |
{"method": "local", "nsec": "nsec1..."} |
Switch to a new identity (same params as login) |
Login response:
{
"id": 1,
"success": true,
"data": {
"method": "local",
"pubkey": "a1b2c3...",
"npub": "npub1...",
"readonly": false
}
}
Architecture change: The agent server must start before the login
dialog. In main.c, the startup sequence becomes:
gtk_init()+ load settings/history- Create the main window (but don't show it yet)
- Start the agent server (so agents can connect)
- If
agent_server_enabledand an agent connects within a timeout, wait for the agent to calllogin - Otherwise (no agent, or timeout), show the GTK login dialog as before
- After login (via agent or dialog), create tabs and show the window
This means the agent server has two states:
- Pre-login: Only
login_status,login,logouttools are available - Post-login: All browser tools become available
Summary
| Category | agent-browser commands | Our Phase 1 | Our Phase 2 | Our Phase 3 | N/A |
|---|---|---|---|---|---|
| Login (unique) | 0 (8 new) | 8 | 0 | 0 | 0 |
| Core | 29 | 10 | 0 | 14 | 5 |
| Get info | 9 | 4 | 0 | 4 | 1 |
| Check state | 3 | 0 | 0 | 3 | 0 |
| Find elements | 10 | 0 | 0 | 10 | 0 |
| Wait | 6 | 2 | 0 | 4 | 0 |
| Batch | 1 | 0 | 0 | 1 | 0 |
| Clipboard | 4 | 0 | 0 | 4 | 0 |
| Mouse | 4 | 0 | 0 | 4 | 0 |
| Settings | 7 | 0 | 0 | 4 | 3 |
| Cookies/storage | 8 | 0 | 0 | 11 | 0 |
| Network | 6 | 0 | 0 | 0 | 6 |
| Tabs/windows | 5 | 4 | 0 | 0 | 1 |
| Frames | 2 | 0 | 0 | 2 | 0 |
| Dialogs | 3 | 0 | 0 | 3 | 0 |
| Diff | 3 | 0 | 0 | 0 | 3 |
| Debug | 8 | 0 | 0 | 4 | 4 |
| Navigation | 3 | 3 | 0 | 0 | 0 |
| Setup | 2 | 0 | 0 | 0 | 2 |
| Auth | 2 | 0 | 0 | 0 | 2 |
| Sessions | 4 | 0 | 0 | 2 | 2 |
| Total | 119 + 8 | 30 | 0 | 70 | 29 |
Phase 1 (30 tools): Login tools (8) + the minimal set for agent-driven
browser automation (22) — login, navigate, snapshot, inspect, interact,
and manage tabs, plus screenshot. This is enough for an LLM to log in,
browse, read, fill forms, click buttons, and multi-tab. The agent server
starts before login so the agent can authenticate without human
interaction.
Phase 2 (0 tools): Reserved for refinements to Phase 1 tools based on real agent usage — better snapshot formatting, error handling, edge cases.
Phase 3 (70 tools): Everything else that has a WebKitGTK equivalent.
The original plan counted 67, but implementation added 3 extra
storage_session_* tools (the plan listed storage as 8 local-only tools;
we implemented 11 by adding storage_session_get, storage_session_get_key,
storage_session_set, and storage_session_clear). Deferred until Phase 1
is stable and tested with real agents.
N/A (29 tools): CDP-specific features (network interception, Chrome DevTools profiler, trace), cloud provider integrations, and features that don't map to our architecture (device emulation, multi-window). Some could be revisited if WebKitGTK adds equivalent APIs.
Snapshot engine
The snapshot is the heart of the agent system. It injects a JavaScript script into the active tab's webview that:
- Walks the DOM accessibility tree (using
TreeWalkeror recursivequerySelectorAll) - For each interactive element (links, buttons, inputs, selects, textareas,
elements with role/aria-label), assigns a sequential ref (
e1,e2, ...) - Stores a mapping from ref → unique CSS selector (for later re-querying)
- Returns a text representation of the tree:
- heading "Example Domain" [ref=e1] [level=1]
- paragraph "This domain is for use in..."
- link "More information..." [ref=e2]
- textbox "Search" [ref=e3]
- button "Submit" [ref=e4]
Ref storage
The ref map is stored as a JSON string in a hidden window.__agentRefs
global on the page. When a tool like click @e2 is called:
- The tool injects JS:
var el = document.querySelector(window.__agentRefs['e2']); el.click(); - If the element is gone (page navigated), the tool returns an error suggesting a re-snapshot.
Interactive-only mode
When interactive: true, only elements that are focusable or have an
ARIA role are included. This dramatically reduces the snapshot size for
LLM context windows.
New files
src/agent_server.h / src/agent_server.c
/*
* WebSocket server for agent tool commands.
* Uses libsoup SoupServer with a websocket handler at /agent.
*/
void agent_server_start(int port);
void agent_server_stop(void);
int agent_server_get_port(void);
gboolean agent_server_is_running(void);
/* Broadcast a push event to all connected clients. */
void agent_server_emit_event(const char *event_name, cJSON *data);
src/agent_tools.h / src/agent_tools.c
/*
* Tool dispatch: receives a JSON request, executes the tool,
* returns a JSON response.
*/
/* Process a tool request JSON and return response JSON.
* The caller frees the returned cJSON*. */
cJSON *agent_tools_dispatch(cJSON *request);
src/agent_snapshot.h / src/agent_snapshot.c
/*
* Accessibility tree snapshot via JS injection.
* Assigns refs to interactive elements and returns a text tree.
*/
/* The JS script injected for snapshot. Exposed for testing. */
extern const char *AGENT_SNAPSHOT_JS;
/* Parse a snapshot result JSON from the JS execution. */
cJSON *agent_snapshot_parse(const char *js_result);
New files (login-specific)
src/agent_login.h / src/agent_login.c
/*
* Agent-driven Nostr login — calls nostr_core_lib directly,
* bypassing the GTK login dialog. Used by the agent server
* so an AI agent can authenticate without human interaction.
*/
/* Check current login state. Returns JSON response. */
cJSON *agent_login_status(void);
/* Perform login with the given method and params.
* params is a cJSON object with "method" and method-specific fields.
* Returns JSON response with pubkey/npub on success. */
cJSON *agent_login(cJSON *params);
/* Log out — clear signer, reset state. Returns JSON response. */
cJSON *agent_logout(void);
/* Switch identity — same as login but frees the old signer first. */
cJSON *agent_switch_identity(cJSON *params);
/* Returns TRUE if logged in (signer or readonly loaded). */
gboolean agent_is_logged_in(void);
Modified files
src/settings.h / src/settings.c
Add settings:
agent_server_enabled(bool, default: true)agent_server_port(int, default: 17777)agent_server_allowed_origins(string, default: "*" for localhost)agent_login_timeout_ms(int, default: 30000) — how long to wait for an agent to log in before falling back to the GTK dialog
src/tab_manager.h / src/tab_manager.c
- Add
GHashTable *ref_maptotab_info_tfor per-tab element refs - Emit events to
agent_serveron load, title change, tab switch
src/main.c
Major change to startup sequence:
- Call
agent_server_start()beforedo_login()(not after) - After starting the server, wait for either:
- An agent to call
loginwithinagent_login_timeout_ms, OR - The timeout to expire, then show the GTK login dialog
- An agent to call
- After login (via agent or dialog), proceed to create tabs and show window
- Call
agent_server_stop()inon_window_destroy()
Makefile
Add src/agent_server.c src/agent_tools.c src/agent_snapshot.c src/agent_login.c to SRC.
Add libsoup-3.0 to pkg-config if not already included.
Implementation order
Phase 1 — Login + core browser automation
- settings — add agent server settings (port, enabled, allowed origins)
- agent_server — WebSocket server using libsoup SoupServer, JSON protocol
- agent_login — login tools (login_status, login, logout, switch_identity) that call nostr_core_lib directly, bypassing the GTK dialog
- main.c integration (pre-login) — start agent server before login dialog, support agent-driven login with fallback to GTK dialog
- agent_snapshot — the JS injection script + ref assignment
- agent_tools — tool dispatch + core tools (open, snapshot, click, fill, type, eval, get_text, get_html, get_attr, get_url, get_title)
- main.c integration (post-login) — wire browser tools to tab_manager, emit events (load, tab_changed)
- Navigation tools — back, forward, reload, scroll, press, wait, wait_for
- Tab tools — tab_list, tab_new, tab_switch, tab_close
- Interaction tools — hover, focus, close
- Screenshot tool — WebKitGTK snapshot API
- UI status indicator — show server port in toolbar
- Makefile + docs — build and documentation
Phase 2 — Refinements
- Polish Phase 1 tools based on real agent usage
- Better snapshot formatting and error messages
- Edge case handling (page navigation during interaction, stale refs)
Phase 3 — Extended tool catalog (COMPLETE)
- All remaining tools from the comparison table (70 tools — 67 planned
- 3 extra
storage_session_*tools)
- 3 extra
- Find elements, check state, clipboard, mouse control, cookies/storage, frames, dialogs, console/errors, batch, state save/load, etc.
- Implemented in 8 batches (see
plans/phase3-tools.md):- Batch 1: Extended interaction (11 tools)
- Batch 2: Get info + check state (7 tools)
- Batch 3: Find elements (10 tools)
- Batch 4: Wait + batch (5 tools)
- Batch 5: Cookies + storage (11 tools)
- Batch 6: Mouse + clipboard + settings (13 tools)
- Batch 7: Frames + dialogs + debug (10 tools)
- Batch 8: Complex tools (3 tools)
Testing the tools
The agent server starts before login, so the first thing an agent does is authenticate. Once logged in, all browser tools become available.
Login flow
# Using websocat (install: cargo install websocat)
# Check login status (before logging in)
echo '{"id":1,"tool":"login_status","params":{}}' | websocat ws://localhost:17777/agent
# Login with local key (nsec)
echo '{"id":2,"tool":"login","params":{"method":"local","nsec":"nsec1..."}}' | websocat ws://localhost:17777/agent
# Login read-only with npub
echo '{"id":3,"tool":"login","params":{"method":"readonly","npub":"npub1..."}}' | websocat ws://localhost:17777/agent
# Login with seed phrase
echo '{"id":4,"tool":"login","params":{"method":"seed","mnemonic":"abandon abandon abandon ... abandon about"}}' | websocat ws://localhost:17777/agent
# Login with NIP-46 remote signer
echo '{"id":5,"tool":"login","params":{"method":"nip46","bunker_url":"bunker://..."}}' | websocat ws://localhost:17777/agent
# Login with n_signer via Other Qube (qrexec)
echo '{"id":6,"tool":"login","params":{"method":"nsigner","transport":"qrexec","device":"nostr_signer","service":"qubes.NsignerRpc","index":0}}' | websocat ws://localhost:17777/agent
Browser automation flow (after login)
# Open a page
echo '{"id":10,"tool":"open","params":{"url":"https://example.com"}}' | websocat ws://localhost:17777/agent
# Snapshot — get accessibility tree with refs
echo '{"id":11,"tool":"snapshot","params":{"interactive":true}}' | websocat ws://localhost:17777/agent
# Click by ref
echo '{"id":12,"tool":"click","params":{"ref":"@e2"}}' | websocat ws://localhost:17777/agent
# Fill an input
echo '{"id":13,"tool":"fill","params":{"ref":"@e3","value":"test@example.com"}}' | websocat ws://localhost:17777/agent
# Get text
echo '{"id":14,"tool":"get_text","params":{"ref":"@e1"}}' | websocat ws://localhost:17777/agent
# List tabs
echo '{"id":15,"tool":"tab_list","params":{}}' | websocat ws://localhost:17777/agent
# Open new tab
echo '{"id":16,"tool":"tab_new","params":{"url":"https://example.org"}}' | websocat ws://localhost:17777/agent
Python example (persistent connection)
import websocket, json
ws = websocket.create_connection("ws://localhost:17777/agent")
# Login first
ws.send(json.dumps({"id": 1, "tool": "login", "params": {
"method": "readonly", "npub": "npub1..."
}}))
print(json.loads(ws.recv()))
# Then browse
ws.send(json.dumps({"id": 2, "tool": "open", "params": {
"url": "https://example.com"
}}))
print(json.loads(ws.recv()))
ws.send(json.dumps({"id": 3, "tool": "snapshot", "params": {
"interactive": True
}}))
print(json.loads(ws.recv()))
Future: Nostr relay integration
Since the WebSocket server is already in the browser process, a future enhancement could expose Nostr relay connections through the same server. An agent could subscribe to Nostr events and react to them — e.g., a remote agent controlling the browser via Nostr DMs. This is a natural extension of the "sovereign identity" thesis: your Nostr identity controls your browser, not just your signing.
Implementation Status
All phases complete. 100 tools total.
| Phase | Tools | Status |
|---|---|---|
| Phase 1 — Login + core | 30 | ✅ Complete |
| Phase 2 — Refinements | 0 | ✅ N/A (folded into Phase 1/3) |
| Phase 3 — Extended catalog | 70 | ✅ Complete (8 batches) |
| Total | 100 | ✅ |
Phase 3 batches (all complete)
| Batch | Category | Tools | Count |
|---|---|---|---|
| 1 | Extended interaction | dblclick, select, check, uncheck, scroll_into_view, keyboard_type, insert_text, keydown, keyup, drag, close_all | 11 |
| 2 | Get info + check state | get_value, get_count, get_box, get_styles, is_visible, is_enabled, is_checked | 7 |
| 3 | Find elements | find_role, find_text, find_label, find_placeholder, find_alt, find_title, find_testid, find_first, find_last, find_nth | 10 |
| 4 | Wait + batch | wait_for_text, wait_for_url, wait_for_load, wait_for_fn, batch | 5 |
| 5 | Cookies + storage | cookies_get, cookies_set, cookies_clear, storage_local_get, storage_local_get_key, storage_local_set, storage_local_clear, storage_session_get, storage_session_get_key, storage_session_set, storage_session_clear | 11 |
| 6 | Mouse + clipboard + settings | mouse_move, mouse_down, mouse_up, mouse_wheel, clipboard_read, clipboard_write, clipboard_copy, clipboard_paste, set_viewport, set_offline, set_headers, set_credentials, set_media | 13 |
| 7 | Frames + dialogs + debug | frame_switch, frame_main, dialog_accept, dialog_dismiss, dialog_status, console, errors, highlight, state_save, state_load | 10 |
| 8 | Complex tools | upload, pdf, screenshot_annotated | 3 |
| Total | 70 |
Verification
make clean && make— builds cleanly (only pre-existing warnings inlogin_dialog.candagent_mcp.c).tools/listreturns exactly 100 tools.initialize/ping/login_statuswork without login.- Browser tools (
dblclick, etc.) correctly returnNOT_LOGGED_INbefore login. batchdispatches without login; sub-commands enforce login individually.- Roo Code MCP config (
mcp_settings.json)alwaysAllowupdated with all 100 tool names.