local-LLM-agent-mcp-interaction

# Local LLM agent that reads AND acts on web pages (camofox-browser + MCP) Code for Part 6 of the *Build Your Own Claude Code* series: *Acting on Web Pages with a Small Local LLM — Click, Type, Submit*. The series so far: - Part 1: the agent (CLI, tools, skills, history, compaction) - Part 2: the browser UI (FastAPI + WebSockets) - Part 3: web search (SearXNG via MCP) - Part 4: web browsing (camofox-browser via MCP) + composing with Part 3's search - Part 5: reading whole pages without truncation — chunked `extract` and `summarize`, plus a family of small single-purpose reader tools (`fetch_snippet`, `fetch_urls`, `fetch_structure`) - Part 6 (this repo): interaction — a persistent-tab lifecycle (`open_tab` / `read_tab` / `close_tab`) plus action tools (`click`, `type_into`, `press_key`, `select_option`, or a single `interact` tool, depending on config) so the agent can fill in a search box, click Next on a paginated list, or submit a form, not just read what is already on the page. The problem this part solves: Part 5's reader tools are URL-first — each one opens a fresh tab, snapshots it, and closes it, so there is nothing to act on. Real tasks (search a site, page through results, fill out and submit a form) need a tab that stays open across calls and element references that survive long enough to click or type into. camofox keeps a tab alive until told to close it; this part adds the MCP plumbing on top — open a persistent tab, read its current state to find an element's `[eN]` ref, act on that ref, then re-read before the next action because every action renumbers the refs. ## Install ```bash git clone https://github.com/jfjensen/local-LLM-agent-mcp-interaction.git cd local-LLM-agent-mcp-interaction python -m venv .venv # Linux / macOS: source .venv/bin/activate # Windows PowerShell: .\.venv\Scripts\Activate.ps1 pip install -e . ``` You will also need: - **Docker** to run camofox-browser and SearXNG (Stage 1 brings up both). - **Ollama** with a tool-capable model. The default is `qwen3.5:9b`: ```bash ollama pull qwen3.5:9b ``` SearXNG comes pre-configured (the settings file is mounted into the container), so the JSON API works without editing anything. ## Configuration All tunable settings live in `config.toml` at the repo root: - model name, temperature, thinking; - SearXNG and camofox URLs; - chunking and reading budgets (the page snapshot size is capped by camofox itself, not here — see the note in `config.toml`'s `[browser]`); - `[acting]` — the new section for Part 6: `action_tool_style` picks between four intent-named action tools (`click`, `type_into`, `press_key`, `select_option`) and a single `interact(action=...)` tool; `wait_timeout_ms` / `wait_for_network` control how long the server waits for a page to settle after an action; `action_result_snippet` optionally attaches a head-snippet of the post-action page to the action's result; - log verbosity (`logging.level = "DEBUG"` shows each tool call, its arguments, and a preview of what came back). To see what is currently active: ```bash mcp-config-show ``` The loader looks for `config.toml` in the current working directory first, then falls back to the one bundled with the repo. ## How to run | Stage | What it is | How to run | |-------|------------------------------------------------------------------|------------| | 1 | Standing up camofox-browser and SearXNG via Docker | `cd stage1 && docker compose up -d` (see `stage1/README.md` for the camofox image build step) | | 2 | The full agent: search plus the browser reader and action tools | `mcp-agent-stage2` | The browser MCP server (`mcp-browser-stage2`) exposes three groups of tools. **URL-first readers** (from Part 5; each opens a one-shot tab, reads it, closes it): - **`fetch_snippet`** — the head of a page, for a quick look. - **`fetch_urls`** — the page's links as `{text, url}` pairs, absolute. - **`fetch_structure`** — the heading outline. - **`extract`** — named fields as JSON, via chunk-and-merge over the whole page (no truncation). - **`summarize`** — a prose summary, built by combining per-chunk summaries (map-reduce by default, or refine). **Persistent-tab lifecycle and readers** (new in Part 6; for when you need to act on a page, not just read it): - **`open_tab`** — opens a page in a tab that stays open and returns a `tab_id`. - **`read_tab`** — reads the CURRENT state of an open tab: `mode="snippet"` (default, shows the `[eN]` element refs you need for an action), `"urls"`, or `"structure"`. - **`summarize_tab`** / **`extract_tab`** — the same whole-page, no-truncation `summarize` / `extract` logic, applied to the live tab instead of a fresh URL fetch (use these after navigating or acting your way to a page that may not be re-fetchable by URL, such as a form's POST target). - **`close_tab`** — closes one tab; **`close_all_tabs`** — tears down every tab in the session (the agent calls this automatically on shutdown). **Action tools** (new in Part 6; act on an `[eN]` ref from the latest `read_tab`). The surface shown to the model is picked by `config.toml`'s `[acting].action_tool_style`: - `"separate"` (default) — four intent-named tools: **`click`**, **`type_into`** (optionally `submit=True` to press Enter afterwards), **`press_key`**, **`select_option`** (for native `<select>` dropdowns). - `"interact"` — one **`interact(tab_id, action, ...)`** tool with an action enum (`click` / `type` / `press` / `select`), mirroring camofox's own `/act` dispatcher. Every action re-snapshots the tab and reports the resulting `url` and a fresh `refsCount`: refs are renumbered after every action, so the rule is always read_tab → act → read_tab again before the next action. Plus the search server (`mcp-search-part3`), a copy of Part 3's SearXNG MCP server, so the repo is self-contained. ## Probing MCP servers with `inspect_any.py` `inspect_any.py` makes one tool call per process, which is fine for the URL-first readers but not for interaction (each process would get its own tab). For the lifecycle/action tools, use `test_act_flow.py` instead (see below). ```bash # List a server's tools: python inspect_any.py mcp_browser_02.main # Call a reader tool: python inspect_any.py mcp_browser_02.main fetch_snippet --kv url=https://example.com python inspect_any.py mcp_browser_02.main fetch_urls --kv url=https://example.com # Call extract with a JSON Schema (use --args or --args-file for nested args): python inspect_any.py mcp_browser_02.main extract --args-file extract_args.json ``` Where `extract_args.json` might look like: ```json { "url": "https://en.wikipedia.org/wiki/Vleteren", "schema": { "type": "object", "properties": { "mayor": {"type": "string", "description": "The current mayor, from the infobox"}, "postal_code": {"type": "string", "description": "The postal code"}, "population": {"type": "string", "description": "The total population"} } } } ``` ## Exercising the action tools with `test_act_flow.py` Unlike `inspect_any.py`, this opens a single MCP stdio session and runs the whole interaction loop in one process, so the persistent tab stays warm across calls: ```bash python test_act_flow.py # or against a different target: python test_act_flow.py https://en.wikipedia.org/wiki/Main_Page ``` It runs `open_tab` → `read_tab` → `type_into(..., submit=True)` → `read_tab` → a deliberate reuse of the now-stale ref (to demonstrate why refs must be re-read after every action) → `close_tab`. The default target is DuckDuckGo Lite, a tiny HTML page whose search box makes a clean before/after-submit comparison. ## Probing camofox directly with `probe_camofox.py` A lower-level probe that talks to camofox's HTTP API directly (no MCP layer), useful when something in the action tools misbehaves and you need to rule out the MCP server as the cause. It first checks which "open a tab" endpoint your camofox build actually answers to, then drives open → snapshot → type → submit → snapshot → a stale-ref check → close against a real public form: ```bash python probe_camofox.py # or against a different form: python probe_camofox.py https://html.duckduckgo.com/html/ ``` ## Notes - The agent creates a `history/` folder in the current working directory on first run. - The repo ships a copy of Part 3's SearXNG MCP server as `mcp-search-part3`, byte-for-byte the same, so you do not need to install Part 3. - The agent calls `close_all_tabs` automatically on shutdown, since the model does not reliably close tabs itself; persistent tabs would otherwise accumulate in camofox across sessions. ## License MIT © 2026 Jes Fink-Jensen. See [LICENSE](LICENSE) for details. ## Troubleshooting - **The Docker build fails with "dist not found".** Use `Dockerfile.ci` instead of the default `Dockerfile`. See `stage1/README.md`. - **`FileNotFoundError: [WinError 2]` when the agent spawns an MCP server.** Your venv is not activated. Activate it so console scripts are on `PATH`. - **Script exits silently on Windows.** The default asyncio event loop on Windows cannot spawn subprocesses. The agent sets `WindowsProactorEventLoopPolicy` at startup; do the same if you copy the code elsewhere. - **`extract` returns nulls on a page you know has the data.** With the chunked extract this should be rare, but a very large page makes many model calls. Lower `chunking.chunk_chars` for smaller, more numerous chunks, or raise it for fewer, larger ones. - **`fetch_structure` returns few or no headings.** Some pages (short stubs, pages whose content lives in tables or infoboxes) have a thin heading outline. Use `summarize` or `extract` for those. - **camofox returns a small or empty snapshot.** Some pages need more than the default 1.5-second settle. Bump `browser.settle_seconds`. - **An action fails with a stale-ref or "page may have changed" error.** Every action renumbers the tab's `[eN]` refs. Call `read_tab` again to get fresh refs before retrying; never reuse a ref from before an action. - **An action on a result page seems to silently target the wrong element, or the page doesn't look like what you expected.** Don't re-fetch the result by URL — use `read_tab` / `summarize_tab` / `extract_tab` on the tab instead. Some result pages (a form's POST target, for example) only exist inside the tab and are not independently fetchable. - **`select_option` fails even though the dropdown is visible.** camofox only exposes a ref for `<select>`/combobox elements that its accessibility snapshot assigns one to; if `read_tab` shows no `[eN]` ref on the dropdown, it cannot be selected on that build. Use `click` on the visible options instead if the site renders them as a custom (non-`<select>`) widget. - **`probe_camofox.py` can't find an open endpoint.** It tries the canonical `POST /tabs` route first, falling back to the older `POST /tabs/open` shim. If neither returns a `tabId`, your camofox build's API has likely changed; paste the script's output to see which step failed.