screamingfrog-audit-mcp

<p align="center"> <img src="assets/banner.svg" alt="screamingfrog-audit-mcp — drive the Screaming Frog SEO Spider from any MCP client" width="100%"> </p> <p align="center"> <a href="https://github.com/mshahiddigital/screamingfrog-audit-mcp/actions/workflows/ci.yml"><img src="https://github.com/mshahiddigital/screamingfrog-audit-mcp/actions/workflows/ci.yml/badge.svg" alt="CI"></a> <a href="https://pypi.org/project/screamingfrog-audit-mcp/"><img src="https://img.shields.io/pypi/v/screamingfrog-audit-mcp" alt="PyPI"></a> <a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a> <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT"></a> </p> Drive the [Screaming Frog SEO Spider](https://www.screamingfrog.co.uk/seo-spider/) from Claude, Cursor, or any other MCP client. Crawl a site, get a ranked issue register back, ask questions of the crawl data, and render a shareable report — without opening the GUI or writing a single command. ### It works on the free, unlicensed SEO Spider That is the point of this server, and it is unusual: the other MCP servers for Screaming Frog build on its saved-crawl database, which is a licensed feature, so they need a paid install. This one drives the crawl directly and never touches that database. The free tier caps you at 500 URLs **per invocation** — not per site. So `full=true` reads `robots.txt` and the sitemaps, splits the URLs into batches under the cap, runs each through list mode, and merges the exports. A 3,000-page site audits completely on a free install. A licence removes the cap and unlocks `config=` for JavaScript rendering and custom extraction. Both tiers are supported and the server adapts to whichever you have. ``` You: Crawl example.com and tell me what's actually broken. → start_crawl(url="https://example.com") → crawl_status() # 248 URLs, 51s → get_issues(priority="High") Claude: Three high-priority problems. The big one: robots.txt disallows /_next/, which hides 85 JS and CSS bundles from Google... ``` ## Install ### Step 1 — the two things you need **Screaming Frog SEO Spider**, installed on the same computer ([download](https://www.screamingfrog.co.uk/seo-spider/)). The free version is fine. **Python 3.10 or newer** — macOS and most Linux systems already have it; on Windows get it from [python.org](https://www.python.org/downloads/) and tick **"Add python.exe to PATH"** on the first screen. Then install `uv`, which runs the server for you. Open a terminal and paste one line: **macOS / Linux** — open *Terminal*: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` **Windows** — open *PowerShell* (press Start, type `powershell`): ```powershell powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" ``` **Close the terminal and open a new one** so it picks up the change. ### Step 2 — one command sets up your apps ```bash uvx screamingfrog-audit-mcp --install ``` That finds Claude Desktop, Cursor and VS Code on your machine and configures each one. You never open a JSON file. It: - **keeps everything already in your config** — other MCP servers and settings are untouched - **writes the full path** to the program, because Claude Desktop and other desktop apps cannot find commands the way a terminal can - **saves a timestamped backup** before changing anything - **re-reads the file afterwards** to prove it is still valid, and restores your backup if it is not - **refuses to touch a config that is already broken**, and tells you instead Want to see what it would do first? Add `--dry-run`. Want one app only? `--install claude-desktop`, `--install cursor` or `--install vscode`. Then **fully quit and reopen the app** — on macOS that is Cmd+Q, not just closing the window. Ask it: *"check my screaming frog install"*. <details> <summary><b>Only using Claude Desktop? Full walkthrough, Mac and Windows</b></summary> You still need a terminal **once**, to install the pieces. After that you never open one again — everything happens inside Claude Desktop. **1. Install Screaming Frog SEO Spider** [Download it](https://www.screamingfrog.co.uk/seo-spider/) and open it once so it finishes setting itself up. The free version is fine. Then close it. **2. Install Python** (skip on macOS, it is already there) Windows: [python.org/downloads](https://www.python.org/downloads/), run the installer, and **tick "Add python.exe to PATH"** on the first screen. That checkbox matters; without it nothing below works. **3. Open a terminal** - **macOS** — press Cmd+Space, type `Terminal`, press Enter - **Windows** — press the Start key, type `powershell`, press Enter **4. Install uv** — copy the line for your system, paste, press Enter macOS: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Windows: ```powershell powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" ``` **5. Close that terminal window and open a new one.** It needs a fresh start to see what you just installed. **6. Set up Claude Desktop** — paste this and press Enter: ```bash uvx screamingfrog-audit-mcp --install claude-desktop ``` You should see `[ OK ] Claude Desktop: Added to ...`. It writes the settings file for you and backs up anything already there. **7. Fully quit Claude Desktop and reopen it** - **macOS** — Cmd+Q. Closing the window is not enough - **Windows** — right-click the Claude icon near the clock, choose Quit **8. Check it worked.** In Claude Desktop, ask: > check my screaming frog install It should report the Spider it found and your licence tier. Then try: > crawl example.com and tell me what is broken **If step 6 says "command not found"**, step 5 was skipped — close the terminal and open a new one. **If Claude Desktop shows no tools**, it was not fully quit; use Cmd+Q or the tray icon, not the window close button. </details> ### Command-line assistants These have their own one-liners, so there is nothing to edit: ```bash claude mcp add screaming-frog -- uvx screamingfrog-audit-mcp codex mcp add screaming-frog -- uvx screamingfrog-audit-mcp hermes mcp add screaming-frog --command uvx --args screamingfrog-audit-mcp ``` ### If something is not working ```bash uvx screamingfrog-audit-mcp --doctor ``` It checks Python, the MCP SDK, whether the Spider is found *and actually runs*, which options your Screaming Frog version supports, your licence tier, and whether the audit folder is writable — then prints the exact config for your machine. ``` [PASS] Python: Python 3.12.7 on Darwin [PASS] MCP SDK: MCP SDK 2.1.1, using MCPServer (mcp 2.x) [FAIL] SEO Spider: Screaming Frog SEO Spider not found Install it from https://www.screamingfrog.co.uk/seo-spider/ , or set SCREAMING_FROG_PATH to the executable. ``` An MCP server talks over a pipe, so when it fails to start the app just shows a dead server with no reason. The doctor is how you see the reason. <details> <summary><b>Editing the config by hand instead</b></summary> `--install` exists so you do not have to, but if you prefer: | App | File | |---|---| | Claude Desktop (macOS) | `~/Library/Application Support/Claude/claude_desktop_config.json` | | Claude Desktop (Windows) | `%APPDATA%\Claude\claude_desktop_config.json` | | Cursor | `~/.cursor/mcp.json` | | VS Code | Command Palette → *MCP: Open User Configuration* | Three things go wrong by hand, all silently: 1. **Use the full path to `uvx`, never bare `uvx`.** Desktop apps launch without your shell's `PATH`. Find it with `which uvx` (macOS/Linux) or `where.exe uvx` (Windows). 2. **On Windows, double every backslash.** `C:\Users\you\...` must be written `C:\\Users\\you\\...` in JSON, or the file is invalid and the app ignores *every* server in it. 3. **Add to what is already there.** Do not paste over the file. ```json { "mcpServers": { "screaming-frog": { "command": "/Users/you/.local/bin/uvx", "args": ["screamingfrog-audit-mcp"] } } } ``` VS Code differs: the key is `servers`, not `mcpServers`, and each entry needs `"type": "stdio"`. </details> ### ChatGPT, and what cannot work The **ChatGPT desktop app** bundles Codex and reads the same `~/.codex/config.toml`, so the `codex mcp add` line above configures it too. **ChatGPT on the web or on your phone cannot use this**, and no config changes that. Those connectors call a remote web address, while this is a program that runs on your computer and drives the Screaming Frog installed there. It has to live where the Spider lives. The same is true of any assistant that only accepts a remote URL. ### Optional settings Add these to the `env` block of any config (a `[mcp_servers.screaming-frog.env]` table in Codex): | Variable | What it does | |---|---| | `SCREAMING_FROG_PATH` | Path to the SEO Spider executable, if it is installed somewhere non-standard | | `SF_MCP_AUDIT_DIR` | Where crawl folders are written (default `~/.screamingfrog-audit-mcp/audits`) | | `SF_ALLOWED_DOMAINS` | Comma-separated domains this server may crawl. Set it for anything unattended | ```json { "mcpServers": { "screaming-frog": { "command": "uvx", "args": ["screamingfrog-audit-mcp"], "env": { "SF_MCP_AUDIT_DIR": "/Users/you/audits", "SF_ALLOWED_DOMAINS": "example.com,acme.co.uk" } } } } ``` ## The free-tier situation The received wisdom is that the Screaming Frog CLI needs a licence. It does not. Verified against a build reporting `Licence Status: Missing`: | | | |---|---| | **Works unlicensed** | `--headless`, spider / list / sitemap crawl modes, every tab export, every saved report, sitemap generation | | **Licence-gated** | save/load crawl, crawl comparison, config files, JavaScript rendering, scheduling, and the GA4 / Search Console / PageSpeed / Ahrefs / Moz integrations | | **Capped** | 500 URLs **per invocation** — not per site | Because the cap is per invocation, `start_crawl(full=true)` discovers URLs from `robots.txt` and the sitemaps, batches them under the cap through list mode, and merges the exports back into one set. That crawls a site of any size on the free tier. A licence removes the cap and makes `full` unnecessary. Everything else works the same. ## Tools | Tool | What it does | |---|---| | `check_install` | Install status, licence status, current limits, and how to fix a failed lookup | | `available_filters` | The export names **your** Screaming Frog build accepts | | `start_crawl` | Background headless crawl. `full` beats the free cap, `everything` exports every table | | `crawl_status` | Poll a running crawl. Omit `job_id` for the most recent | | `cancel_crawl` | Stop a crawl, keep partial exports | | `list_crawls` | Crawl folders, newest first, with headline counts | | `get_issues` | The priority-ranked issue register. **Start here** | | `get_analysis` | What the *set* of URLs means: depth, link equity, sitemap accuracy, content depth, performance, indexability, duplication | | `list_exports` | The CSV exports in a crawl, with row counts | | `read_export` | Rows from one export: column-selectable, paged, capped, filtered by `contains` / `exact` / `regex` on any column | | `aggregate_export` | **Counts and group-by without returning rows** — "how many 404s", "status codes by folder" — in one small response | | `storage_summary` | Disk used per saved crawl, largest first | | `delete_crawl` | Permanently delete a crawl folder (requires `confirm`) | | `build_report` | The branded deliverable: **`audit-workbook.xlsx`** (Summary, Issue register, Analysis, Data index, and every export as its own highlighted sheet), a printable `report.html`, `report.md` and `analysis.json`. `consolidate=True` folds the CSV exports into the workbook and deletes them, leaving one file | ## Two design decisions worth knowing **Crawls are background jobs.** A crawl takes minutes; an MCP call should answer in seconds. `start_crawl` forks a detached child and hands back a `job_id`. Nothing blocks unless you pass `wait_seconds`. The crawl survives the MCP server restarting. **Reads are capped, on purpose.** A finished crawl folder is tens of megabytes of CSV. Feeding that to a model is both useless and expensive. Every read tool caps at 500 rows, lets you pick columns, and truncates long cells. Ask `get_issues` first — it's the whole site in about 60 lines — and `aggregate_export` when the question is "how many" or "broken down by", since counting rows by hand through a model is the expensive way to get a number. Reach for `read_export` only when you actually need the rows. ## The deliverable `build_report` writes four files into the crawl folder: | File | What it is | |---|---| | `audit-workbook.xlsx` | The master workbook. Summary, Issue register, Analysis, Data index, and **every crawl export as its own sheet** — around 70 tables on a full crawl | | `report.html` | Printable summary. Open it and Print to PDF | | `report.md` | The same content as plain text | | `analysis.json` | The derived layer, machine-readable | Every sheet has a frozen header, autofilter, banded rows, sized columns and a coloured tab. **Cells are highlighted where the value is the finding** — issue priority, 4xx/5xx status codes, non-indexable URLs, thin content, slow responses — so the problems are visible before you read a cell. Reports carry a credit line naming the tool and its author. ### One workbook instead of seventy CSVs A finished crawl leaves around seventy exports in the folder, and the workbook already carries every one of them. `build_report(consolidate=True)` folds them in and deletes them, so the folder holds the workbook, the report files, and a `consolidated.json` manifest naming which sheet holds which table. The delete is earned. Each sheet is written, the saved workbook is **reopened from disk**, and every sheet is checked against the row count it should hold. Only then is a file removed. If the save fails, the reopen fails, or a sheet comes back short, nothing is deleted and the reason is reported. An export too large to carry in full is kept on disk and named in the manifest, because a sampled sheet is not a substitute for the file. It is off by default: `read_export` and `aggregate_export` read those CSVs. Turn it on when the folder is a finished deliverable rather than a crawl you are still asking questions about. Those tools then explain the consolidation and point at the workbook instead of reporting the folder as empty. ## Where crawls are stored `~/.screamingfrog-audit-mcp/audits/<label>/` by default. Each folder holds the raw Screaming Frog CSV exports, `audit-summary.json`, and whatever `build_report` wrote — or, after a consolidated report, the workbook and manifest in place of those exports. Override with `SF_MCP_AUDIT_DIR`: ```json { "mcpServers": { "screaming-frog": { "command": "uvx", "args": ["screamingfrog-audit-mcp"], "env": { "SF_MCP_AUDIT_DIR": "/Users/you/audits" } } } } ``` Set `SCREAMING_FROG_PATH` if the Spider is installed somewhere non-standard. ## Restricting what it may crawl By default this server will crawl any host it is asked to. That is fine on your own machine, and a liability when an agent runs unattended: a confused or prompt-injected one can point a crawler at internal addresses or at third parties who did not ask to be crawled. Set `SF_ALLOWED_DOMAINS` and `start_crawl` refuses anything else: ```json "env": { "SF_ALLOWED_DOMAINS": "example.com,acme.co.uk" } ``` Subdomains of a listed domain are allowed; look-alikes are not, so `shop.example.com` passes and `example.com.evil.com` does not. `--doctor` reports whether an allowlist is active. ### Environment variables All three are listed with examples under [Optional settings](#optional-settings) in the install section. ## Use it without MCP The crawl pipeline is a plain module: ```bash python -m screamingfrog_audit_mcp.runner --url https://example.com --output ./audit python -m screamingfrog_audit_mcp.runner --url https://example.com --output ./audit --full ``` ## Gotchas found the hard way - **An unknown command-line flag aborts the whole crawl**, it is not ignored: `FATAL - SeoSpider failed to start ... UnrecognizedOptionException`. The option set differs between versions — `--skip-empty` does not exist in 19.8, for instance — so this server reads the binary's own `--help` and passes only what your build advertises. Works on old and new versions alike. - **One wrong filter name aborts the whole crawl.** Screaming Frog renames tab filters between versions, and an unrecognised name fails the run with a Java stack trace rather than skipping it. Every name is validated against your installed binary at crawl time, so an upgrade degrades instead of breaking. - **Its own `--help` output contains a poisoned entry.** The binary lists a placeholder `UNDEF:Unknown`, and passing it back aborts the crawl with `Using UNDEF as tab is not supported`. It's filtered out. - **`os.kill(pid, 0)` is not a liveness probe on Windows.** Any signal other than CTRL_C/CTRL_BREAK routes to `TerminateProcess`, so the usual "does this pid exist" idiom would kill the crawl and then report it finished. Windows uses `tasklist` to check and `taskkill` to cancel, and never signals. Detaching differs too: `start_new_session` is POSIX-only. - **`everything` mode is curated, not literal.** The Spider lists ~1,150 tab filters, but 800+ are Custom Extraction / Custom Search / Custom JavaScript / AI filters that need a licence-gated config file and are permanently empty. Requesting them costs minutes and returns nothing, so those groups plus the API-dependent ones are excluded. ## Development ```bash git clone https://github.com/mshahiddigital/screamingfrog-audit-mcp cd screamingfrog-audit-mcp pip install -e ".[dev]" pytest ``` The test suite runs on synthetic export fixtures, so it passes on a machine that has never had Screaming Frog installed. ## Changelog See [CHANGELOG.md](CHANGELOG.md). ## License MIT. Not affiliated with or endorsed by Screaming Frog Ltd. You need your own copy of the SEO Spider; issue names, descriptions and fix guidance in the output are Screaming Frog's own.