Gov-Stat-MCP-Server

# GovData MCP A Model Context Protocol (MCP) server that lets AI agents **search, explore, and fetch open datasets from official government data portals**. One unified tool interface across the United States, United Kingdom, Canada, and Australia — with per-portal quirks (API migrations, auth walls, bilingual metadata, stale DataStore flags) absorbed behind a common adapter layer. Built for the typical agent workflow: *"Is there a government dataset for X?"* → find candidates → inspect the files → preview rows or grab a download link and move on. ### Key Features - **Four national portals, one interface.** US (data.gov v4 / DCAT), UK (data.gov.uk / CKAN), Canada (open.canada.ca / CKAN), Australia (data.gov.au / CKAN + DataStore). - **Token-frugal by design.** Descriptions are HTML-stripped and truncated, search results are compact summaries, and large files are never inlined by accident. - **Row-level preview.** Uses CKAN DataStore where available (Australia), and transparently falls back to reading the head of CSV/TSV files everywhere else. - **Agent-friendly failure modes.** Errors are returned as actionable text ("use `get_download_link` instead", "retry with `dataset_id`", "available portals: …") — never raw tracebacks. - **Extensible registry.** Adding another CKAN-based portal is a single config entry; non-CKAN portals plug in via a small adapter class. - **Dual transport.** `stdio` for local clients (Claude Desktop etc.), stateless `streamable-http` for serverless deployment (Google Cloud Run). ### Supported Portals | Code | Portal | API | Search | Explore | Row Preview | Notes | |------|--------|-----|:------:|:-------:|:-----------:|-------| | `us` | [data.gov](https://catalog.data.gov) | data.gov v4 (DCAT) | ✅ | ✅ | CSV fallback | Requires free [api.data.gov](https://api.data.gov/signup/) key | | `uk` | [data.gov.uk](https://www.data.gov.uk) | CKAN | ✅ | ✅ | CSV fallback | Resource lookups need the parent `dataset_id` | | `ca` | [open.canada.ca](https://open.canada.ca/en/open-data) | CKAN | ✅ | ✅ | CSV fallback | Bilingual titles normalized to English | | `au` | [data.gov.au](https://data.gov.au) | CKAN + DataStore | ✅ | ✅ | ✅ DataStore | Full row queries with text filtering | ### Requirements - Python 3.11+ - An [api.data.gov](https://api.data.gov/signup/) API key for the US portal (free; other portals need no credentials) ### Installation ```bash git clone https://github.com/YOUR_ORG/Gov-Stat-MCP-Server.git cd Gov-Stat-MCP-Server # pip pip install -e ".[dev]" # or conda conda env create -f environment.yml conda activate opendata-mcp ``` Copy `.env.example` to `.env` and set your key: ```bash DATAGOV_API_KEY=your-api-data-gov-key ``` ### Getting Started **Standard config** for most MCP clients (stdio transport): ```json { "mcpServers": { "govdata": { "command": "govdata-mcp", "env": { "DATAGOV_API_KEY": "your-api-data-gov-key" } } } } ``` <details> <summary>Claude Code</summary> ```bash claude mcp add govdata -e DATAGOV_API_KEY=your-key -- govdata-mcp ``` </details> <details> <summary>Claude Desktop</summary> Follow the MCP install [guide](https://modelcontextprotocol.io/quickstart/user) and use the standard config above. Note: `govdata-mcp` must be on the PATH Claude Desktop uses — provide an absolute path to the executable inside your virtualenv/conda env if needed, e.g. `/path/to/envs/mcp-env/bin/govdata-mcp`. </details> <details> <summary>Cursor / Windsurf / other JSON-config clients</summary> Use the standard config above in the client's MCP settings file. </details> <details> <summary>HTTP transport (remote / long-running server)</summary> Run the server in HTTP mode: ```bash MCP_TRANSPORT=http govdata-mcp ``` Then point your client at the endpoint: ```json { "mcpServers": { "govdata": { "url": "http://localhost:8080/mcp" } } } ``` </details> ### Tools The intended agent flow: `list_portals` → `search_datasets` → `get_dataset` → `preview_resource` / `fetch_resource` / `get_download_link`. <details> <summary><b>Discovery</b></summary> - **list_portals** - Description: List available government open-data portals and their capabilities. - Parameters: None - Read-only: **true** - **search_datasets** - Description: Full-text search for datasets on a portal. Returns compact summaries with dataset ids. - Parameters: - `portal` (string): Portal code — `us`, `uk`, `ca`, `au` - `query` (string): Search terms, e.g. `"air quality monitoring"` - `limit` (number, optional): Max results (default 10, max 50) - `org` (string, optional): Publisher/organization slug filter (CKAN portals) - `format` (string, optional): Resource format filter, e.g. `"CSV"` (CKAN portals) - Read-only: **true** </details> <details> <summary><b>Exploration</b></summary> - **get_dataset** - Description: Full dataset metadata — organization, license, tags, description, and the list of downloadable resources with their ids, formats, and sizes. - Parameters: - `portal` (string): Portal code - `dataset_id` (string): Dataset id/slug from `search_datasets` - Read-only: **true** - **preview_resource** - Description: Preview rows from a tabular resource. Tries the portal DataStore first (row counts + text filtering); falls back to reading the head of CSV/TSV files. Non-tabular formats can't be previewed. - Parameters: - `portal` (string): Portal code - `resource_id` (string): Resource id from `get_dataset` - `rows` (number, optional): Rows to return (default 20, max 100) - `query` (string, optional): Full-text row filter (DataStore-backed resources only) - `dataset_id` (string, optional): Parent dataset id — recommended; required on `uk` - Read-only: **true** </details> <details> <summary><b>Retrieval</b></summary> - **fetch_resource** - Description: Download a small text resource (CSV/JSON/XML/…) and return its content inline, truncated at a size cap on a clean line boundary. Binary content is refused with a pointer to the download URL. - Parameters: - `portal` (string): Portal code - `resource_id` (string): Resource id from `get_dataset` - `max_kb` (number, optional): Max kilobytes to inline (default 256, cap 512) - `dataset_id` (string, optional): Parent dataset id — recommended; required on `uk` - Read-only: **true** - **get_download_link** - Description: Return the direct download URL, format, and size for a resource so the agent can proceed on its own. Works for any format, including large/binary files. - Parameters: - `portal` (string): Portal code - `resource_id` (string): Resource id from `get_dataset` - `dataset_id` (string, optional): Parent dataset id — recommended; required on `uk` - Read-only: **true** </details> ### Configuration All settings are environment variables (or a local `.env` file): | Variable | Default | Description | |----------|---------|-------------| | `MCP_TRANSPORT` | `stdio` | `stdio` (local clients) or `http` (streamable-http server) | | `PORT` | `8080` | Listen port for HTTP transport (injected automatically on Cloud Run) | | `DATAGOV_API_KEY` | — | api.data.gov key for the US portal; falls back to `DEMO_KEY` (~30 req/hr) | | `LOG_LEVEL` | `INFO` | Logging verbosity (always written to stderr) | | `HTTP_TIMEOUT_SECONDS` | `30` | Timeout for outbound portal requests | | `HTTP_MAX_RETRIES` | `2` | Retries on transient portal errors (5xx / transport), exponential backoff | | `SEARCH_DEFAULT_LIMIT` | `10` | Default number of search results | | `NOTES_TRUNCATE_CHARS` | `200` | Description truncation length in search summaries | | `FETCH_MAX_BYTES` | `524288` | Hard cap for inlined resource content | ### Docker ```bash make docker-build # build image make docker-run # run on :8080, exactly as Cloud Run would ``` The image runs the HTTP transport in stateless mode and is ready for serverless platforms (Google Cloud Run deployment script included under `deploy/`). ### Development ```bash make run # stdio mode make run-http # HTTP mode on :8080 make inspect # MCP Inspector interactive UI (requires Node.js) make test # offline unit tests make test-live # live end-to-end tests against all four real portals make lint # ruff check + format check ``` The live suite (`pytest -m live`) verifies the full search → explore → preview → fetch chain per country plus portal-specific behavior (Canada's bilingual normalization, UK's dataset-scoped resource lookup, Australia's DataStore paths). Run it before deploying — government portals change without notice. ### Architecture ```text src/govdata_mcp/ ├── server.py # FastMCP instance + tool registration ├── config.py # env-driven settings ├── ckan/ # generic async CKAN client, models, typed errors ├── portals/ │ ├── registry.py # portal registry — single source of truth │ ├── base.py # Portal config + default (vanilla CKAN) adapter │ ├── us.py # data.gov v4 DCAT adapter (auth, cursor paging) │ ├── uk.py # dataset-scoped resource lookup │ └── ca.py # bilingual metadata normalization ├── tools/ # MCP tools: search, explore, fetch └── utils/ # formatting (token economy), CSV-head preview ``` **Adding a portal:** for a standard CKAN portal, add one `Portal(...)` entry to `portals/registry.py` — done. For portals with quirks, subclass `PortalAdapter` and override the narrow hooks (`normalize_dataset`, `build_search_params`) or, for non-CKAN APIs, the operation methods themselves (see `us.py` for a full custom adapter). ### TODO / Roadmap - [ ] **🇨🇳 China portal support** — no unified national open-data API exists; planned as a custom adapter targeting National Bureau of Statistics data (via an existing stats package) with graceful degradation. - [ ] **🇯🇵 Japan portal support** — e-Gov Data Portal ([data.e-gov.go.jp](https://data.e-gov.go.jp)) is CKAN-compatible and should slot into the existing registry; the richer e-Stat statistics API (app-ID auth, Japanese-language metadata) planned as a dedicated adapter. - [ ] Offline unit test suite with recorded portal fixtures (respx) for CI - [ ] Google Cloud Run deployment walkthrough (service is container-ready; script in `deploy/`) - [ ] US v4 publisher/format search filters (pending API documentation) - [ ] More CKAN portals (New Zealand [data.govt.nz](https://data.govt.nz) is a near-free addition) - [ ] Content-type sniffing to rescue mislabeled resource formats ### Security Notes - This server is **read-only** toward the portals — no tool can create, modify, or delete anything. - `fetch_resource` and `preview_resource` download from URLs contained in portal metadata; content is size-capped and binary-checked before being returned to the model, but treat fetched content as untrusted input. - The server itself has no authentication. For remote deployment, put it behind an authenticating proxy or platform-level auth (e.g. Cloud Run with `--no-allow-unauthenticated`). - Keep `DATAGOV_API_KEY` in `.env` / secret manager — never commit it. ### License MIT