mcp-conformance-kit

# mcp-conformance-kit A specification conformance test suite for Model Context Protocol servers. Point it at any MCP server, stdio or HTTP, and it tells you exactly where that server departs from the spec: missing handshake fields, wrong JSON-RPC error codes, malformed tool schemas, capabilities served but never declared, cursors that loop forever, and log lines printed to stdout that quietly corrupt the transport. ``` $ mcpck run "npx -y @modelcontextprotocol/server-filesystem /tmp" PASS LIFE-02 initialize returns protocolVersion, capabilities and serverInfo FAIL RPC-04 Unknown methods return -32601 Method not found expected code -32601, got -32603 WARN TOOL-05 Every tool carries a description undocumented: move_file, list_allowed_directories FAIL HYG-01 stdout carries JSON-RPC frames only 2 non-JSON line(s) on stdout, which corrupts the stdio stream MUST conformance: 88% ``` Exit code is `0` when every MUST-level check passes, `1` when any fails. That makes it a one-line CI gate. ## Why this exists There are plenty of MCP servers now and almost no way to tell a rigorous one from a weekend one. Client authors keep writing defensive workarounds for servers that half-implement the protocol. A conformance suite gives server authors a target to hit and gives client authors evidence when something is not their bug. This is the pre-release counterpart to a runtime guardrail. Guardrails police behaviour in production; this polices correctness before you ship. ## Install ```bash pip install mcp-conformance-kit # stdio servers, zero dependencies pip install "mcp-conformance-kit[http]" # adds streamable HTTP support ``` Python 3.10 or newer. No runtime dependencies for the stdio path, deliberately: a conformance tool that drags in the SDK it is testing against is not measuring anything. ## Usage ```bash # a stdio server, given as the command that launches it mcpck run "python -m my_mcp_server" mcpck run "node dist/server.js" --cwd ./packages/server --env API_KEY=xxx # an HTTP server mcpck run https://example.com/mcp --header "Authorization: Bearer $TOKEN" # narrow the run mcpck run "python -m my_mcp_server" --group tools --group jsonrpc mcpck run "python -m my_mcp_server" --only RPC-04 --only HYG-01 # machine-readable output mcpck run "python -m my_mcp_server" --format json -o report.json mcpck run "python -m my_mcp_server" --format markdown -o CONFORMANCE.md mcpck run "python -m my_mcp_server" --format badge -o badge.json # list everything the suite knows how to check mcpck list ``` By default nothing invokes your tools for real. `--allow-invoke` opts in to the small number of checks that call a live tool, so the suite is safe to run against a server wired to production systems. `--strict` also fails the run on SHOULD-level warnings. ## What it checks | Group | Covers | | --- | --- | | `lifecycle` | initialize shape, serverInfo, version negotiation, unsupported-version handling, pre-handshake requests, ping, stale cancellations | | `jsonrpc` | id echo, `jsonrpc` field, result-xor-error, `-32601` for unknown methods, `-32602` for bad params, error-object shape, unknown notifications, capabilities served without being declared | | `tools` | `tools/list`, name uniqueness and character set, `inputSchema` is an object, `required` names properties that exist, descriptions present, unknown-tool errors, missing-argument rejection | | `resources` | `resources/list`, absolute URIs, unknown-URI errors, RFC 6570 templates | | `prompts` | `prompts/list`, argument shape | | `pagination` | invalid cursors rejected with `-32602`, `nextCursor` actually advances instead of returning the same page forever | | `hygiene` | stdout carries JSON-RPC only, process survives the run, diagnostics go to stderr | Every check is tagged MUST, SHOULD or MAY. Only MUST failures affect the exit code and the headline percentage. Run `mcpck list` for the full catalogue with ids. ## In CI ```yaml - name: MCP conformance run: | pip install mcp-conformance-kit mcpck run "python -m my_mcp_server" --format markdown -o conformance.md ``` For a README badge, publish `badge.json` somewhere reachable and point shields.io at it: ```markdown  ``` ## Writing your own checks Checks are plain functions with a decorator. Return a result, yield several, or return `None` to mean pass. ```python from mcpck import Severity, check from mcpck.core import Context, fail @check( "ACME-01", "Every tool name is namespaced with acme_", severity=Severity.SHOULD, group="house-style", spec_ref="internal/conventions", ) def namespaced_tools(ctx: Context): for tool in ctx.cached_list("tools/list", "tools"): if not tool["name"].startswith("acme_"): yield fail("ACME-01", "Every tool name is namespaced with acme_", Severity.SHOULD, f"{tool['name']!r} is not namespaced") ``` `Context` gives you `call()` for a raw JSON-RPC round trip, `result_of()` for the happy path, `supports()` for capability gating, and `fresh()` when a check needs a second, un-initialised connection. ## Try it without a server Two fixture servers ship with the repo, one compliant and one broken on purpose: ```bash python examples/toy_server.py good # scores 100% python examples/toy_server.py bad # scores 50%, with six blocking failures mcpck run "python examples/toy_server.py bad" ``` ## Design notes The kit speaks raw JSON-RPC rather than using an MCP SDK client. SDKs normalise error codes, coerce types and retry, which is precisely the behaviour under test. Reading the bytes off the wire is the only way to assert on them honestly. Checks are non-destructive by default, capability-gated so a small server is not punished for features it never claimed, and separated into MUST, SHOULD and MAY so that a report distinguishes a spec violation from a matter of taste. ## Status Early. The check catalogue is the interesting part and contributions there are very welcome, particularly around sampling, elicitation, roots, completions and the HTTP transport's session semantics. ## License Apache-2.0.