citeguard

Guard [](https://github.com/xiaweiyi713/citeguard/actions/workflows/ci.yml) [](https://github.com/xiaweiyi713/citeguard/actions/workflows/canary.yml) [](LICENSE) [](pyproject.toml) [](https://pypi.org/project/citationguard/) English · [English](README.en.md) **CiteGuard is a "falsification-first" citation verifier**: For agent writing workflows, it checks three things for each citation - **does the paper exist**, **is the metadata (title/author/year/venue/DOI) correct**, and **does it actually support the claim you're making** - by directly querying OpenAlex / Crossref / arXiv / Semantic Scholar and other real academic databases, and can be used as a **MCP tool** directly called by mainstream agents such as Claude Code, Codex, and Cursor. LLM writing assistants often hallucinate references: fabricating non-existent papers, scrambling real paper metadata, or citing real papers that don't support the argument. CiteGuard acts as a "skeptical reviewer": it treats each citation as a `claim → citation → evidence` problem to **falsify**; if unsure, it clearly says "can't verify" instead of guessing an answer. > **Status:** Alpha (`v0.1.1`, released on [PyPI](https://pypi.org/project/citationguard/) and [MCP official registry](https://registry.modelcontextprotocol.io)). The current active development product surface includes `citeguard.*` audit packages, CLI, MCP server, batch workflows, cache replay, and release gate; historical legacy writing agent experiments are only retained in source code checkouts and are not part of the release package. ## Try it out  You can run `python3 scripts/demo_verify.py` in the source code checkout to see the live demo (it will actually access OpenAlex + arXiv); for installed packages, use `citeguard` / `citeguard-mcp` entry points. ```text Verifying 2 citations against OpenAlex + arXiv ... [OK] VERIFIED (confidence 0.7) Vaswani et al., "Attention Is All You Need", arXiv:1706.03762 sources checked: openalex, arxiv Citation resolves to a real record and the provided metadata matches. [X] NOT_FOUND (confidence 0.8419) (LLM-fabricated) "Quantum Teleportation of Citation Hallucinations in Synthetic Benchmarks" sources checked: openalex, arxiv Could not be verified in openalex, arxiv. ``` > The output is real-time collected, and the confidence and matched records will drift with the source data. **Chinese literature also works out of the box** (GB/T 7714 parsing + global DOI registry, both with real output): ```text $ citeguard extract refs_zh.md # GB/T 7714 Chinese references {"title": "迈向第三代人工智能", "authors": ["张钹", "朱军", "苏航"], "venue": "中国科学: 信息科学", "year": 2020, "reference_format": "gbt7714"} $ citeguard verify --title "迈向第三代人工智能" --doi "10.1360/SSI-2020-0204" verdict: not_found # Open academic sources don't have its metadata…… doi_registration: registered: true # ……but the global DOI registry confirms the paper exists resolution_url: engine.scichina.com/doi/10.1360/SSI-2020-0204 ``` ## What it does CiteGuard checks two questions against **OpenAlex, Crossref, arXiv, and Semantic Scholar**. ### 1. Does the paper exist? Is the metadata correct? `verify_citation` / `audit_citations` parse citations (prioritizing identifiers, otherwise searching by title), and compare your provided metadata field by field: | Verdict | Meaning | |---|---| | `verified` | The paper exists, and your provided metadata matches. | | `metadata_mismatch` | The paper exists but some fields don't match - with **suggested corrections**. | | `not_found` | Could not be verified in the checked sources (marked as high risk, but **not** asserting fabrication). | | `ambiguous` | Multiple candidates are hard to distinguish - requiring DOI / arXiv ID to disambiguate. | ### 2. Does the paper support the claim? (Deep mode) `check_claim_support` parses the paper, then uses reranker + NLI integration to judge the relationship between its abstract and your claim sentence: | Verdict | Meaning | |---|---| | `supported` | The abstract implies the claim. | | `weakly_supported` | Partial/related evidence, but not strong enough. | | `insufficient_evidence` | The abstract doesn't involve the claim - **abstains**, not "doesn't support". | | `contradicted` | The abstract contradicts the claim. | Supportive results come with machine-readable `evidence_scope` fields, so agents won't mistake abstract-level evidence for full-text conclusions. Full-text support has two optional channels: either the caller provides a valid excerpt or local text/PDF files; or set `CITEGUARD_OA_FULLTEXT=1` to let CiteGuard automatically fetch **open-access (OA) paper full texts** (arXiv is all applicable; PDF parsing requires `pip install "citationguard[pdf]"`). CiteGuard doesn't crawl restricted sources or bypass paywalls; remote full texts are limited to OA addresses and disabled by default. Two guardian principles ensure it's "honest": **unreachable sources will never upgrade to "fabrication"** (only reducing confidence, with `outage_limited=true` and reporting `sources_available` / `sources_failed` / `source_failure_mode`); `insufficient_evidence` / `not_found` always state "can't verify", leaving final judgment to humans or host agents. Moreover, **identifiers are hard decisions**: DOI / arXiv IDs are directly checked on their authoritative sources (Crossref / arXiv); if matched, they override any same-name records; if authoritative queries fail, results degrade to `ambiguous` with `outage_limited` marked, and never use pure title matching to impersonate high-confidence conclusions (output includes `identifier_lookup` fields). Multi-source queries run concurrently, with a default budget of 8 seconds (`CITEGUARD_SOURCE_BUDGET`); slow sources are marked as `budget_exceeded dragging down the entire verification. ## Quick Start The core verification library runs on Python ≥ 3.9; Python ≥ 3.10 base installations also include the MCP SDK, allowing registry-browsing clients to directly search and install. > ℹ️ **Package name note**: The project's PyPI release name is **[`citationguard`](https://pypi.org/project/citationguard/)** - install with `pip install citationguard`, but code imports remain `import citeguard`, and command-line entries are still `citeguard` / `citeguard-mcp`. The `citeguard` package on PyPI is an unrelated project by another organization; avoid confusion. Install via PyPI: ```bash python -m pip install citationguard # includes MCP server (Python >= 3.10) python -m pip install "citationguard[models]" # + reranker/NLI model stack for supportive deep mode (heavier) ``` Source code checkout: `python -m pip install -e .` (with extras as above). Check local configuration, then verify citations from the command line: ```bash citeguard status # local readiness; add --check-sources for real-time source probing citeguard verify \ --title "Attention Is All You Need" \ --author "Ashish Vaswani" \ --year 2017 \ --arxiv-id 1706.03762 citeguard audit examples/citations.json --jobs 4 # batch: JSON array or .jsonl citeguard audit examples/references.md --high-risk-only # extract and audit reference files citeguard support \ --claim "The Transformer relies entirely on attention." \ --title "Attention Is All You Need" \ --arxiv-id 1706.03762 citeguard support-audit examples/claim_citations.json # batch "claim-citation" pairs citeguard support-audit examples/claim_citations.jsonl --high-risk-only citeguard support-set examples/citations.json \ --claim "Citation auditing should verify existence, metadata, and claim support." citeguard extract examples/references.md # extract citation candidates from documents citeguard counterevidence --claim "The Transformer relies entirely on attention." ``` Extraction supports Markdown/plain text references, **GB/T 7714 Chinese citation formats** (`[J]/[M]/[C]/[D]` tags, automatically parsed into title/author/journal), LaTeX `\bibitem`, BibTe compiled `.bbl` products, LaTeX `\bibliography{refs}` / `\addbibresource{refs external links (including `\input{...}` / `\include{...}` subfiles), and `.docx` - all using standard libraries (i.e., Markdown/LaTeX/BibTeX/BBL/DOCX citation extraction). Extraction lines retain `source_path` / `source_locator` / line number audit results can point back to original reference entries. output JSON with stable `next_action` enums, risk sorting, and machine-readable errors. Complete CLI documentation (including `cache` checks/exports/cleanup and offline fixture replay [docs/cli_reference.md](docs/cli_reference.md); agent-side complete field contracts see [docs/agent_output_contract.md](docs/agent_output_contract.md). ### Integrate as an agent tool (MCP) - recommended path CiteGuard is listed in the [MCP official registry](https://registry.modelcontextprotocol.io) (`io.github.xiaweiyi713/citeguard`); supported registry-browsing clients can directly search and install. Manual installation: Released package: ```bash python -m pip install citationguard # MCP server requires Python >= 3.10 citeguard-mcp # stdio transport ``` Source code checkout: ```bash python -m pip install -e ".[mcp]" citeguard-mcp ``` Register in any MCP client (e.g., Claude Code): ```json { "mcpServers": { "citeguard": { "command": "citeguard-mcp" } } | Tool | Purpose | |---|---| | `citeguard_status_tool` | Doesn't-time queries; checks MCP/Python readiness, cache, source configuration, and model dependency status. | | `verify_c` | Verifies a single citation; returns verdict,, field-by-field differences, repair suggestions, and checked sources. | | `audit_citations_tool` | Batch-verifies citations; reports per-item + aggregate verdict counts. | | `check_claim_support_tool` | Determines if a paper supports a claim sentence (deep mode). | | `check_claim_support_set_tool Determines if a set of citations jointly supports a claim. | | `search_counterevidence_tool` | Retrieves potential counterevidence线索; only for review线索, not constituting contradiction judgments. | | `audit_claim_support_tool` | Batch-judges "claim-citation" pairs and summarizes support verdicts. | After connecting, call `citeguard_status_tool` once - it reports source health, cache, and model readiness without initiating real-time queries; see [docs/mcp_setup.md](docs/mcp_setup.md) and [docs/agent_output_contract.md](docs/agent_output_contract.md) for details. <sub>mcp-name: io.github.xiaweiyi713/citeguard</sub> Agent clients supporting skills can use [`skills/citeguard-verify/SKILL.md`](skills/citeguard-verify/SKILL.md) to let CiteGuard** verify citations during (presenting results modifying your text), applicable to MCP clients like Code Claude Code, and Cursor. ### As a ```python import os from citeguard.retrieval.scholarly_clients import build_live_metadata_source from citeguard.verification import parse_citation, verify_citation, check_claim_support source = build_live_metadata_source(["openalex", "arxiv"], mailto=os.environEGUARD_MAILTO"]) result =itation(parse_citation(title="Attention Is All You Need", arxiv_id=".03762"), source) print(result.value, result.confidence) # -> verified.7 support = check_claim_support("The entirely on attention.", parse_citation(title="Attention Is All You Need", arxiv_id="1706."), source) print(support.ver, support.engine) ``` ## Configuration | Environment Variable | Default Value | Purpose | |---|---|---| | `CITEGUARD_SOURCES` | `openalex,crossref,arxiv` | Sources to query (also supports `semantic_scholar` / `s2`); unknown source names will report a configuration error | | `CITEGUARD_MAILTO` | — | Real contact email for OpenAlex/Crossref politeness pool; not set will not send `mailto` | | `SEMANTIC_SCHOLAR_API_KEY` | — | Optional, improves Semantic Scholar access | | `CITEGUARD_CACHE` | System user cache directory | Local SQLite parsing cache | | `CITEGUARD_FIXTURE_CITATIONS` | — | JSON/JSONL citation fixture, for deterministic offline runs | | `CITEGUARD_HTTP_TIMEOUT` | `10` | Real-time academic API call timeout (seconds) | | `CITEGUARD_REMOTE_EVIDENCE` | `0` | Set to `1` to fetch landing page abstract snippets | | `CITEGUARD_OA_FULLTEXT` | `0` | Set to `1` to automatically fetch open access (OA) paper full text for full-text level support determination; only OA addresses, never bypass paywalls | | `CITEGUARD_RERANKER_MODEL` / `CITEGUARD_NLI_MODEL` | English model | Supporting deep mode model - non-English arguments please configure a multilingual model | See [docs/configuration.md](docs/configuration.md) for the complete runtime contract (retry/backoff, evidence timeout, cache path, remote evidence boundary). Supporting deep mode downloads model weights on first use, can use `citeguard models warmup` to pre-download. Without `[models]` installed, supporting checks run with labeled `heuristic` engine (never outputs `supported` or `contradicted`); `citeguard status` will report `support_models.engine=heuristic_fallback` and `next_action=install_or_configure_dependency`. ## Chinese Support Text matching is friendly to CJK (Chinese characters are preserved and tokenized as character bigrams, **with zero additional dependencies**); **GB/T 7714 Chinese references** (`author. Title [J]. Journal, Year, Volume(Issue): Page` etc.) can be automatically parsed into structured fields for verification. When determining the support of Chinese arguments, please point `CITEGUARD_RERANKER_MODEL` / `CITEGUARD_NLI_MODEL` to a multilingual model. **Practical boundaries for Chinese papers**: English papers by Chinese authors (SCI/EI/arXiv) have complete coverage; most Chinese core journals are not in the OpenAlex/Crossref system (their DOIs are registered with Chinese DOI/ISTIC), and title verification usually returns `not_found` - this indicates "open source not found", not a fabrication accusation. Crossref has no effect on searching pure Chinese titles, CiteGuard will automatically skip this query to avoid invalid requests. **Chinese citations with DOIs have a safety net**: if open source is not found, CiteGuard will query the global DOI registry (doi.org Handle system, covering all registration agencies including Chinese DOI/ISTIC), and report in the `doi_registration` field whether the DOI is truly registered and its official resolution address - if registered, it indicates "metadata not open" rather than "paper does not exist"; if not registered, it will further reduce credibility, but still not constitute a fabrication proof. Can use `CITEGUARD_DOI_REGISTRY=0` to disable. CNKI and Wanfang **are not** integrated: both do not have open/free APIs, and we do not crawl restricted content. ChinaXiv feasibility survey conclusion is NO-GO (its OAI endpoint has access restrictions) - see [`docs/chinaxiv_spike.md`](docs/chinaxiv_spike.md); a pluggable source interface is reserved, and an adapter can be added if an open endpoint appears. ## Parsing Process 1. **Parse** input; DOIs / arXiv IDs / years in free text will be automatically extracted. 2. **Identifier priority**: DOIs or arXiv IDs determine paper resolution. 3. **Otherwise, search by title** in selected sources, scoring candidate matches primarily by title. 4. **Only compare fields you actually provided**, giving differences field by field. 5. **Make a determination** (existence/metadata, or support based on abstract-level evidence). ## Boundaries and Known Limitations **Current capabilities**: existence + metadata verification, abstract-level support verification, user-provided local full-text evidence files, multi-citation argument checking, multi-source adapters, SQLite caching, Markdown/LaTeX/BibTeX/BBL/DOCX reference extraction, MCP server, Claude Code skill, offline evaluation. **Known limitations** - **Identifiers are a reliable path**. When a DOI or arXiv ID is present, resolution is deterministic - provide it if possible. - **Title-only matching is best-effort**. The same title may correspond to multiple records (e.g., original paper + reprints with different years); without identifiers, correct citations may match to same-name records with mismatched `year`/`venue`, reporting `metadata_mismatch`. Consider year/venue mismatches under title-only matching as "to confirm". - **Support determination is abstract-level by default**, unless you provide full-text evidence. It judges based on abstracts, fetched metadata snippets, and your provided legitimate local text/PDF evidence; abstentions (`insufficient_evidence`) are common and expected. - **Support evaluation is a synthetic seed fixture**, split into train/dev/test sets - it's a regression test suite, not a final human review benchmark. **Not yet implemented**: automatic full-text retrieval from arbitrary or restricted sources, cross-paper full-text multi-hop reasoning, counter-evidence verdicting, large-scale human review benchmarks. Currently only supports explicitly enabled open access full text and user-authorized local/abstract evidence. See [ROADMAP.md](ROADMAP.md). ## Testing and Reproduction ```bash python3 -m unittest discover -s tests -v # Full unit test suite; MCP stdio smoke test skips if SDK is missing python3 scripts/smoke_mcp.py --require-sdk # MCP stdio smoke test; MCP SDK requires Python 3.10+ python3 scripts/eval_verification.py # Offline deterministic existence/metadata evaluation python3 scripts/eval_support.py --report --split test --quality-gate python3 scripts/release_package_gate.py # Development/contract testing, does not grant release qualification python3 -m pip install -e ".[models]" python3 scripts/automated_release_review.py --output automated-release-review.json python3 scripts/release_package_gate.py --release-claim-mode software --automated-review-report automated-release-review.json --require-build-tools ``` All tests and evaluations are offline and run in CI. Evaluation datasets are located in [`data/eval/`](data/eval/). Supporting evaluation workflow - metrics, quality gates, labeled sidecar, blinded labeling packages - see [docs/support_eval.md](docs/support_eval.md); release smoke tests and process see [docs/release_checklist.md](docs/release_checklist.md). Production model review can pass ordinary software releases but will not and cannot upgrade maintainer-synced labels to human review benchmarks; such claims still require real independent labeling and adjudication. ## Project Structure ```text citeguard/ verification/ # Core: parsing, resolution, verification, auditing, caching, support, evaluation cli.py # Zero-dependency `citeguard` command: status/verify/audit runtime.py # Shared environment, source, cache, and state configuration mcp/ # FastMCP server, exposes status + verification tools retrieval/ # Academic source adapters (OpenAlex/Crossref/arXiv/Semantic Scholar) + retrievers verifiers/ # Existence/metadata + reranker+NLI support integrations citation/ graph/ audit/ # Shared models and tools benchmark/ # Supporting evaluation metrics, calibration, and experiment artifacts (shipped with package) skills/citeguard-verify/ # Reusable Codex/Claude/Cursor agent skill scripts/ # demo + evaluation + corpus/model tools data/eval/ # Offline benchmarks docs/ # Release documentation, architecture, benchmark notes, survey notes legacy/ # Historical writing agent prototypes (orchestrator/planner/writer/api), only present in source checkout, not shipped with package tests/ # unittest suite ``` New code should import from `citeguard` or `citeguard.*`. Historical root package compatibility layers have been removed; source checkout and release package expose `citeguard.*` product surface; see [`docs/public_api_migration.md`](docs/public_api_migration.md). ## Documentation - Installation and reference: [`docs/claude_code_quickstart.md`](docs/claude_code_quickstart.md) · [`docs/troubleshooting.md`](docs/troubleshooting.md) · [`docs/configuration.md`](docs/configuration.md) · [`docs/mcp_setup.md`](docs/mcp_setup.md) · [`docs/cli_reference.md`](docs/cli_reference.md) · [`docs/agent_output_contract.md`](docs/agent_output_contract.md) · [`docs/error_codes.md`](docs/error_codes.md) · [`docs/public_api_migration.md`](docs/public_api_migration.md) - Benchmarking: [`docs/support_eval.md`](docs/support_eval.md) · [`docs/benchmark_design.md`](docs/benchmark_design.md) · [`docs/benchmark_todo.md`](docs/benchmark_todo.md) · [`docs/support_labeling_guidelines.md`](docs/support_labeling_guidelines.md) - Release and security: [`docs/release_checklist.md`](docs/release_checklist.md) · [`docs/security_compliance.md`](docs/security_compliance.md) - Architecture: [`docs/architecture.md`](docs/architecture.md) · Roadmap: [`ROADMAP.md`](ROADMAP.md) · ChinaXiv survey: [`docs/chinaxiv_spike.md`](docs/chinaxiv_spike.md) ## Citation For academic use, please cite the software record in [`CITATION.cff`](CITATION.cff). ## Contributing See [`CONTRIBUTING.md`](CONTRIBUTING.md). Released under the [MIT License](LICENSE).