Content
# Kalkan
**Kalkan** (Turkish for *shield*) is an integrity layer between an AI agent and your document: the agent can only *propose* structured patches — a deterministic engine validates them, applies them atomically, snapshots every version, and rolls back byte-exactly without a single model call.
*Türkçe sürüm: [README.tr.md](README.tr.md)*
## The problem
In April 2026, Microsoft Research published [**"LLMs Corrupt Your Documents When You Delegate"**](https://arxiv.org/abs/2604.15597) (Philippe Laban, Tobias Schnabel, Jennifer Neville), introducing the **DELEGATE-52** benchmark: 19 LLMs editing documents across 52 professional domains over sessions of 10 edit/revert round-trips. Even frontier models corrupted about a quarter of document content by the end of long delegated sessions — and the damage is often silent: the agent reports success while the file quietly loses rows, wording, and structure. Kalkan attacks the mechanism behind that failure mode: it removes the model from the write path entirely.
## Results
Measured with our DELEGATE-style harness — 6 documents × 10 edit+revert pairs (20 interactions each) under three conditions, Claude Sonnet via `claude -p`. Full tables: `eval/RAPOR-sonnet-k2.md`.
<!-- Placeholders are filled from eval/RAPOR-sonnet-k2.md once run k2 finishes; see lansman/placeholder-envanteri.md for the exact cell mapping. -->
| Condition | Final similarity (avg, 1 = intact) | Untouched-region damage % (avg) | Allowed-region residual drift % (avg) | Catastrophic events |
|---|---|---|---|---|
| **A1 — full rewrite** (paste doc, ask for edited copy) | 0.7381 | 30.38 | 25.61 | 4 |
| **A2 — plain file tools** (`read_lines` + `str_replace`, no validation) | 0.9946 | 0.00 | 5.98 | 0 |
| **B — Kalkan** (validated patches + versioned rollback) | 1.0000 | 0.00 | 0.00 | 0 |
Honesty note: plain file tools (A2) performed well most of the time, but with no guarantee — its worst run ended at 0.9823 final similarity, and the *allowed-region residual drift* metric exists precisely to catch what its failed reverts leave behind inside the edited regions.
## How it works
Four pieces, all deterministic:
1. **MCP tool set** — six tools over stdio (`open_document`, `read_section`, `propose_patch`, `list_versions`, `revert`, `get_report`). Reading is windowed (max 200 lines per call, deliberately no full-document dump); `propose_patch` is the *only* write path.
2. **Patch engine** — pure TypeScript, no model, no filesystem. Every operation (`replace` / `insert_after` / `delete_section`) must anchor on text that occurs **exactly once**; ambiguous or missing anchors are rejected with the closest matching passage and its line number so the agent can self-correct. Overlapping operations are rejected by name; a single patch may not touch more than 40% of the document (blast-radius cap). A patch list is all-or-nothing.
3. **Version store** — a hidden `.kalkan/` folder next to the document keeps a full copy of every version plus a manifest (id, timestamp, operation summary, agent rationale). No git dependency. `revert` restores a version **byte-exactly as a file operation** — the model never regenerates "restored" content.
4. **Diff & session report** — every rejection carries an actionable reason; `get_report` summarizes patches, rejections, changed line ranges, and version count, so you can audit what the agent actually did.
```mermaid
flowchart LR
A[AI agent] -->|MCP / stdio| T[Kalkan tool set]
T --> P[Patch engine<br/>exactly-once anchors, overlap check,<br/>blast-radius cap, atomic apply]
P -->|validated write| D[(document.md)]
P -->|snapshot each version| V[(.kalkan/ version store)]
V -->|revert: byte-exact restore,<br/>no model call| D
T --> R[Diff / session report]
```
## Quickstart
```bash
npm install
npm run build # compiles to dist/
npm test # vitest suite
```
### Claude Desktop
Add to `claude_desktop_config.json` (adjust the path):
```json
{
"mcpServers": {
"kalkan": {
"command": "node",
"args": ["C:/path/to/kalkan/dist/index.js"]
}
}
}
```
### Claude Code
```bash
claude mcp add kalkan -- node /path/to/kalkan/dist/index.js
```
Or project-scoped in `.mcp.json`:
```json
{
"mcpServers": {
"kalkan": {
"command": "node",
"args": ["./dist/index.js"]
}
}
}
```
### 3-minute walkthrough
A sample document ships in `ornek/ornek-belge.md`. Try this conversation with your agent:
1. **Open** — "Open `ornek/ornek-belge.md` with kalkan." → metadata arrives; `.kalkan/` now holds `v0`.
2. **Patch** — "In the İletişim section, change the contact sentence to mention e-mail, with a rationale." → applied deterministically, `v1` saved.
3. **Fail on purpose** — "Replace the sentence 'this text does not exist'." → rejected atomically; the error suggests the closest real passage; the file is byte-identical to before.
4. **Roll back** — "Roll back to v0." → the document is byte-identical to `v0`; the restore itself is recorded as `v2`.
5. **Audit** — "Get the session report." → 1 patch, 1 rejection with its reason, 3 versions.
## Measurement
Full methodology: [`eval/METHODOLOGY.md`](eval/METHODOLOGY.md). In short: every condition runs the **same documents, same instructions, same order** — 10 edit+revert pairs per document, each interaction a fresh conversation. **A1** pastes the document and asks for a full edited copy back (the unprotected baseline). **A2** gives the model unvalidated `read_lines`/`str_replace` file tools — today's typical agent setup. **B** gives it Kalkan's tools, where reverts are version restores instead of model output. Metrics are line-LCS based and fully deterministic: final similarity to the original, damage outside the regions the scenario was allowed to touch, residual drift *inside* allowed regions (failed reverts), and a catastrophe counter (>10% of the document changed in a single interaction).
Reproduce it yourself:
```bash
npm run eval:pilot # 1 document × 3 rounds × 3 conditions, asks confirmation first
npm run eval:full # full run: 6 documents × 10 rounds × 3 conditions
npm run eval:report # regenerates eval/RAPOR-<model>[-<label>].md from result JSONs
```
The harness runs over your installed Claude Code subscription (`claude -p`, no API key) or the Anthropic API (`eval/config.json` → `saglayici`). Results land in `eval/sonuclar/` as raw JSON — including every model response verbatim — so every number in the report can be audited. If you run it with a different model or provider, we'd love to see your `RAPOR-*.md` (see [CONTRIBUTING.md](CONTRIBUTING.md)).
## Known limits & roadmap
- **Whole-document patches are capped, not scoped** — a single operation touching >40% of the document is rejected, but finer *blast-radius enforcement* (per-section scopes) is v2.
- **No semantic invariants yet** — Kalkan protects structure and untouched regions; rules like "totals must stay consistent" are v2.
- **Markdown / plain text only** — no DOCX, no spreadsheets, no code-aware patching.
- **Single-model measurement** — published numbers are one model on six Turkish documents; the harness supports re-running with other models, and cross-model results are exactly the contribution we're hoping for.
- v2 ordering will follow community feedback — open an issue if one of these blocks you.
## Contributing & license
See [CONTRIBUTING.md](CONTRIBUTING.md) — bug reports, patch-engine edge cases, and *especially* your own measurement runs are welcome. Licensed under [MIT](LICENSE).
