Content
# microfiche
Store your agent's context as images. Read it back at up to 2.8x fewer tokens.
A 1092x1092 PNG page holds ~13,500 characters of prose in 10px monospace and
costs 1521 image tokens when a model reads it. The same characters cost about
4,300 text tokens. microfiche renders text into dense pages, certifies every
page with local OCR so nothing lossy ever enters your context, and gives your
agent search plus recall over the store. Claude Opus 4.8 reads these pages back
at 99.55 percent character accuracy, measured against the real API.
Named after the original optical context compression technology: libraries
stored a century of newspapers as tiny images and read them back with a lens.
## Why
Long-context agents burn most of their window and budget re-reading reference
material: docs, logs, transcripts, research dumps. Image tokens are the one
discounted input channel every frontier API exposes. Recent work (DeepSeek-OCR,
Glyph, Text or Pixels, AgentOCR) shows 2x to 4x compression is real for
rendered prose; the full literature review, including the failure modes, is in
[docs/LITERATURE.md](docs/LITERATURE.md).
microfiche is the deployable version of that result, with the safety rail the
papers say you need: a certification gate. Every rendered page is read back by
tesseract and compared to ground truth. Pages that clear the gate (prose,
logs) are recalled as images at ~2.8x compression. Pages that fail (code,
hashes, dense tables) stay as exact text automatically. You never trade
correctness for tokens without knowing it.
## Measured results
Real Claude API, prose corpus, char-level accuracy vs measured token ratio
(full grid: [bench-results/ROUNDTRIP.md](bench-results/ROUNDTRIP.md)):
| reader | font | accuracy | token ratio |
|---|---|---|---|
| Opus 4.8 | 10px | 99.55% | 2.83x |
| Opus 4.8 | 9px | 99.74% | 2.84x |
| Sonnet 5 | 12px | 99.18% | 1.89x |
Local pipeline: 7.4M chars sectionized in 0.9s, pages rendered at 2.9/s,
BM25 search at 33ms p50 over 2,184 sections.
## Install
```bash
pip install microfiche # or: pip install "microfiche[mcp]"
sudo apt-get install tesseract-ocr # or: brew install tesseract
```
## Use it from the command line
```bash
microfiche init --reader opus # 10px pages, 2% OCR gate
microfiche ingest --root .microfiche docs/*.md notes.txt
microfiche flush --root .microfiche # render + certify pages
microfiche search --root .microfiche "the thing you remember"
microfiche stats --root .microfiche # compression scoreboard
```
Recall is any tool that can read an image: point your agent at
`.microfiche/pages/<page_id>.png` and it pays 1521 tokens instead of ~4,300.
## Use it from Claude Code
Copy the skill and let the agent manage its own store:
```bash
mkdir -p ~/.claude/skills && cp -r skills/microfiche ~/.claude/skills/
```
The skill teaches the pack, search, and read-the-PNG workflow, including the
rules (never image-recall an uncertified page, quote exact strings from text).
## Use it from any MCP client (Claude Code, Codex, ...)
```bash
pip install "microfiche[mcp]"
claude mcp add microfiche -- microfiche-mcp --root .microfiche
```
Four tools: `pack` (files in, certified pages out), `search`, `recall`
(returns the page PNG as image content, or exact text when certification
failed), `stats`. Codex and other MCP clients: run `microfiche-mcp` over
stdio with `MICROFICHE_ROOT` set.
## How it works
```
text files
-> sectionizer split into canonical sections (source of truth, never deleted)
-> renderer pack sections into 1092x1092 supersampled mono pages
-> certifier tesseract reads every page back; CER <= gate or the page
is rejected and its sections stay text
-> index BM25 over section titles, keywords, gists
-> recall certified page PNG (1521 tokens) or exact section text
```
Reader profiles are calibrated against the real API: `--reader opus` renders
at 10px with a 2 percent gate (tesseract is a conservative proxy: Opus reads
what tesseract reads, plus margin). `--reader sonnet` is 12px, 1 percent.
Page cost is fixed by the patch formula ceil(w/28) x ceil(h/28); 1092x1092 is
the largest square that survives every Claude model tier undownscaled.
## The rules the research imposes
Three findings from the literature and our own measurements, baked in:
1. Prose compresses; entropy does not. Random strings, hashes, and code fail
image recall on every model tested. The OCR gate catches this class
automatically because they fail local OCR too.
2. Read back, then reason. Models retrieve well from rendered pages but
reason poorly across walls of compressed images (VTCBench). Recall pages
into the conversation and work from the transcription.
3. Density is a cliff, not a slope. Accuracy vs font size is a sharp phase
transition, so stay in the measured stable band and certify every page.
## Development
```bash
python3 -m venv .venv && .venv/bin/pip install -e ".[mcp]"
.venv/bin/python -m unittest discover -s microfiche/tests -t . # 95 tests
./run_e2e.sh # research pipeline end to end
./run_bench.sh # full local benchmark suite
```
The original research scaffolding (sweeps, fidelity grids, cost simulations,
the real-API roundtrip harness) lives in the repo root and `bench-results/`;
the research trail is in `RESEARCH.md`, `REPORT.md`, and `SCALE.md`.
## License
MIT
