rpg-encoder

<h1 align="center">rpg-encoder</h1> <p align="center"> <strong>Give your AI agent a brain for your codebase.</strong> </p> <p align="center"> <a href="https://github.com/userFRM/rpg-encoder/actions"><img src="https://github.com/userFRM/rpg-encoder/workflows/CI/badge.svg" alt="CI"></a> <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square" alt="MIT License"></a> <a href="https://www.rust-lang.org"><img src="https://img.shields.io/badge/rust-1.85%2B-orange.svg?style=flat-square" alt="Rust 1.85+"></a> <a href="https://www.npmjs.com/package/rpg-encoder"><img src="https://img.shields.io/npm/v/rpg-encoder?style=flat-square" alt="npm"></a> <a href="https://modelcontextprotocol.io/"><img src="https://img.shields.io/badge/MCP-compatible-green.svg?style=flat-square" alt="MCP"></a> <a href="https://github.com/userFRM/rpg-encoder/stargazers"><img src="https://img.shields.io/github/stars/userFRM/rpg-encoder?style=flat-square" alt="Stars"></a> </p> <br> AI coding agents waste most of their tool calls fumbling through your codebase with `grep`, `cat`, `find`, and file reads. `rpg-encoder` fixes that. It builds a **semantic graph** of your code with [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) — not just *what calls what*, but *what every function does* — and gives your AI assistant whole-repo understanding via [MCP](https://modelcontextprotocol.io/) in a single tool call. <p align="center"> <img src="diagrams/hero-tool-waste.webp" alt="Without RPG: 34,000 chaotic grep/cat/find calls. With RPG: one semantic_snapshot call returns a structured map of the whole repo." width="90%" /> </p> --- ## Quick Start ```bash claude mcp add rpg -- npx -y -p rpg-encoder rpg-mcp-server ``` One command. Works with Claude Code, Cursor, opencode, Windsurf, or any MCP-compatible agent. No Rust toolchain, no cloning, no building — `npx` downloads a pre-built binary for your platform. Then open any repo and tell your agent: > *"Build and lift the RPG for this repo"* Your agent handles everything: indexes entities (seconds), reads each function and adds intent-level features (a few minutes), organizes them into a semantic hierarchy, and commits `.rpg/graph.json` for your team. For repos with ~100+ entities, `lifting_status` will tell your agent to delegate the lifting loop to a sub-agent or a cheaper model — feature extraction is pattern-matching, not novel reasoning. If your runtime has no sub-agent mechanism, run `rpg-encoder lift --provider anthropic|openai` from the terminal with an API key — the CLI drives an external LLM directly with no agent involvement. After the CLI finishes, call `reload_rpg` in your session to load the updated graph. The CLI lifts entities with no features; re-lifting stale entities (features present but outdated after code changes) is handled by the in-session MCP flow, not the CLI. Once lifted, try: - *"What handles authentication?"* — finds code even when nothing is named "auth" - *"Show everything that depends on the database connection"* - *"Plan a change to add rate limiting to API endpoints"* --- ## Use RPG before `grep`, `cat`, `find` The server instructions tell your agent to reach for RPG tools FIRST for any question about code structure or behavior. That reflex matters — `grep`, `cat`, and ad-hoc file reads burn tokens and miss semantic relationships RPG already knows. | If you'd otherwise reach for... | Use this instead | |---|---| | `grep -r` / `rg` (by intent) | `search_node(query="...")` | | `grep -r` / `rg` (by name) | `search_node(query="...", mode="snippets")` | | `cat` / reading a function | `fetch_node(entity_id="file:name")` | | chained greps for callers/callees | `explore_rpg(entity_id="...", direction="...")` | | recursive grep for "what depends on X" | `impact_radius(entity_id="...")` | | `wc -l` / `find` / `tree` | `rpg_info` | | reading many files for context | `semantic_snapshot` | | manual search → fetch → explore chains | `context_pack(query="...")` | | "how do I refactor X safely" | `plan_change(goal="...")` | Fall back to `grep`, `cat`, or file reads only when the query is about literal text (string search, comments, TODOs, log messages) — not about structure. --- ## How It Works <p align="center"> <img src="diagrams/how-it-works.webp" alt="Four-stage pipeline: Parse (tree-sitter) → Lift (verb-object features) → Organize (3-level hierarchy) → Understand (LLM gets full repo knowledge)" width="95%" /> </p> 1. **Parse** — Tree-sitter extracts entities (functions, classes, methods) and dependency edges (imports, calls, inheritance) from 15 languages. 2. **Lift** — An LLM (your agent, or a cheap API like Haiku) reads each entity and writes verb-object features: *"validate JWT tokens"*, *"serialize config to disk"*. 3. **Organize** — Features cluster into a 3-level semantic hierarchy (Area → Category → Subcategory) that emerges from *what the code does*, not the file tree. 4. **Understand** — `semantic_snapshot` compresses the whole graph into ~25K tokens. Your LLM reads it once and *knows the repo*. ### The semantic snapshot <p align="center"> <img src="diagrams/semantic-snapshot.webp" alt="The whole repo — ~500K tokens of source — compressed 20x into a ~25K token snapshot containing hierarchy, features, dependencies, and hot spots" width="80%" /> </p> Instead of grepping through files, the LLM calls `semantic_snapshot` once and receives: - **Hierarchy** — every functional area with aggregate features - **Entities** — every function, class, method grouped by area, with its semantic features - **Dependency skeleton** — condensed call graph with qualified names - **Hot spots** — top 10 most-connected entities (the architectural backbone) ~25K tokens covers ~1000 entities. That's 2-3% of a 1M context window — the LLM starts every session already knowing your repo. ### Self-maintaining graph <p align="center"> <img src="diagrams/auto-staleness.webp" alt="Git HEAD moves → RPG Server auto-syncs → update_rpg applies additions/modifications/removals → graph always fresh, zero agent action" width="80%" /> </p> Whenever your working tree changes — committed, staged, or unstaged — the MCP server automatically re-syncs before responding to the next query. A changeset hash over `(path, size, mtime)` means repeated saves of the same file trigger one sync, and idle queries trigger none. Reverts are detected too: if a previously-dirty file returns to its HEAD state, the graph is restored. ### Two ways to lift | Mode | Command | Cost | Who pays | |------|---------|------|----------| | **Agent lifting** | *"Build and lift the RPG"* | Subscription tokens | Your Claude Code / Cursor subscription | | **Autonomous lifting** | `auto_lift(provider="anthropic", api_key_env="ANTHROPIC_API_KEY")` | ~$0.02 per 100 entities | External API key (Haiku, GPT-4o-mini, OpenRouter, Gemini) | `auto_lift` calls a cheap external LLM directly — your coding subscription never touches the lifting work. Use `api_key_env` to resolve keys from environment variables so they never appear in tool call transcripts. --- ## Architecture <p align="center"> <img src="diagrams/architecture.webp" alt="Your codebase (15 languages) → RPG Engine (5 Rust crates: parser, encoder, nav, lift, mcp) → Clients (Claude Code, Cursor, opencode) via MCP Protocol" width="95%" /> </p> Seven Rust crates, one MCP server binary, one CLI binary: | Crate | Role | |-------|------| | `rpg-core` | Graph types (RPGraph, Entity, HierarchyNode), storage, LCA algorithm | | `rpg-parser` | Tree-sitter entity + dependency extraction (15 languages) | | `rpg-encoder` | Encoding pipeline, lifting utilities, incremental evolution | | `rpg-nav` | Search, fetch, explore, snapshot, TOON serialization | | `rpg-lift` | Autonomous LLM lifting (Anthropic, OpenAI, OpenRouter, Gemini) | | `rpg-cli` | CLI binary (`rpg-encoder`) | | `rpg-mcp` | MCP server binary (`rpg-mcp-server`) with 27 tools | --- ## MCP Tools (27) <details> <summary><strong>Build & Maintain</strong> (4 tools)</summary> | Tool | Description | |------|-------------| | `build_rpg` | Index the codebase (run once, instant) | | `update_rpg` | Incremental update from git changes | | `reload_rpg` | Reload graph from disk after external changes | | `rpg_info` | Graph statistics, hierarchy overview, per-area lifting coverage | </details> <details> <summary><strong>Navigate & Search</strong> (5 tools)</summary> | Tool | Description | |------|-------------| | `semantic_snapshot` | Whole-repo semantic understanding in one call (~25K tokens for 1000 entities) | | `search_node` | Search entities by intent or keywords (hybrid embedding + lexical scoring) | | `fetch_node` | Get entity metadata, source code, dependencies, and hierarchy context | | `explore_rpg` | Traverse dependency graph (upstream, downstream, or both) | | `context_pack` | Single-call search + fetch + explore with token budget | </details> <details> <summary><strong>Plan & Analyze</strong> (7 tools)</summary> | Tool | Description | |------|-------------| | `impact_radius` | BFS reachability analysis — "what depends on X?" | | `plan_change` | Change planning — find relevant entities, modification order, blast radius | | `find_paths` | K-shortest dependency paths between two entities | | `slice_between` | Extract minimal connecting subgraph between entities | | `analyze_health` | Code health: coupling, instability, god objects, clone detection | | `detect_cycles` | Find circular dependencies and architectural cycles | | `reconstruct_plan` | Dependency-safe reconstruction execution plan | </details> <details> <summary><strong>Semantic Lifting</strong> (11 tools)</summary> | Tool | Description | |------|-------------| | `auto_lift` | One-call autonomous lifting via cheap LLM API (Haiku, GPT-4o-mini, OpenRouter, Gemini) | | `lifting_status` | Dashboard — coverage, per-area progress, NEXT STEP | | `get_entities_for_lifting` | Get entity source code for your agent to analyze | | `submit_lift_results` | Submit the agent's semantic features back to the graph | | `finalize_lifting` | Aggregate file-level features, rebuild hierarchy metadata | | `get_files_for_synthesis` | Get file-level entity features for holistic synthesis | | `submit_file_syntheses` | Submit holistic file-level summaries | | `build_semantic_hierarchy` | Get domain discovery + hierarchy assignment prompts | | `submit_hierarchy` | Apply hierarchy assignments to the graph | | `get_routing_candidates` | Get entities needing semantic routing (drifted or newly lifted) | | `submit_routing_decisions` | Submit routing decisions (hierarchy path or "keep") | </details> --- ## Supported Languages 15 languages via Tree-sitter: | Language | Entity Extraction | Dependency Resolution | |----------|------------------|----------------------| | Python | Functions, classes, methods | imports, calls, inheritance | | Rust | Functions, structs, traits, impl methods | use, calls, trait impls | | TypeScript | Functions, classes, methods, interfaces | imports, calls, inheritance | | JavaScript | Functions, classes, methods | imports, calls, inheritance | | Go | Functions, structs, methods, interfaces | imports, calls | | Java | Classes, methods, interfaces | imports, calls, inheritance | | C / C++ | Functions, classes, methods, structs | includes, calls, inheritance | | C# | Classes, methods, interfaces | using, calls, inheritance | | PHP | Functions, classes, methods | use, calls, inheritance | | Ruby | Classes, methods, modules | require, calls, inheritance | | Kotlin | Functions, classes, methods | imports, calls, inheritance | | Swift | Functions, classes, structs, protocols | imports, calls, inheritance | | Scala | Functions, classes, objects, traits | imports, calls, inheritance | | Bash | Functions | source, calls | --- ## Install ### MCP server (recommended) ```bash # Claude Code claude mcp add rpg -- npx -y -p rpg-encoder rpg-mcp-server # Cursor — add to ~/.cursor/mcp.json { "mcpServers": { "rpg": { "command": "npx", "args": ["-y", "-p", "rpg-encoder", "rpg-mcp-server"] } } } ``` The server auto-detects the project root from the current working directory — no path argument needed. <details> <summary><strong>CLI</strong></summary> ```bash npm install -g rpg-encoder # Build a graph rpg-encoder build # Query rpg-encoder search "parse entities from source code" rpg-encoder fetch "src/parser.rs:extract_entities" rpg-encoder explore "src/parser.rs:extract_entities" --direction both --depth 2 rpg-encoder info # Autonomous lifting via API rpg-encoder lift --provider anthropic --dry-run # estimate cost rpg-encoder lift --provider anthropic # lift with Haiku (~$0.02/100 entities) # Incremental update rpg-encoder update # Pre-commit hook (auto-updates graph on commit) rpg-encoder hook install ``` </details> <details> <summary><strong>Build from source</strong></summary> ```bash git clone https://github.com/userFRM/rpg-encoder.git cd rpg-encoder && cargo build --release ``` Then point your MCP config at `target/release/rpg-mcp-server`. </details> --- ## Documentation - [How RPG Compares](docs/comparison.md) — honest comparison with GitNexus, Serena, Repomix, and others - [Paper Fidelity](docs/paper_fidelity.md) — algorithm-by-algorithm comparison with the research paper - [Use Cases](use_cases.md) — practical examples of what RPG enables - [CHANGELOG](CHANGELOG.md) — release history --- ## Inspirations & References rpg-encoder is built on the theoretical framework from the RPG-Encoder research paper, with original extensions inspired by tools across the code intelligence landscape: - **[RPG-Encoder paper](https://arxiv.org/abs/2602.02084)** (Luo et al., 2026, Microsoft Research) — semantic lifting model, 3-level hierarchy construction, incremental evolution algorithms, formal graph model `G = (V_H ∪ V_L, E_dep ∪ E_feature)`. - **[GitNexus](https://github.com/abhigyanpatwari/GitNexus)** — precomputed relational intelligence, blast radius analysis, Claude Code hooks. Showed that a code graph tool must be invisible to be essential. - **[Serena](https://github.com/oraios/serena)** — symbol-level precision via LSP. Demonstrated that real-time code awareness matters more than batch analysis. - **[TOON](https://github.com/toon-format/toon)** — Token-Oriented Object Notation for LLM-optimized output. This is an independent implementation. All code is original work under the MIT license. Not affiliated with or endorsed by Microsoft. --- ## License [MIT](LICENSE)