starmap

# scip-code-kg Build a **queryable code knowledge graph** from [SCIP](https://github.com/scip-code/scip) indexes and serve it to AI coding agents over **MCP**. For a detailed, implementation-level description of the current architecture, schema, build pipeline, graph semantics, query algorithms, publishing protocol, and known boundaries, see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) (Chinese). The pipeline ingests compiler-grade SCIP indexes (Java via `scip-java`, C++ via `scip-clang`), materializes them into a **self-contained SQLite database per release** — symbols, precise references, a derived call graph, inheritance relations, cross-language RPC/JNI bridges, a zlib snapshot of every source file, a message-template index, and optionally per-symbol change history — and exposes ten query tools through one stdio MCP server that hosts **many versions at once** (version is a query parameter, not deployment topology). Everything an agent needs to answer *"who calls this?"*, *"what breaks if I change this?"*, *"which code produces this error line?"*, or *"how does this C++ function reach that Java implementation?"* becomes a millisecond-level indexed SQL lookup. Battle-tested on the [StarRocks](https://github.com/StarRocks/starrocks) codebase (1.3M LOC Java frontend + 0.9M LOC C++ backend + JNI extensions): **484k symbols, 6.4M occurrences, 2.55M call edges, ~250 cross-language bridge edges, 32k message templates per release**, built end-to-end in a few minutes (excluding index generation), serving a seven-release fleet from one process. ## Why SCIP instead of tree-sitter heuristics? Tree-sitter-based graph tools resolve names lexically: overloads, templates, and virtual dispatch produce guessed edges with confidence scores. SCIP indexes come out of the real compiler (`javac` plugin / Clang frontend), so: - every reference points to the **exact overload** the compiler resolved (`log(String)` vs `log(String, Throwable)` are different symbols); - interface/virtual dispatch is modeled explicitly via `is_implementation` relations, expandable on demand; - the same protobuf format covers every language, so a Java frontend and a C++ backend land in **one graph with one ID space**. ## Architecture ``` scip-java (Java FE) ─┐ scip-clang (C++ BE) ─┼─► scip_loader.py ─► kg-<ver>-<sha>.db (self-contained) scip-java (Java ext) ─┘ │ │ post-processing, all idempotent: ├─ be_kg_patch.py C++ function-body ranges via │ tree-sitter (scip-clang emits none) ├─ thrift_bridges.sql declarative Java↔C++ RPC bridges ├─ bridge_patch.py lexical thrift + JNI + brpc bridges ├─ kg_enrich.sql metrics, file/dir deps, FTS index ├─ snapshot_text.py zlib source snapshot INTO the db ├─ extract_messages.py error/log template index └─ kg_changes.py (optional) per-symbol PR history │ kg_mcp_server.py (stdio MCP, 10 tools, multi-version: KG_DIR scan) ``` The primary and default deployment is **SQLite end-to-end**: one self-contained `.db` per indexed release, zero services to operate, recursive CTEs for transitive queries. Since design v2 the db embeds its own source text (`file_text`), so the serving machine needs **no checkout, no git, no network** — the server scans `KG_DIR` for `kg*.db`, keys each by its `snapshot_meta.version_label`, and every tool takes `version=`. ## Quick start ### 1. One-time setup ```bash python3 -m venv venv && ./venv/bin/pip install -r requirements.txt ./venv/bin/python -m grpc_tools.protoc --proto_path=. --python_out=. scip.proto ``` ### 2. Generate SCIP indexes ```bash cd $REPO/fe && scip-java index # rides the real Maven build cd $REPO && scip-clang --compdb-path=be/build/compile_commands.json \ --index-output-path=index-be.scip -j $(nproc) ``` `scip-clang` needs a compilation database: configure the C++ build with `-DCMAKE_EXPORT_COMPILE_COMMANDS=ON`. ### 3. Build the graph ```bash ./build_kg.sh /path/to/repo # full pipeline, or: SKIP_INDEX=1 ./build_kg.sh /path/to/repo # reuse existing index.scip files (~2 min) ``` Output: `kg-<branch>-<sha>.db`. Every stage writes into `<final>.db.building`; `kg_publish.py` then verifies (`PRAGMA quick_check`, required tables, non-empty graph), stamps the manifest into `snapshot_meta` (`schema_version`, `artifact_id = <label>@<full_sha>`, `builder_git_sha`, `input_hashes`, `build_status=ready`) and publishes with an atomic same-directory rename. **Published artifacts are immutable**: an existing final is refused (`REPLACE=1` for a deliberate rebuild), and enrichment of a *served* db (new `kg_changes.py` ranges, an enrich re-run) goes through `tools/republish_enrich.sh` — copy, enrich, verify, atomic replace, then restart server sessions (`tools/kg_health.sh` checks the fleet). Rebuilds are full, not incremental — neither `scip-java` nor `scip-clang` supports incremental indexing (nightly CI or on-demand). ### 4. Serve it to an agent ```bash claude mcp add code-kg \ -e KG_DIR=/path/to/dir-with-kg-dbs \ -- /path/to/venv/bin/python /path/to/kg_mcp_server.py # or a single db: -e KG_DB=/path/to/kg-<ver>-<sha>.db ``` One server hosts every `kg*.db` under `KG_DIR` (lazy read-only connections; `graph_info` with no arguments lists the fleet). Artifact identity is `label@<full_commit_sha>`, so two commits of the same branch can be hosted side by side; `version=` accepts the bare label whenever it is unique, otherwise `label@shaprefix` (or a bare unique sha prefix) — an ambiguous label returns the candidates instead of guessing. Databases that fail verification (missing v3 enrich tables, unsupported `schema_version`, `build_status` ≠ ready) are skipped at startup with a reason on stderr. Every tool takes `version=` except `diff_symbol`, which takes `version_a=`/`version_b=`. The server speaks stdio, so no network is involved when the client runs on the same machine. To use it from a remote client, tunnel the same command through `ssh -T`. Do **not** expose it as an unauthenticated HTTP endpoint. ## MCP tools | Tool | Answers | |---|---| | `find_symbol` | fuzzy FTS search by name (word-split camelCase queries auto-joined); returns `{status, matches, total}` with ids, kind and signature | | `neighbors` | one symbol's direct relations — `relation=callers\|callees\|references\|implementations`; caller/callee edges **include cross-language bridges** | | `impact` | depth-bounded reverse transitive closure over calls ∪ bridges; seeds with implemented interfaces to cover virtual dispatch | | `read_definition` | full source of a function/class body, exact to the indexed commit | | `graph_info` | snapshot provenance (commit, indexers, table sizes); with `dir_prefix=` a directory-level dependency map | | `analyze_stack` | one-shot crash-stack resolution: paste raw stack text (C++/Java, OCR noise tolerated) — every recognizable frame comes back as symbol id + location + definition source + the PRs that changed it since the snapshot; runtime noise is skipped, unmatched lines are listed, nothing is silently dropped | | `trace_path` | concrete call path(s) between two symbols via bidirectional BFS over calls ∪ bridges ∪ interface dispatch (JDK functional interfaces excluded — they wormhole everything to everything; dispatch direction is validated against the type-level supertype closure); a 16-hop FE→BE path crossing the brpc bridge resolves in ~0.7 s; `detail=source` inlines each hop's call-site window and both endpoint bodies under a byte budget | | `symbol_history` | which PRs changed a symbol after the indexed snapshot — from the `kg_changes.py` enrichment: one cumulative `git diff --unified=0` whose old-side line numbers are exactly snapshot coordinates, interval-joined with `def_ranges`; per-PR attribution via `git log -L`/`-G` | | `find_message` | symptom → code entry: paste a raw error/log line (filled-in values, glog/log4j prefixes, OCR noise all fine); an embedded `file.cpp:123` resolves directly, otherwise normalized template matching over the `messages` index returns every direct producer with the origin function and inline source; unresolvable dynamic text is flagged `partial`/`miss`, never silently empty | | `diff_symbol` | cross-release structural answer to "is that bug still present in X": resolves a symbol on `version_a`, matches its counterpart on `version_b` by package-version-independent SCIP descriptor (artifact versions embedded in FE symbol strings change across releases), and returns a unified body diff — or `identical:true`, or an honest not-found with closest candidates | The surface is deliberately small (ten tools) and the outputs deliberately terse — compact JSON, qualified short names instead of full SCIP symbol strings, small default limits, native batching (up to 10 comma-separated ids / semicolon-separated queries per call). Every byte a tool returns is re-sent to the model on each later turn of the session, so output size dominates the real cost of a graph tool; query latency (~1 ms) never does. The pre-merge tool names (`find_callers`, `find_references`, …) still work in CLI mode as aliases. Guardrails built in: expected outcomes are success-shaped `status` payloads (`ok|partial|ambiguous|not_found|invalid_argument|unsupported`) rather than transport errors, so an agent never sees a wall of isError; ambiguous names return structured candidates with IDs instead of guessing among overloads; list responses carry `returned/total/truncated`; `read_definition` enforces a ~48KB response budget at JSON-item granularity; external and location-less symbols (stdlib, dependency jars) are de-prioritized; bridge edges are never starved out of `neighbors` by call-edge limits; test files are excluded from path traversal and impact; every call is audit-logged (`mcp_calls.jsonl`) with version, latency, output size, and answer-quality signals for post-hoc analysis. > **Tip:** published evaluations consistently show agents underuse graph tools unless prompted. Add a line to your project instructions such as: *"For call-graph, impact, or cross-language questions, prefer the `code-kg` tools over grep."* ## How the call graph is derived SCIP has no native "calls" relation. Edges are materialized at build time with one interval join: a reference occurrence that falls inside a definition's enclosing range is a call from that definition — attributed **innermost-wins**: when body ranges nest (a method-local class inside a test helper, a local functor inside a function), only the smallest containing body is the caller; equal-span ties are all kept, because C++ template instantiations share one source body under N symbols and each must keep its out-edges or paths break at the instantiation boundary. ```sql INSERT INTO calls (caller_id, callee_id, file_id, line) SELECT DISTINCT caller_id, callee_id, file_id, line FROM ( SELECT dr.symbol_id AS caller_id, o.symbol_id AS callee_id, o.file_id AS file_id, o.start_line AS line, RANK() OVER (PARTITION BY o.rowid ORDER BY dr.end_line - dr.start_line) AS rn FROM occurrences o JOIN def_ranges dr ON dr.file_id = o.file_id AND o.start_line BETWEEN dr.start_line AND dr.end_line JOIN symbols callee ON callee.id = o.symbol_id AND callee.symbol LIKE '%).' JOIN symbols caller ON caller.id = dr.symbol_id AND caller.symbol LIKE '%).' WHERE (o.roles & 1) = 0 AND dr.symbol_id <> o.symbol_id ) WHERE rn = 1; ``` `def_ranges` comes from SCIP `enclosing_range` where available (scip-java), and from tree-sitter function-definition nodes where it is not (scip-clang) — syntactically exact either way. Query-time lookups are then plain index scans; transitive queries (`impact`) are recursive CTEs. A related subtlety the queries must handle: SCIP emits method-override relationship rows in **both directions with identical flags**, so "which side is the interface" is recovered through the type-level (`is_reference = 0`) supertype closure of the parent classes — that keeps an implementation's interface out of its `implemented_by` list, keeps implementations listed in an interface's impact, and stops paths from hopping impl → iface → sibling-impl. ## Cross-language bridges Java and C++ never reference each other's symbols, so the graph would naturally split into disconnected components. The `bridges` table (`src`, `dst`, `kind`, `evidence`) reconnects them: - **`thrift`** — declarative: both sides' generated code descends from the same IDL, so `(service, method)` name equality joins Java `Svc#Client#m()` to C++ `SvcIf#m()` and vice versa. Only emitted when the target side has a real (non-generated, non-test) implementation. - **`brpc`** — declarative: the modern FE→BE execution path (protobuf RPC) is invisible to both indexes — FE uses jprotobuf annotation stubs, BE implements `PInternalServiceImplBase`/`LakeServiceImpl`. Joined by method name: FE camelCase minus the `Async` suffix equals BE snake_case (`execPlanFragmentAsync` ↔ `exec_plan_fragment`). - **`thrift-lex`** — lexical: scip-clang emits no reference for member calls through `ClientConnection<T>::operator->` (the dominant RPC idiom), so a scanner matches `client->method(` sites, validates the method against the service's interface table, and attributes the call to the enclosing function. - **`jni`** — lexical: Java class-name string literals in C++ (`FindClass`-style) are linked to the corresponding Java class symbols. The thrift/JNI scanners target StarRocks RPC idioms specifically, but the pattern — declarative name-join over generated stubs plus a narrow lexical pass for idioms the indexer misses — transfers to any polyglot codebase. Query-side, traversals simply use `calls UNION ALL bridges`. ## Database schema Core tables (written by `scip_loader.py`): | Table | Contents | |---|---| | `meta` | indexer names/versions, project root, **indexed commit SHA** | | `files` | path, language | | `symbols` | SCIP symbol string, display name, kind, parent (containment), definition site, docs, signature, `is_external` | | `occurrences` | every definition/reference with exact line/col and SCIP role bitmask | | `def_ranges` | full body range of each definition (the call-graph backbone) | | `relationships` | `is_implementation` / `is_reference` / `is_type_definition` / `is_definition` (UNIQUE-indexed; deduped at ingest) | | `calls` | derived caller → callee edges with call-site location | | `bridges` | cross-language edges with kind and evidence | Derived tables (`kg_enrich.sql`, rebuilt in ~10 s): `symbol_metrics` (fan-in/fan-out/ref counts), `file_deps`, `dir_deps`, `symbols_fts` (FTS5 name search). Self-containment and enrichment tables: | Table | Written by | Contents | |---|---|---| | `file_text` | `snapshot_text.py` | per-file zlib blob of the source at build time + resolved repo-relative path (`read_definition`/`analyze_stack`/`find_message` read this, never the filesystem) | | `snapshot_meta` | `snapshot_text.py` | `version_label` (the registry key), snapshot timestamp | | `messages` (+`messages_fts`) | `extract_messages.py` | every log/error/exception template with kind, sink, severity, origin function; named constants expanded through `occurrences` | | `symbol_changes` | `kg_changes.py` | per-symbol PR attribution over a `<kg-commit>..<newer-release>` range (one endpoint must be the indexed commit; standing forward ranges appended per release) | ## Known limitations & design notes 1. **Full rebuild only.** No SCIP indexer supports incremental updates; the per-commit-artifact model embraces that. 2. **scip-clang ≤ 0.4.0 gaps**, both patched here: no `enclosing_range` (upstream issue #323 closed *not planned*; PR #504 unmerged) → tree-sitter patch; missing references for `operator->` member calls → lexical bridge pass. 3. **Both SCIP range encodings are supported** — legacy `repeated int32` and the newer `typed_range`/`typed_enclosing_range` oneofs (typed wins) — because current scip-java emits only the typed form. 4. **`SELECT DISTINCT` in the calls join is load-bearing**: scip-clang emits duplicate definition occurrences that would otherwise fabricate ~25% duplicate edges. Relationship rows are additionally deduped at ingest and UNIQUE-indexed so a regression fails the build instead of duplicating silently. 5. **SCIP `local N` symbols are skipped everywhere** — they are document-scoped per spec; interning them globally merges unrelated anonymous classes into chimera nodes (`tools/purge_local_symbols.py` retrofits older dbs). 6. **Static over-approximation by design.** Callers of an interface method are reported as potential callers of every implementation — the right bias for impact analysis. 7. **Lexical bridges and message extraction are heuristic.** Every bridge row carries an `evidence` string, every message row its sink; unresolvable dynamic messages surface as `partial`/`miss`, never as fabricated origins. 8. `probe_index.py` / `probe_bridge.py` sanity-check any new index before a build: enclosing-range coverage, symbol scheme, relationship counts, bridge anchors. ## Optional: analytical warehouse export For team-wide analytics (dead-code sweeps, dependency dashboards) the same graph can additionally be exported to an OLAP warehouse: `kg_ddl.sql` + `export_kg.py` + `load_sr.sh` load it into a StarRocks cluster via Stream Load. This is strictly optional and secondary to the SQLite artifact — note that most OLAP engines lack recursive CTEs, so transitive queries there require bounded self-joins or client-side loops. ## Repository layout ``` build_kg.sh orchestrator: index → load → patch → bridge → enrich → snapshot → messages → publish kg_publish.py verify (quick_check/tables/counts) + manifest stamp + atomic publish scip_loader.py SCIP protobuf → SQLite (streaming mmap parser, plain ANSI INSERTs) be_kg_patch.py C++ def_ranges via tree-sitter + calls re-materialization thrift_bridges.sql declarative thrift bridges bridge_patch.py lexical thrift + JNI + brpc bridges kg_enrich.sql metrics / file_deps / dir_deps / FTS snapshot_text.py zlib source snapshot into the db (self-containment) extract_messages.py log/error/exception template index (find_message backend) kg_changes.py per-symbol PR history over <kg-commit>..<release> (symbol_history backend) kg_mcp_server.py stdio MCP server (10 tools, multi-version via KG_DIR) test_kg_mcp.py protocol-level smoke test (drives the server as a real MCP client) probe_index.py new-index compatibility probe probe_bridge.py bridge-anchor probe tools/ build_version_kg.sh (historical-release one-shot), republish_enrich.sh (enrich a SERVED db: copy → enrich → verify → atomic replace), kg_health.sh (post-publish fleet check), gen_compile_commands.sh, purge_local_symbols.py + rematerialize_calls.py (older-db retrofits), kg-ab eval/ A/B eval harness: cases/, run_eval.sh, score.py, anatomy.py skills/code-kg/ agent-facing skill card (routing + cost discipline) docs/ design notes (v2 self-contained multi-version) kg_ddl.sql optional warehouse DDL export_kg.py optional warehouse export (TSV + JSONL) load_sr.sh optional warehouse Stream Load scip.proto vendored from scip-code/scip (Apache-2.0), pinned for parse stability ``` ## Acknowledgments - [SCIP](https://github.com/scip-code/scip) and its indexers ([scip-java](https://github.com/scip-code/scip-java), [scip-clang](https://github.com/sourcegraph/scip-clang)) — Apache-2.0. - Call-graph derivation via enclosing-range joins follows the approach used by Meta's Glean and other SCIP consumers.