Content
# DevRAG
A local RAG system for developer teams that ingests **code**, **GitHub PRs**, **GitHub issues**, **Jira Cloud tickets**, **Slite pages**, **Slack conversations**, **documents**, and **Claude Code session logs** — surfaced through Claude Code slash commands and a standalone CLI. Designed for zero-friction adoption: index your repo, search immediately.
## Why DevRAG?
Most code search tools treat source files as plain text. DevRAG uses **tree-sitter AST parsing** to chunk code along semantic boundaries (functions, classes, methods), so searches return meaningful units instead of fragments split mid-function.
It combines **vector search** (semantic) with **BM25** (keyword) via Qdrant's server-side Reciprocal Rank Fusion, which is critical for code where exact identifiers like `refreshToken` matter as much as the concept of "token refresh." Both signals live in Qdrant as named vectors on the same points (`dense` + `bm25`), so search is a single round trip per collection and metadata filters apply equally to both legs.
PR history, issues, Jira tickets, and Slite wiki pages are first-class data sources — encoding *why* code changed, what bugs and features are tracked, what project management context exists, and what internal knowledge base content says.
## Quick Start
### Install
Install directly from GitHub (no clone required):
```bash
# As a global CLI tool (recommended)
uv tool install git+https://github.com/tomharris/dev-rag.git
# Or run without installing
uvx --from git+https://github.com/tomharris/dev-rag.git devrag --help
# Or add as a project dependency
uv add git+https://github.com/tomharris/dev-rag.git
# pip install git+https://github.com/tomharris/dev-rag.git
```
For development (from source):
```bash
git clone https://github.com/tomharris/dev-rag.git
cd dev-rag
uv sync
uv run devrag --help
```
### Index and Search
```bash
# Index the current repository
devrag index repo .
# Search across code, PRs, issues, and docs
devrag search "how does authentication work"
# Check what's indexed
devrag status
```
### Requirements
- Python 3.12+
- [Ollama](https://ollama.com/) running locally with `nomic-embed-text` pulled:
```bash
ollama pull nomic-embed-text
```
### Models on a network that blocks Hugging Face
DevRAG's reranker and BM25 models come from Hugging Face. On first use it
auto-downloads a self-hosted bundle of both models from a dev-rag GitHub release
(reachable even when huggingface.co is blocked) and caches them locally; later
runs load offline. To fetch them explicitly:
```bash
devrag download-models # fetch/refresh the model bundle
devrag download-models --force # re-download even if cached
```
Overrides (in `~/.config/devrag/devrag.yaml` or `.devrag.yaml`):
```yaml
network:
auto_download_models: true # set false to disable auto-fetch (CI/air-gapped)
model_bundle_url: "" # internal/air-gapped mirror of the bundle
model_bundle_sha256: "" # expected checksum for a custom model_bundle_url
sparse_embedding:
cache_dir: "" # FastEmbed cache; "" = ~/.cache/devrag/fastembed
```
Note: the bundle carries the **default** model names. If you override
`retrieval.reranker_model` or `sparse_embedding.model`, you need Hugging Face (or
a faithful mirror) access for your chosen model.
## CLI Reference
### Search
```bash
devrag search "query" # Search all sources
devrag search "query" --scope code # Code only
devrag search "query" --scope prs # PR history only
devrag search "query" --scope issues # Issues only
devrag search "query" --scope slite # Slite pages only
devrag search "query" --scope slack # Slack conversations only
devrag search "query" --scope docs # Documents only
devrag search "query" --scope sessions # Claude Code session logs only
devrag search "query" --top-k 10 # More results
# Narrow with metadata filters (all optional, AND-combined)
devrag search "auth" --repo myrepo --chunk-type diff
devrag search "bug" --pr-number 1234
devrag search "refund flow" --ticket-key PROJ-123
devrag search "deploy" --session-id <uuid>
devrag search "incident" --channel C0123ABCD # Slack channel id
```
The query router automatically classifies intent and targets relevant collections. "Why did we switch to Redis?" routes to PR history; "is there a bug with login?" routes to issues and Jira tickets; "what sprint is auth in?" routes to Jira; "what does our wiki say about deploys?" routes to Slite pages and documents; "how does the auth middleware work?" routes to code.
### Indexing
```bash
# Code (AST-aware, incremental by default; also indexes the repo's docs)
devrag index repo . # Current directory (code + docs)
devrag index repo /path/to/repo # Specific path
devrag index repo . --full # Force full re-index
devrag index repo . --name my-repo # Name for multi-repo support
devrag index repo . --no-docs # Code only, skip the repo's docs
devrag index remove-repo my-repo # Remove a repo from the index (code + docs)
devrag index refresh # Re-index ALL registered repos (incremental; code + docs)
devrag index refresh --full # Full rebuild of every registered repo (external sources untouched)
# `index repo` also indexes the repo's own docs (md/mdx/txt/rst/html/adoc) into
# the documents collection, tagged with the repo name (so `search --repo <name>`
# narrows to them). Use `index docs` below only for a standalone docs tree.
# Documents (section-aware splitting; standalone directory, not tied to a repo)
devrag index docs ./specs # Default: **/*.md
devrag index docs ./docs --glob "**/*.txt,**/*.rst"
# GitHub PRs (incremental with cursor tracking)
export GITHUB_TOKEN=ghp_xxx
devrag index prs owner/repo # Incremental from cursor (90d on first run)
devrag index prs owner/repo --since 180d # Override cursor for backfill
# GitHub Issues (incremental, skips PRs)
devrag index issues owner/repo # Last 90 days
devrag index issues owner/repo --since 30d # Custom lookback
# Jira Cloud tickets (incremental via JQL)
export JIRA_EMAIL=you@company.com
export JIRA_TOKEN=your-api-token
devrag index jira # Uses JQL from .devrag.yaml
devrag index jira --since 180d # Custom lookback
# Slite pages (incremental, section-aware chunking)
export SLITE_TOKEN=your-api-token
devrag index slite # Last 90 days, all channels
devrag index slite --since 30d # Custom lookback
# Slack conversations — NO Slack App required (uses your browser session)
export SLACK_XOXC_TOKEN=xoxc-... # web-client token (see "Slack without an app")
export SLACK_XOXD_COOKIE=xoxd-... # the `d` cookie value
devrag index slack # Public channels you're in, last 90 days
devrag index slack --since 30d # Custom first-sync lookback
# Claude Code session logs (local JSONL, incremental via file mtime)
devrag index sessions # Incremental from cursor (60d first run)
devrag index sessions --since 30d # Override cursor
devrag index sessions --logs-dir /custom/path
```
#### Slack without an app
Most Slack integrations require creating a Slack App and having a workspace admin
install it with OAuth scopes. DevRAG skips that entirely: it authenticates as **you**,
the logged-in user, using the same credentials your browser already holds — so no app,
no admin approval, and it indexes exactly what you can already see.
**Get the credentials automatically** (recommended; you just need to be logged in to
the Slack desktop app or a browser):
```bash
eval "$(devrag auth slack --workspace mycorp)" # mycorp = the <x> in https://<x>.slack.com
devrag index slack
```
`devrag auth slack` reads both credentials from the **Slack desktop app by default** —
the `d` cookie from its cookie store and the `xoxc` token from its localStorage (where
Slack now keeps the token; it's no longer in any web page) — validates the pair, and
prints the two `export` lines, so `eval "$(…)"` sets `SLACK_XOXC_TOKEN` /
`SLACK_XOXD_COOKIE` in your shell. The desktop app is a Chromium app, so its cookie store
decrypts with the same machinery used for browsers (no extra setup) and its localStorage
is read with a bundled pure-python LevelDB reader — both work even while Slack is running
(the stores are copied first; nothing is written to disk by devrag). Set `slack.workspace`
in `.devrag.yaml` to skip `--workspace` on re-runs.
`--browser` selects the source explicitly: `slack` (the desktop app) or
`chrome`/`firefox`/`brave`/`edge`/`chromium`. With no `--browser`, devrag tries the Slack
desktop app first, then sweeps every supported browser, skipping any whose profile it
can't read — so an unsupported/missing app or browser doesn't abort the search. The cookie
must live **on the machine running the command** — if you're logged in only on a different
machine, or your app/browser stores its profile in a non-standard location, use `--browser`
or the manual fallback below.
<details>
<summary><b>Manual fallback</b> — if cookie decryption fails (locked keyring / Chrome
app-bound encryption)</summary>
Extract the two credentials from your browser (while logged in to Slack on the web):
1. Open Slack in your browser → DevTools (F12) → **Console**.
2. Token (`xoxc-…`): run
`JSON.parse(localStorage.localConfig_v2).teams[Object.keys(JSON.parse(localStorage.localConfig_v2).teams)[0]].token`
(or find `xoxc-` under Application → Local Storage).
3. Cookie (`xoxd-…`): DevTools → **Application → Cookies → `https://app.slack.com`** → copy
the value of the cookie named **`d`** (URL-decode `%2F`→`/` etc. if needed).
4. `export SLACK_XOXC_TOKEN=xoxc-…` and `export SLACK_XOXD_COOKIE=xoxd-…`, then
`devrag index slack`.
</details>
**Scope:** public channels you belong to by default. To index only specific channels
(of any type), set an allowlist in `.devrag.yaml` under `slack.channel_ids`. Threads are
indexed as single chunks; loose channel chatter is grouped into time-window chunks.
> ⚠️ **Caveats.** These are personal session credentials, not an official API token, and
> using Slack's internal web API is against a strict reading of Slack's Terms of Service —
> use at your own risk on workspaces where you have permission to read the content. The
> token/cookie **expire** when your browser session rotates or you log out; when that
> happens `devrag index slack` fails with a clear "re-extract from browser" error rather
> than silently indexing nothing. Credentials are read from env vars only and never stored
> in config files.
Code indexing is **incremental** — file content hashes are tracked in SQLite, so unchanged files are skipped on re-index.
### Other Commands
```bash
devrag status # Show index stats
devrag serve # Start MCP server for Claude Code
devrag reindex --all # Clear all indexes and re-embed code repos + their docs (re-sync PRs/issues/Jira/Slite/Slack manually)
devrag reindex --name my_project # Re-index a single repo's code + docs (preserves other repos and external sources)
devrag config set embedding.model nomic-embed-text
devrag config get vector_store.backend
```
### Evaluation
Test retrieval quality with a JSONL file of queries and expected results:
```bash
devrag eval run test_queries.jsonl --output results.jsonl --top-k 5
devrag eval compare results_v1.jsonl results_v2.jsonl
```
Query file format:
```jsonl
{"query": "How does rate limiting work?", "expected_files": ["src/middleware/rate_limit.ts"]}
{"query": "Why did we migrate to Redis?", "expected_prs": [1234, 1301]}
```
Metrics: precision@k, recall@k, and MRR (Mean Reciprocal Rank).
## Claude Code Integration
DevRAG ships as a [FastMCP](https://github.com/jlowin/fastmcp) server that integrates directly with Claude Code.
### Setup
```bash
# If installed globally via `uv tool install`:
claude mcp add devrag -- devrag serve
# Or run on demand without installing:
claude mcp add devrag -- uvx --from git+https://github.com/tomharris/dev-rag.git devrag serve
```
### Skills
The repo includes Claude Code skills in `.claude/skills/`:
| Skill | Description |
|-------|-------------|
| `/rag-search <query>` | Search code, PRs, and docs with results grouped by source type |
| `/rag-index [path]` | Index a repo, docs directory, or sync PRs |
| `/rag-pr <query>` | Search PR history for why code changed |
| `rag-first` (auto) | Auto-triggers on codebase questions to search DevRAG before Grep/Glob/Explore |
The `rag-first` skill fires automatically when Claude detects a codebase question — no slash command needed.
For a hard safety net, the repo ships `.claude/hooks/rag_first_gate.py` — a per-turn block gate that **denies** the first `Grep`, `Glob`, or `Agent`(`Explore`/`general-purpose`) call of a turn until `mcp__devrag__search` has run, then releases for the rest of the turn. The script is tracked, but hook registration is per-user. To enable it, add this `hooks` block to your `.claude/settings.local.json` (alongside `permissions`):
```jsonc
"hooks": {
"PreToolUse": [
{
"matcher": "Grep|Glob|Agent",
"hooks": [{
"type": "command",
"command": "python3 ${CLAUDE_PROJECT_DIR}/.claude/hooks/rag_first_gate.py",
"timeout": 5
}]
}
],
"PostToolUse": [
{
"matcher": "mcp__devrag__search",
"hooks": [{
"type": "command",
"command": "python3 ${CLAUDE_PROJECT_DIR}/.claude/hooks/rag_first_gate.py",
"timeout": 5
}]
}
],
"UserPromptSubmit": [
{
"matcher": "",
"hooks": [{
"type": "command",
"command": "python3 ${CLAUDE_PROJECT_DIR}/.claude/hooks/rag_first_gate.py",
"timeout": 5
}]
}
]
}
```
Restart your Claude Code session after editing so the hooks register.
## Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ DEVELOPER INTERFACES │
│ CLI (Typer) MCP Server (FastMCP) │
│ devrag search|index|status Claude Code slash commands │
└────────────┬──────────────────────────────────┬─────────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ RETRIEVAL LAYER │
│ Query Router ──→ Hybrid Search (server-side dense+BM25 RRF) ──→ Reranker│
│ (intent → collections) (cross-encoder) │
│ └─→ repo-preference boost ──→ dedup per source ──→ final_k │
└────────────┬────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ STORAGE LAYER │
│ QdrantStore (HTTP or embedded local path) │
│ points carry named vectors: {dense, bm25 sparse} │
│ SQLite: file hashes, chunk mappings, sync cursors, metrics │
└────────────┬────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ INGESTION LAYER │
│ Code PR Issue Jira Slite Doc │
│ Indexer Indexer Indexer Indexer Indexer Indexer │
│ (AST) (GitHub) (GitHub) (Cloud) (REST) (sections) │
│ │
│ Ollama Embedder (nomic-embed-text, 768d) │
└─────────────────────────────────────────────────────────────────────┘
```
### Collections
| Collection | Source | Contents |
|-----------|--------|----------|
| `code_chunks` | Code indexer | AST-extracted functions, classes, methods |
| `pr_diffs` | PR indexer | PR descriptions and file-level diff hunks |
| `pr_discussions` | PR indexer | Review comments and threads |
| `issue_descriptions` | Issue indexer | Issue titles and bodies |
| `issue_discussions` | Issue indexer | Issue comments |
| `jira_descriptions` | Jira indexer | Jira ticket summaries and descriptions |
| `jira_discussions` | Jira indexer | Jira ticket comments |
| `slite_pages` | Slite indexer | Slite page sections (markdown) |
| `slack_messages` | Slack indexer | Slack threads and time-window conversations |
| `documents` | Doc indexer | Markdown/text sections |
| `session_logs` | Sessions indexer | Claude Code user↔assistant exchanges from local JSONL |
## Configuration
DevRAG uses YAML configuration with two layers (project overrides user):
- **User config:** `~/.config/devrag/devrag.yaml`
- **Project config:** `.devrag.yaml` (commit this to share with your team)
```yaml
embedding:
model: nomic-embed-text # Embedding model name
provider: ollama # ollama or sentence-transformers
ollama_url: http://localhost:11434
batch_size: 64
max_tokens: 8192 # Truncation limit for embedding context
vector_store:
qdrant_url: http://localhost:6333 # Qdrant HTTP endpoint
qdrant_path: "" # optional: local filesystem path for embedded mode (no server needed)
embedding_dim: 768 # Vector dimensionality
retrieval:
top_k: 20 # Candidates before reranking
final_k: 5 # Results after reranking (dedup runs on the full pool first)
rerank: true
reranker_model: cross-encoder/ms-marco-MiniLM-L-6-v2
reranker_max_length: 512 # Cross-encoder token limit; raise for longer chunks
max_per_source: 2 # Max results kept per file/PR/issue/etc.
repo_boost: 0.15 # Soft preference for the cwd repo (fraction of score spread; 0 = off)
code:
chunk_max_tokens: 512
respect_gitignore: true
exclude_patterns:
- "*.min.js"
- "vendor/**"
- "node_modules/**"
- "*.lock"
- "*.generated.*"
prs:
github_token_env: GITHUB_TOKEN # Env var name (token never stored in config)
backfill_days: 90
include_draft: false
chunk_max_tokens: 512 # Max tokens per PR chunk
issues:
github_token_env: GITHUB_TOKEN # Reuses same token as PRs
backfill_days: 90
chunk_max_tokens: 512
include_labels: [] # Only index issues with these labels (empty = all)
exclude_labels: ["wontfix"] # Skip issues with these labels
jira:
jira_token_env: JIRA_TOKEN # Env var for API token
jira_email_env: JIRA_EMAIL # Env var for email (basic auth)
instance_url: https://yoursite.atlassian.net
jql: "project = DEV" # JQL filter for scoping tickets
backfill_days: 90
chunk_max_tokens: 512
slite:
slite_token_env: SLITE_TOKEN # Env var for API token
channel_ids: [] # Filter to specific channels (empty = all)
backfill_days: 90
chunk_max_tokens: 512
chunk_overlap_tokens: 50
slack:
slack_token_env: SLACK_XOXC_TOKEN # Env var for the xoxc web-client token
slack_cookie_env: SLACK_XOXD_COOKIE # Env var for the xoxd `d` cookie
channel_ids: [] # Allowlist of channel ids (empty = all public channels you're in)
gap_minutes: 30 # Quiet gap that starts a new conversation window
backfill_days: 90
chunk_max_tokens: 512
chunk_overlap_tokens: 50
sessions:
logs_dir: ~/.claude/projects # Claude Code JSONL log directory
backfill_days: 60
chunk_max_tokens: 512
documents:
glob_patterns: ["**/*.md", "**/*.mdx", "**/*.txt", "**/*.rst", "**/*.html", "**/*.adoc"]
chunk_max_tokens: 512
chunk_overlap_tokens: 50
network:
ca_bundle: "" # PEM CA bundle for all outbound HTTPS (Slack/Jira/Slite/GitHub)
```
### Corporate proxies (TLS interception)
If you're behind a TLS-intercepting corporate proxy, `devrag auth slack`, `index prs/issues/jira/slite/slack`, and their MCP equivalents may fail with:
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain
```
`httpx` verifies against the bundled `certifi` CA list and does **not** consult the macOS keychain or honor `SSL_CERT_FILE` on its own, so the proxy's injected root CA is unknown to it. Point DevRAG at a PEM bundle containing that root CA, either via config:
```yaml
network:
ca_bundle: ~/corp-ca.pem
```
or via the standard env vars (honored as a fallback when `ca_bundle` is empty):
```bash
export REQUESTS_CA_BUNDLE=~/corp-ca.pem # or SSL_CERT_FILE
```
To export your proxy's root CA from the macOS keychain to a PEM:
```bash
security find-certificate -a -p -c "<your proxy CA name>" \
/Library/Keychains/System.keychain > ~/corp-ca.pem
```
(or in Keychain Access → select the root CA → File → Export Items → `.pem`). The Ollama embedder talks to localhost and is unaffected.
### File Exclusion
DevRAG respects `.gitignore` by default. For additional exclusions specific to the index, create a `.devragignore` file. It uses full `.gitignore` syntax — directory patterns (`docs/internal/`), anchoring, `**`, and `!` negation all work — and applies to both code and docs indexing. Unlike `.gitignore`, it can also exclude files that are tracked by git.
## Deployment Modes
DevRAG uses Qdrant as its vector store. You can run it either embedded (local filesystem, no server needed) or against a Qdrant server.
**Embedded mode** (single-user, no server):
```yaml
vector_store:
qdrant_path: ~/.local/share/devrag/qdrant
```
**Server mode** (shared, scales horizontally):
```bash
docker run -d -p 6333:6333 -v qdrant_data:/qdrant/storage qdrant/qdrant
```
```yaml
vector_store:
qdrant_url: http://localhost:6333
```
Switching modes is a config change; re-index with `devrag reindex --all` after switching.
## Development
```bash
git clone https://github.com/tomharris/dev-rag.git
cd dev-rag
uv sync --extra dev
# Run tests
uv run pytest tests/
# Run a single test
uv run pytest tests/test_config.py::test_load_config -v
# Integration tests (requires Ollama running with nomic-embed-text)
SKIP_INTEGRATION=0 uv run pytest tests/test_integration.py -v
```
## License
MIT
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available