axon

# Axon Version: 7.1.5 Axon is a self-hosted RAG stack for crawling, scraping, ingesting, embedding, searching, and asking questions over indexed content. The production release is Docker Compose first: one Axon server container, Qdrant, Hugging Face TEI with `Qwen/Qwen3-Embedding-0.6B`, and Chrome for JS-heavy pages. ## Current Runtime vs Future Source Pipeline This README documents the current production runtime. The clean-break pipeline unification target is tracked in [`docs/pipeline-unification/`](docs/pipeline-unification/README.md) and [GitHub issue #298](https://github.com/jmagar/axon/issues/298). That target moves source acquisition/indexing to `axon <source>`, routes CLI/MCP/REST through shared `SourceRequest` DTOs, and removes legacy public aliases rather than preserving backwards compatibility. ## Production Contract Supported production runtime: - Docker Compose only. - Qdrant only for vector storage. - Hugging Face TEI only for embeddings. - `Qwen/Qwen3-Embedding-0.6B` as the production embedding model. - Gemini CLI is the default LLM synthesis path; OpenAI-compatible endpoints such as llama.cpp are supported when configured with `AXON_LLM_BACKEND=openai-compat`. - Web search/research uses a self-hosted SearXNG instance (`AXON_SEARXNG_URL`) when configured, falling back to Tavily (`TAVILY_API_KEY`) otherwise. - Local NVIDIA RTX 4070 target with NVIDIA Container Toolkit. - CLI and MCP run all actions in-process; deploy the `axon serve` container only when you need HTTP API access. - One shared config home: `~/.axon/.env`, `~/.axon/config.toml`, `~/.axon/jobs.db`, `~/.axon/output`, `~/.axon/logs`, `~/.axon/artifacts`, `~/.axon/screenshots`, and `~/.axon/tei`. Qdrant storage lives on the host running Qdrant; dookie points at tootie by default. Not supported in the production path: - systemd deployment of the Axon binary. - Postgres, Redis, RabbitMQ, AMQP, or external worker services. - OpenAI-compatible first-run LLM configuration. Configure `AXON_LLM_BACKEND=openai-compat` manually after setup when using llama.cpp or another OpenAI-compatible `/v1/chat/completions` endpoint. - Neo4j or graph retrieval. - Multiple competing `.env` or `config.toml` locations. ## Install ### Linux Prerequisites: - Linux x86_64. - Docker and Docker Compose. - NVIDIA driver, `nvidia-smi`, and NVIDIA Container Toolkit. - Gemini CLI installed and already authenticated, unless using a configured OpenAI-compatible endpoint for LLM synthesis. - `curl`, `sha256sum`, and `install`. One-line installer: ```bash curl -fsSL https://raw.githubusercontent.com/jmagar/axon/main/install.sh | sh ``` The installer verifies the release checksum, installs the `axon` binary to `~/.local/bin`, then delegates setup to `axon setup`. Useful installer controls: ```bash AXON_INSTALL_DRY_RUN=1 ./install.sh AXON_INSTALL_PREFIX=/opt/axon ./install.sh AXON_VERSION=vX.Y.Z ./install.sh # pin a specific release; defaults to latest AXON_INSTALL_SKIP_SETUP=1 ./install.sh ``` ### Windows Prerequisites: - Windows x86_64 (PowerShell 5.1+ or PowerShell Core). - Docker Desktop with the WSL 2 backend (for the full stack). - NVIDIA driver and NVIDIA Container Toolkit (for GPU acceleration). - Gemini CLI installed and already authenticated, or a configured OpenAI-compatible endpoint for LLM synthesis. One-line installer (PowerShell): ```powershell irm https://raw.githubusercontent.com/jmagar/axon/main/install.ps1 | iex ``` The installer verifies the release checksum, installs `axon.exe` to `%USERPROFILE%\.local\bin`, adds it to your user PATH, then delegates setup to `axon setup`. Useful installer controls: ```powershell $env:AXON_INSTALL_DRY_RUN='1'; irm .../install.ps1 | iex $env:AXON_INSTALL_PREFIX='C:\tools\axon'; irm .../install.ps1 | iex $env:AXON_VERSION='vX.Y.Z'; irm .../install.ps1 | iex # pin a version $env:AXON_INSTALL_SKIP_SETUP='1'; irm .../install.ps1 | iex ``` Claude Code plugin install: ```bash claude plugin install <path-to-this-repo> ``` The plugin ships no binary. Install `axon` first via `install.sh`, then install the plugin. Its `SessionStart` hook runs `scripts/plugin-setup.sh`, which syncs `CLAUDE_PLUGIN_OPTION_*` settings into process env and delegates to `axon setup plugin-hook`. That subcommand is **probe-only and never deploys**: it checks `/readyz` and exits silently when the stack is up, or prints a one-line `run /axon-deploy` advisory when it is down. `ConfigChange` runs the same script so updated plugin settings take effect immediately. Provisioning is the `/axon-deploy` slash command (or `axon setup` / `axon compose up`). No systemd unit is created. ## Setup Flow `axon setup` is the convenience bootstrap path. It is idempotent and safe to rerun. It: 1. Creates or refreshes `~/.axon`. 2. Creates or preserves `~/.axon/config.toml`. 3. Creates or preserves `~/.axon/.env`, filling only missing runtime values and preserving secrets. 4. Writes Docker Compose assets under `~/.axon/compose`. 5. Checks Docker, Docker Compose, `nvidia-smi`, Gemini CLI auth, and OAuth config when requested. 6. Pulls and starts the Compose stack. 7. Waits for Qdrant, TEI, Chrome, and Axon server health. Focused commands: ```bash axon setup # init + compose up + preflight axon setup init # create ~/.axon, config.toml, .env, and compose assets axon preflight # check prerequisites, auth config, and service readiness axon compose up # pull/start services, then follow logs until Ctrl-C axon compose down # stop services axon compose restart # restart services axon compose rebuild # rebuild the Axon image and start services axon smoke # TEI prewarm + crawl/ask proof axon setup plugin-hook # probe-only SessionStart path (never deploys; advises /axon-deploy when down) axon setup targets # list SSH aliases discovered from ~/.ssh/config (informational) ``` For local bearer-token operation, no manual env values are required. `setup init` defaults to loopback MCP HTTP, writes `AXON_AUTH_MODE=bearer`, and generates `AXON_HTTP_TOKEN`. Optional features need credentials: Gemini auth under `~/.gemini` for default LLM features or `AXON_LLM_BACKEND=openai-compat` plus `AXON_OPENAI_BASE_URL` and `AXON_SYNTHESIS_OPENAI_MODEL` for OpenAI-compatible synthesis, `TAVILY_API_KEY` for search/research, `GITHUB_TOKEN` for higher-rate GitHub ingest, and `REDDIT_CLIENT_ID` plus `REDDIT_CLIENT_SECRET` for Reddit ingest. OAuth mode also requires `AXON_PUBLIC_URL`, `AXON_GOOGLE_CLIENT_ID`, `AXON_GOOGLE_CLIENT_SECRET`, and `AXON_AUTH_ADMIN_EMAIL`. The warm-path setup goal is under 2 minutes once images and model weights are cached. Cold starts that pull images and model weights can take longer; target-hardware timing still needs to be measured against published release artifacts. ## Local Indexing Use `axon embed /path` to register local files with a background local code-index watcher. Existing local file and directory inputs start watching by default; use `axon embed /path --watch` to keep the watcher attached in the foreground, or `axon embed /path --no-watch` for a one-shot local embed that exits after writing vectors. SourceLedger currently tracks generation and cleanup state for crawl refreshes and mutable Git ingest sources in the same SQLite runtime DB as Axon jobs. The removed `code-search-watch` command now exits as a tombstone; use `embed` for local Git checkout/workspace code-indexing. The watcher uses the existing `axon-code-index` lifecycle tables. ## Docker Stack The production compose file starts: | Service | Purpose | Host bind | | --- | --- | --- | | `axon` | HTTP server, web panel, MCP HTTP, action API, in-process workers | `127.0.0.1:8001` | | `axon-qdrant` | vector storage, optional local service; production points at tootie by default | `127.0.0.1:53333`, `127.0.0.1:53334` | | `axon-tei` | Qwen3 embeddings through TEI | `127.0.0.1:52000` | | `axon-chrome` | browser rendering and CDP proxy | `127.0.0.1:6000`, `127.0.0.1:9222`, `127.0.0.1:9223` | Start manually: ```bash docker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml up -d ``` Check: ```bash docker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml ps axon preflight axon doctor ``` Stop: ```bash docker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml down ``` Development stack: ```bash cargo build --bin axon docker compose --env-file .env.example -f docker-compose.yaml build axon docker compose --env-file ~/.axon/.env -f docker-compose.yaml up -d axon ``` The development stack uses the production infrastructure definitions but runs `axon` from the bind-mounted local debug binary under `target/debug`, inside the newer `axon:dev-runtime` image. ## Configuration Axon has two config layers: | File | Purpose | | --- | --- | | `~/.axon/.env` | URLs, secrets, auth, Docker interpolation, runtime bootstrap values | | `~/.axon/config.toml` | non-secret tuning and behavior | Precedence: ```text CLI flags > environment variables > ~/.axon/config.toml > built-in defaults ``` Keep in `.env`: - URLs: `QDRANT_URL`, `TEI_URL`, `AXON_CHROME_REMOTE_URL`, `AXON_SEARXNG_URL`. - Secrets: `AXON_HTTP_TOKEN`, `TAVILY_API_KEY`, `GITHUB_TOKEN`, `GITLAB_TOKEN`, `GITEA_TOKEN`, Reddit credentials, OAuth credentials, `HF_TOKEN`. - Docker/runtime bootstrap: `AXON_HOME`, `AXON_DATA_DIR`, `AXON_IMAGE`, `AXON_HTTP_PUBLISH`, `TEI_HTTP_PORT`, GPU device values. - LLM runtime pointers when needed: `AXON_HEADLESS_GEMINI_CMD`, `AXON_HEADLESS_GEMINI_HOME`, `AXON_LLM_BACKEND`, `AXON_OPENAI_BASE_URL`, `AXON_SYNTHESIS_OPENAI_MODEL` (legacy alias: `AXON_OPENAI_MODEL`), and optional `AXON_OPENAI_API_KEY`. Put in `config.toml`: - collection/search/ask tuning. - worker and job limits. - TEI client tuning. - Qdrant batch sizing. - logging behavior that is not a process-launch concern. - UI/output behavior. `~/.axon/.env` under the config home is never loaded through a symlink. ## First Run After setup: ```bash axon doctor axon crawl https://example.com --wait true axon ask "What did we crawl?" ``` CLI and MCP commands always run in-process against Qdrant and TEI. `axon serve` exposes the same operations over HTTP (`/v1/*`, MCP-over-HTTP) for clients that want API access to a deployed instance. ## Notable Capabilities - **Hybrid search.** New Qdrant collections are created with named `dense` + `bm42` sparse vectors and queried with Reciprocal Rank Fusion (RRF). Legacy unnamed collections fall back to dense-only cosine search. Tune via the `[search]` section in `config.toml`; run `axon migrate --from <old> --to <new>` to copy a legacy unnamed collection into a new named-mode one, then point `AXON_COLLECTION` at it. - **Normalized source planning.** Crawl manifests, scrape results, ingest sources, local files, and memory records are normalized into `SourceDocument` before chunking. The source-doc planner is the only layer that chooses markdown/plain-text/file chunking and emits `PreparedDoc` values for TEI/Qdrant. File origins (`GitFile`, `LocalFile`) get tree-sitter-aware code chunks where supported plus `code_*`, `symbol_*`, `chunk_locator`, `source_range`, and `chunk_content_kind` payload metadata; unsupported code falls back to prose chunks with fallback metadata instead of losing enrichment. - **Vertical extractors.** `scrape` (and the `scrape` MCP/REST action) auto-routes known URLs to structured per-site extractors (GitHub, PyPI, npm, crates.io, Reddit, YouTube, and more) instead of generic HTML→markdown. Vertical payloads keep `extractor_name` and bounded structured data through the same source-doc planner used by generic scrape. Disable with `AXON_ENABLE_VERTICALS=false` or the `[verticals]` config section. - **Freshness schedules.** The CLI can create recurring schedules with `--fresh <Nd>` on `scrape`, `crawl`, `embed`, and `ingest`. `axon serve`/`axon mcp` run the scheduler from SQLite, replaying safe versioned snapshots through the normal service/job paths. Freshness management is CLI-only in v1: use `axon fresh list`, `axon fresh run-now <id>`, and `axon fresh history <id>`. - **Web panel.** `axon serve` hosts an Aurora-styled control panel at the bind address (default `http://127.0.0.1:8001`) with a first-run setup flow, config/stack inspection, and a command runner, alongside the `/v1/*` REST surface and OpenAPI docs at `/docs`. ## CLI Map Core: - `scrape <url>...` - `crawl <url>...` - `map <url>` - `extract <url>...` - `embed [input]` - `query <text>` - `retrieve <url>` - `ask <question>` - `summarize <url>...` - `evaluate <question>` - `diff <url-a> <url-b>` — show what changed between two URLs (content/metadata/links) - `brand <url>` — extract brand identity: colors, fonts, logos, favicon Discovery and ingest: - `search <query>` — web search via SearXNG (or Tavily), auto-queues crawl jobs - `research <query>` — multi-source web research with LLM synthesis - `suggest [focus]` - `endpoints <url>` — discover API endpoints from page HTML and JavaScript bundles - `ingest <target>` — GitHub, GitLab, Gitea/Forgejo, generic Git, Reddit, or YouTube - `sessions` — index AI session exports (Claude, Codex, Gemini) Operations: - `setup` - `update` — install the latest GitHub Release binary and sync the local container - `doctor` - `debug` - `serve` - `mcp` - `status` - `monitor jobs` — stream job lifecycle events (start/complete/fail/cancel) - `fresh` — list, run, and inspect CLI-created freshness schedules - `sources` - `domains` - `stats` - `watch` - `refresh [filter]` — re-crawl/re-ingest previously indexed origins (full docs refresh) - `dedupe` - `migrate` - `screenshot` - `train` — collect human preference votes on retrieved RAG candidates - `config` - `completions` Use command-specific help: ```bash axon --help axon setup --help axon crawl --help axon mcp --help ``` Graph flags are not part of the production CLI, MCP, or `/v1/ask` request contract. ## MCP And Auth Axon exposes one MCP tool named `axon`; actions are routed by `action` and optional `subaction`. Examples: ```json { "action": "doctor" } { "action": "scrape", "url": "https://example.com" } { "action": "ask", "query": "How does setup work?" } { "action": "summarize", "url": "https://example.com" } { "action": "crawl", "subaction": "status", "job_id": "<uuid>" } ``` HTTP auth modes: - Static bearer token with `AXON_HTTP_TOKEN`. - OAuth/lab-auth with `AXON_AUTH_MODE=oauth`. `/mcp` and the `/v1/*` REST routes (e.g. `/v1/ask`, `/v1/scrape`, `/v1/query`) share the same auth policy. Tokenless HTTP is only for loopback development binds. ## Development Build: ```bash cargo build --bin axon cargo build --release --bin axon ``` Test and lint: ```bash cargo fmt --all -- --check cargo check --workspace --all-targets --features test-helpers cargo clippy --workspace --all-targets --features test-helpers -- -D warnings cargo test --workspace --features test-helpers ``` Common focused checks: ```bash cargo test --test cli_help_contract -- --nocapture cargo test parse_setup -- --nocapture cargo test load_dotenv -- --nocapture python3 scripts/generate_mcp_schema_doc.py --check docker compose --env-file .env.example -f docker-compose.prod.yaml config --quiet ``` Module layout policy: - Do not add `mod.rs`. - Rust module roots live in `foo.rs`. - Submodules live in `foo/bar.rs`. ## Release Gates Required before production release: - CI fmt/check/clippy/test. - MCP schema doc sync. - CLI help contract tests. - Docker Compose config validation. - Docker image build and GHCR publish workflow. - Compose smoke workflow. - Self-hosted RTX 4070 smoke for Qwen3 TEI cold/warm setup timing. `cargo machete` should be run when installed; it is not vendored in this repo. ## Troubleshooting Fast checks: ```bash axon preflight axon doctor docker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml ps docker compose --env-file ~/.axon/.env -f ~/.axon/compose/docker-compose.yaml logs --tail=100 axon axon-tei axon-qdrant axon-chrome ``` Important paths: - `~/.axon/.env` - `~/.axon/config.toml` - `~/.axon/jobs.db` - `~/.axon/logs` - `~/.axon/artifacts` - `~/.axon/output` - `~/.axon/tei` - `~/.axon/qdrant` on the host running Qdrant, normally tootie Common failures: - Docker missing: install Docker and Docker Compose, then rerun `axon setup`. - GPU unavailable: verify `nvidia-smi` and NVIDIA Container Toolkit. - Gemini unauthenticated: run Gemini CLI login outside Axon, then rerun setup, or configure `AXON_LLM_BACKEND=openai-compat` for an OpenAI-compatible endpoint. - TEI slow on first boot: model download/cache warmup is the cold path. - Auth failures: make sure Claude/plugin config uses the same token as `AXON_HTTP_TOKEN` in `~/.axon/.env`. ## Related Files - `install.sh` — verified one-line installer bootstrapper. - `docker-compose.prod.yaml` — production Compose stack. - `docker-compose.yaml` — local development stack. - `.env.example` — production environment template. - `config.example.toml` — non-secret tuning template. - `plugins/axon/.claude-plugin/plugin.json` — Claude plugin manifest. - `scripts/plugin-setup.sh` — plugin hook delegating to shared setup. - `docs/reference/mcp/tool-schema.md` — generated MCP wire contract. - `docs/` — full documentation tree: guides, `reference/actions/`, architecture, and operations. ## Related Servers - [soma](https://github.com/jmagar/soma) - RMCP runtime for provider-backed MCP servers. - [unifi-rmcp](https://github.com/jmagar/unifi-rmcp) - UniFi controller REST API bridge. - [tailscale-rmcp](https://github.com/jmagar/tailscale-rmcp) - Tailscale API bridge for devices, users, and tailnet operations. - [unraid-rmcp](https://github.com/jmagar/unraid-rmcp) - Unraid GraphQL bridge for NAS and server management. - [apprise-rmcp](https://github.com/jmagar/apprise-rmcp) - Apprise notification fan-out bridge for many delivery backends. - [gotify-rmcp](https://github.com/jmagar/gotify-rmcp) - Gotify push notification bridge for sends, messages, apps, and clients. - [arcane-rmcp](https://github.com/jmagar/arcane-rmcp) - Arcane Docker management bridge for containers and related resources. - [yarr](https://github.com/jmagar/yarr) - Media-stack bridge for Sonarr, Radarr, Prowlarr, Plex, and related services. - [ytdl-rmcp](https://github.com/jmagar/ytdl-rmcp) - Media download and metadata workflow server. - [synapse-rmcp](https://github.com/jmagar/synapse-rmcp) - Local Synapse workflow server for scout and flux actions. - [cortex](https://github.com/jmagar/cortex) - Syslog and homelab log aggregation MCP server. - [labby](https://github.com/jmagar/labby) - Homelab control plane and MCP gateway project. - [lumen](https://github.com/jmagar/lumen) - Local semantic code search MCP server. ## License MIT