Recall

# Recall A multi-user personal knowledge base. Feed it notes, snippets, and URLs; retrieve them by meaning, not just keyword, via hybrid semantic + full-text search (pgvector + Postgres FTS, fused with Reciprocal Rank Fusion). Retrieval is exposed over the **Model Context Protocol (MCP)** so an MCP-capable LLM client (Claude Desktop, Claude Code, Cursor) can query the knowledge base mid-conversation through one hosted server — no per-client integration code, no local DB credentials. **Live:** | Surface | URL | |---|---| | Landing page | [tryrecall.net](https://tryrecall.net) | | Web app (sign in with GitHub/Google) | [app.tryrecall.net](https://app.tryrecall.net) | | API + MCP (`/mcp`) | `https://api.tryrecall.net` (TLS via ACM at the ALB) | ## Project status **The original 8-week roadmap is complete and live, plus a post-roadmap phase: TLS, multi-user OAuth, a marketing landing page, and a full app redesign.** Working end-to-end: - Notes CRUD (incl. `PATCH` that re-chunks + re-embeds) with cursor pagination - Chunk-level Voyage embeddings + hybrid semantic + keyword search fused with RRF - URL ingestion (`POST /ingest/url`: fetch + main-content extraction) and bulk Markdown import (`POST /ingest/markdown`, frontmatter-aware, atomic batch) - **Multi-user auth**: OAuth sign-in (GitHub/Google via Auth.js) on the Vercel BFF, which mints a short-lived **Ed25519 (EdDSA) JWT** per request; the API verifies it (algorithm pinned, iss/aud checked), get-or-creates the `users` row, and every data query is owner-scoped. The service token stays as a second, outer gate. A malformed signing key fails the BFF at boot with a named cause (`web/instrumentation.ts`). - MCP server (`search_notes` / `add_note` / `get_related`) over Streamable HTTP at `/mcp`, **per-user**: each user mints revocable **personal access tokens** in the web app (stored as SHA-256 hashes, plaintext shown once), and every request resolves its token to the owning user before any tool runs - **CLI** (`recall`) on the same PATs: `recall login` stores the token (masked prompt, `~/.config/recall/config.toml`, `RECALL_API_TOKEN`/`RECALL_API_URL` env overrides), then `add`, `add --url`, `search`, `list`, `import`, `delete` against the live API — the PAT is the sole credential, so there is no shared secret to distribute - Production polish: owner-scoped Redis embedding + query caches (fail-open; ~200× faster warm search), tenacity retries on Voyage, structured JSON logs with per-request correlation ids, Redis-backed per-IP rate limiting - **Live on AWS** (us-east-2): one ECS Fargate ARM64 task behind an ALB with **ACM TLS** at `api.tryrecall.net`, RDS Postgres 16 + pgvector, Redis as an in-task sidecar, secrets in Secrets Manager, CloudWatch logs. GitHub Actions builds and rolls the service on every push to `main` via OIDC (no long-lived AWS keys). - **Web app** (`web/`): Next.js (App Router) client behind a **backend-for-frontend proxy** — the browser never sees the service token or the signing key. Redesigned with a warm light/dark theme, a custom sign-in page, and an ambient "synapse" background that reacts to note activity. The **Connect** page onboards both non-browser surfaces with a guided walkthrough (auto-opens once, reopenable via "?"; MCP and CLI tracks over one token; platform-aware commands; keyboard-navigable stepper). Deploys to Vercel. - **Landing page** (`landing/`): a separate static Next.js app at the apex domain with an interactive product demo; its own Vercel project, so marketing deploys can never break the app. - **Tests:** 99 unit + 55 DB-backed integration tests (testcontainers pgvector; includes cross-user isolation proven on the REST — JWT and PAT paths — and MCP surfaces, identity get-or-create, and cache-scoping suites). Both gate the deploy. **Known gaps / next steps** (details in [SPEC.md §10](SPEC.md#10-whats-next)): 1. **OAuth on `/mcp`** — Claude Desktop / claude.ai custom connectors want OAuth rather than a static bearer; the token-resolution seam is structured so an OAuth verifier is an additive branch. 2. UI for what the API already does: note editing (`PATCH`), bulk Markdown import, and surfacing `get_related`. 3. Per-user (not per-IP) rate limiting; reranking (Voyage `rerank-2.5`); a retrieval-quality eval set; Terraform. The authoritative documents are [SPEC.md](SPEC.md) (architecture, data model, API, deployment, next steps) and [RESEARCH.md](RESEARCH.md) (every dependency, why it was chosen, what it beat). This README is a doorway; depth lives there. ## Architecture ```mermaid graph TD subgraph Clients LLM["LLM clients<br/>(Claude Desktop, Claude Code, Cursor)"] Browser["Browser"] end subgraph "Vercel" Landing["Landing page<br/>(tryrecall.net)"] BFF["Web app + BFF proxy<br/>(app.tryrecall.net)<br/>OAuth via Auth.js, mints EdDSA JWT"] end subgraph "Recall service (one Fargate task, api.tryrecall.net)" MCP["MCP server<br/>(Streamable HTTP, /mcp)"] REST["REST API<br/>(FastAPI, /notes /search)"] Core["Core service layer<br/>chunk · embed · hybrid search · persist<br/>(every query owner-scoped)"] MCP --> Core REST --> Core end subgraph "Managed state" PG[("Postgres 16<br/>+ pgvector + tsvector<br/>(RDS)")] Redis[("Redis<br/>owner-scoped caches,<br/>rate limiting<br/>(in-task sidecar)")] end Voyage["Voyage AI<br/>embeddings API"] Browser --> Landing Browser -->|same-origin, session cookie| BFF Landing -.links to.-> BFF BFF -->|"user JWT + service token (TLS)"| REST LLM -->|"Streamable HTTP + per-user PAT (TLS)"| MCP Core --> PG Core --> Redis Core -->|cache miss| Voyage ``` See [SPEC.md §2](SPEC.md#2-system-architecture) for the component-boundary contract. ## Quick start (local) ```bash cp .env.example .env # Then set in .env: # VOYAGE_API_KEY=... (required to embed on write/search) # AUTH_TOKEN=... (required; REST data endpoints fail closed without it) # RECALL_JWT_PUBLIC_KEY=... (PEM SPKI Ed25519 — verify key for user JWTs) docker compose up --build ``` API at `http://localhost:8000`. Interactive docs (open) at `/docs`; `/health` is open. REST data endpoints authenticate one of two ways: the BFF path needs **two** headers — `X-Recall-Service-Token: $AUTH_TOKEN` plus `Authorization: Bearer <user JWT>` (minted by the BFF — see `web/README.md` to run the frontend locally) — while a **personal access token** in `Authorization: Bearer` alone is complete auth (what the CLI sends; point it here with `recall login --api-url http://localhost:8000`). The MCP endpoint takes the same PAT — generate one on the web app's **Connect** page (or via `POST /tokens`). ## Connect Claude to your knowledge base (MCP) The hosted MCP endpoint is `https://api.tryrecall.net/mcp` (Streamable HTTP), authenticated with a **personal access token**: sign in at [app.tryrecall.net](https://app.tryrecall.net), open **Connect**, and generate one (`recall_pat_…` — shown once, revocable any time). Tools act as the token's owner, so every user connects with their own token. **Claude Code:** ```bash claude mcp add --transport http recall https://api.tryrecall.net/mcp \ --header "Authorization: Bearer <your token>" ``` **Claude Desktop** (its connector UI is OAuth-only, so bridge the bearer token with `mcp-remote`; needs Node.js) — in `claude_desktop_config.json` (macOS: `~/Library/Application Support/Claude/`, Windows: `%APPDATA%\Claude\`), then fully quit and reopen Desktop: ```json { "mcpServers": { "recall": { "command": "npx", "args": ["-y", "mcp-remote", "https://api.tryrecall.net/mcp", "--header", "Authorization:${AUTH_HEADER}"], "env": { "AUTH_HEADER": "Bearer <your token>" } } } } ``` (The `${AUTH_HEADER}` indirection is `mcp-remote`'s documented workaround for a Cursor / Claude Desktop-on-Windows bug that mangles spaces inside `args` — one snippet, every platform. The web app's **Connect** page walks through this track step by step.) Then ask Claude things like "what did I save about rate limiting?" (`search_notes`), "remember that X" (`add_note`), or "what else relates to that note?" (`get_related`). ## Use it from the terminal (CLI) The same token drives the CLI — no separate credential, no service secret: ```bash uv tool install git+https://github.com/adityapall1/Recall # installs `recall` recall login # paste your token at the masked prompt (stored 0600 in ~/.config/recall/config.toml) recall add "RRF fuses ranked lists by 1/(k+rank); k=60 is standard" recall search "reciprocal rank fusion" recall add --url https://example.com/post # fetch + extract + embed a page recall import ./notes/ # bulk Markdown (frontmatter-aware) ``` `RECALL_API_TOKEN` / `RECALL_API_URL` override the config file (useful in CI or against a local API); `recall logout` forgets the stored token. A revoked token fails with a one-line pointer back to `recall login`. ## Key tech decisions at a glance - **FastAPI + async SQLAlchemy 2.0 + asyncpg** — async-native end to end; FastAPI gives Pydantic validation and OpenAPI for free. - **Postgres 16 + pgvector** — one ACID datastore for users, notes, chunks, vectors (HNSW), and full-text (`tsvector` + GIN). No second vector DB to sync. - **Hybrid search with Reciprocal Rank Fusion** — dense (semantic) + sparse (lexical) arms, fused by RRF (`k=60`); no score normalization, robust to either arm failing on a query. - **Dual-gate auth on an internet-facing API** — for the browser path, a shared service token (`X-Recall-Service-Token`) gates traffic and identity is a separate seam: the BFF mints a 5-minute **EdDSA JWT** per request and the API verifies with a pinned algorithm (no `alg` confusion) and only ever holds the public key. A compromised API cannot mint tokens. For MCP and the CLI, per-user **PATs** are the sole credential (revocable, per-user — a shared secret distributed to every user would defeat itself), hashed with SHA-256 — deliberately not bcrypt: a 256-bit random secret has no brute-force surface, and the fast hash keeps the per-request lookup free. Token management itself is web-session-only, so a leaked PAT cannot mint replacements that outlive its revocation. - **Voyage AI `voyage-3.5-lite` embeddings** — 1024-dim, query/document-asymmetric, very low cost; Anthropic's recommended embeddings provider. - **MCP over Streamable HTTP** — FastMCP mounted as an ASGI sub-app at `/mcp` in the same FastAPI process; one hosted endpoint, many clients (each with their own revocable token), no per-machine MCP setup. - **AWS Fargate + RDS + ALB/ACM, Redis as an in-task sidecar** — one Docker image, one always-on ARM64 task (us-east-2); GitHub Actions deploys via OIDC. ElastiCache is the documented scale path; the sidecar saves its monthly fixed cost and is fine because every cache fails open. Full rationale for every pick — including what was considered and rejected — in [RESEARCH.md](RESEARCH.md). ## Documentation - **[SPEC.md](SPEC.md)** — Vision, architecture, data model, API surface, search design, NFRs, deployment topology, next steps. - **[RESEARCH.md](RESEARCH.md)** — Every dependency, why it's here, why it beat the alternative. The doc interview questions come from. - **[docs/multiuser-auth-plan.md](docs/multiuser-auth-plan.md)** — The multi-user auth design (implemented; kept as the rationale record). - **[docs/mcp-pat-plan.md](docs/mcp-pat-plan.md)** — The per-user MCP access-token design (implemented; kept as the rationale record). - **[web/README.md](web/README.md)** — The web app + BFF proxy. **[landing/README.md](landing/README.md)** — the landing page.