big-brain-db

# Big Brain DB Big Brain DB is an agent-first Markdown brain database. It is a local-first knowledge app for folders of Markdown files, with graph navigation, Brain anchors, collaboration primitives, change history, conflict detection, and a local MCP server so agents such as Claude Code and Cursor can read, search, lint, and maintain Brains. It also ships an optional **server layer** (Supabase/PostgreSQL) that adds scope-based brain spaces, server-persisted brain documents, a hybrid graph that renders the database schema alongside the markdown wiki, and an agent-brain ingestion service that keeps durable knowledge in sync. The server layer is opt-in: without Supabase configured, Big Brain DB stays fully local-first. The product is inspired by the personal-wiki and context-engineering style popularized by **Andrej Karpathy**. Big Brain DB is not affiliated with or endorsed by him; the reference is an acknowledgement of the idea that a durable, well-linked knowledge base can become a high-leverage context layer for humans and agents. ## What Big Brain DB Is Big Brain DB treats a Markdown folder as a Brain. Each Brain has one central Brain file, usually named like `my-project-brain.md`, plus internal agent files such as `AGENT_START.md` and `log.md`. Content files become graph nodes. The Brain file becomes the anchor for that cluster. The app is designed for three users at once: - Humans who want a fast local Markdown wiki with properties, graph navigation, and readable folders. - Collaborators who need invite codes, roles, local membership state, and future-ready Supabase persistence. - Agents that need a stable operating contract, a graph, a log, frontmatter metadata, and MCP tools for ingesting, linting, linking, and editing. ## Core Features - Local Markdown vault loading through the browser File System Access API. - Three views — Markdown editor, a 2D neuron-network graph, and a 3D anatomical glass brain. - In-app **Brain Chat**: RAG-grounded Q&A over your notes via OpenRouter, with a model switcher, editable system prompt, source chips, and a context inspector. - Create and (soft-)delete Sub-Brains from the UI, with scaffolded anchor, log, and content folders. - Graph view as the default start view. - Big Brain root node plus Sub-Brain anchor nodes. - Brain metadata files hidden from the graph as separate nodes and shown through Brain panels instead. - Sidebar folders default to closed. - Markdown properties/frontmatter editor. - Canonical actor metadata: `created_by`, `created_at`, `updated_by`, `updated_at`, `last_change_summary`, `content_hash`. - Local collaboration state with actors, memberships, invite codes, roles, change events, versions, baselines, and conflicts. - Supabase-backed invite persistence when configured. - Local in-memory invite fallback for development. - MCP server for Claude Code, Cursor, and other MCP clients, with content-hash change detection (unchanged writes are skipped) and automatic document-type inference. - Optional server layer: scope-based brain spaces (`user`, `base`, `group`, `agent`, `council`), server-persisted brain documents with audit trail, and wiki-link relations. - Hybrid graph filter (`markdown` / `database` / `hybrid`) that renders the brain schema (tables, foreign keys, raw sources) next to the markdown graph, behind an anatomical brain silhouette. - Agent-brain ingestion that auto-generates and refreshes durable-knowledge scaffolding pages with stable-diff change detection. ## Project Status This is a local-first MVP that is useful for development, demos, and single-machine workflows. Production collaboration still needs stronger auth, durable permissions, server-side authorization, migrations, and deployment hardening. ## Install ```bash npm install ``` ## Run The App ```bash npm run dev ``` Open: ```text http://localhost:3000 ``` Use a Chromium-based browser such as Chrome, Edge, or Arc. The local folder picker depends on the File System Access API. ## First Use 1. Start the local app with `npm run dev`. 2. Open `http://localhost:3000`. 3. Click `Add Brain`. 4. Choose a Markdown folder. 5. Big Brain DB will create missing internal Brain files when it has write access: - `<folder-name>-brain.md` - `AGENT_START.md` - `log.md` 6. The app opens in graph view by default. 7. Use the sidebar to open files manually. Folders start closed. Important: if an agent will work with a Brain, keep the local app running on localhost. The browser app owns the local file-handle workflow, visual graph, reconnect flow, and collaboration UI. ## Brain Structure A Brain folder should be understandable by both humans and agents. Recommended files: - `<brain-name>-brain.md`: the Brain anchor, map, and role contract. - `AGENT_START.md`: copyable startup instructions for an agent. - `log.md`: chronological history of ingests, lint passes, maintenance, decisions, and open questions. - Content pages: source notes, concept pages, entity pages, project pages, syntheses, questions, and templates. Avoid folder-level `README.md` files as automatic navigation. Use the Brain file, hub pages, and semantic links instead. ## Creating and Deleting Sub-Brains A **Sub-Brain** is a brain nested inside another brain: any topic can graduate into its own brain when it grows large enough to own its structure. A folder becomes a sub-brain the moment it contains its own name-matching `<name>-brain.md` anchor, so nesting is recursive — you can create sub-brains of sub-brains. ### Create a sub-brain (two ways in the app) **A. From the sidebar** — hover a brain/folder row and click the **`New Sub-Brain`** icon button (next to `New Markdown file`). You'll be prompted for a name. **B. From the `Add Brain` dialog** — open `Add Brain`, use the **`Sub-Brain`** card ("Create a sub-brain under an existing brain."), pick a parent from the **`Choose a parent brain…`** dropdown (nested brains show a `↳` prefix), and click **`Create sub-brain`**. Either way the app writes, inside a new `<name>-brain` folder: - **`<name>-brain.md`** — the anchor, with `type: brain`, `tags: [brain, sub-brain]`, a `## Role In The Big Brain` stub (Owns / Does not own / Connects to `[[../<parent>]]` / Entry point), and a `## Navigation Map`. - **`log.md`** — seeded with a `setup | <title> sub-brain created` entry. - Five standard content subfolders, each with a `.gitkeep`: **`concepts/`, `entities/`, `sources/`, `syntheses/`, `projects/`**. The new anchor opens in edit mode so you can fill in its role and map. (Sub-brains do not get their own `AGENT_START.md` — that lives once at the vault/brain root.) Creating a sub-brain requires write access to the folder. ### Delete a sub-brain (soft-delete) Hover the sub-brain row and click the **`Delete sub-brain`** (trash) icon. You'll be asked to confirm: *"Move the sub-brain "&lt;name&gt;" and everything in it to the Brain trash (.brain-trash)? You can restore it from there."* Deletion is **non-destructive**: the entire subtree is first copied to `.brain-trash/<timestamp>__<name>/` at the vault root, then removed from its original location. Because `.brain-trash` is a hidden dot-folder, its contents never re-appear as graph nodes. To **restore**, move the folder back out of `.brain-trash` manually. ## Views: Editor, Graph, and 3D Brain The top-right view toggle switches between three views: - **`Editor`** — the Markdown + frontmatter editor for the selected file. The editor auto-grows to content height so properties and text scroll together. - **`Graph`** — the 2D wiki graph, rendered as a **neuron network**: nodes are neuron cell bodies (soma + dendrite spikes) colored by their (sub-)brain, edges are curved synapse fibres, and hovering a node lights it and its neighbours up with a signal pulse travelling the fibres. Layout is radial: the Big Brain sits at the center, top-level sub-brains form a ring, nested sub-brains cluster around their parent, and pages orbit their owning brain. Drag to pan, scroll to zoom. - **`Brain`** — a 3D anatomical glass-brain mesh with the same wiki graph rendered *inside* it as an instanced neuron network. Drag to rotate, scroll to zoom, hover a neuron to light up its synapses. Nodes stay dim at rest (no idle auto-firing). Both graph views share a **`markdown` / `database` / `hybrid`** mode toggle: - **`markdown`** — the local wiki-link graph only (default). - **`database`** — the Supabase brain schema (tables, foreign keys, raw sources). Requires Supabase configured; otherwise shows "Schema graph unavailable. Configure Supabase to enable the database layer." - **`hybrid`** — both, merged, behind a faint anatomical brain silhouette. Selecting a node in one view highlights the same node in the other. ## Brain Chat (Ask the Brain) Big Brain DB ships an in-app chat assistant that answers questions **grounded in your notes** via retrieval-augmented generation (RAG). It talks to LLMs through [OpenRouter](https://openrouter.ai), so the model key never reaches the browser. ### Setup Set `OPENROUTER_API_KEY` in `.env.local` (see [Supabase Setup](#supabase-setup) for the full env block). For RAG grounding you also need Supabase running and — for the semantic (vector) half — an embedding key (`OPENAI_API_KEY` or `BRAIN_EMBEDDING_API_KEY`). Without an embedding key, retrieval still works using lexical (full-text) search only. Chat itself only hard-requires `OPENROUTER_API_KEY`; if the database is unreachable the model still answers, just without grounded context. ### Using it - Click the floating brain launcher (bottom-right, **`Ask the Brain`**) to open the "Glass Chatbar". - Type a question (`Message the Brain…`). **Enter** sends, **Shift+Enter** adds a newline. Answers stream token-by-token; grounding **source chips** (`title · brain`) appear from the retrieved documents. - **Model switcher** — a searchable dropdown listing OpenRouter's full model catalogue (default `openai/gpt-4o-mini`); your choice persists locally. - **Settings** (gear icon): - **System prompt** — editable, with `Reset to default`. - **`Use Brain context (RAG)`** — toggle grounding on/off. - **`Context documents`** — 1–20 slider (how many retrieved docs to feed the model). - **Search scope** — `All brains` (global search across every space) or a single brain space. - **Context inspector** — `Inspect context for current question` shows exactly which documents and text would be retrieved, LLM-free, before you send. ### How retrieval works `retrieveBrainContext` runs a hybrid search — full-text (tsvector) **and** dense vector (pgvector cosine) — fused with Reciprocal Rank Fusion. With no scope selected it searches every brain space in one global query. Template/scaffold noise is filtered out, and each top document contributes up to ~4000 chars of content to the assembled context block. Chat API routes: `POST /api/brain/chat` (streaming), `GET /api/brain/models`, `GET /api/brain/spaces`, `POST /api/brain/context` (retrieval preview). ## Collaboration Model Big Brain DB currently has a local-first collaboration model with server-ready concepts: - Actors: humans and agents. - Roles: `owner`, `admin`, `editor`, `commenter`, `viewer`, `agent`. - Memberships: which actor has which role in which Brain. - Invitations: role-scoped invite codes. - Change events: local save/update events with actor, summary, timestamp, and content hash. - File versions: local snapshots created on save. - Baselines: content hashes used for conflict detection. - Conflicts: detected when the disk file changed after the local baseline. The UI intentionally keeps `Acting as` simple: - `Human` - `Agent` The invited person enters their own collaborator name when joining with an invite code. The inviter only chooses a role and receives a code. ## Supabase Setup Supabase is optional for local development but recommended if you want invite codes to persist beyond the current local Next.js server process. Create `.env.local` from the safe placeholder file: ```bash cp .env.local.example .env.local ``` Fill it with your own project values: ```bash # Supabase — powers the database/hybrid graph layers and Brain Chat RAG retrieval NEXT_PUBLIC_SUPABASE_URL=<your-supabase-project-url> NEXT_PUBLIC_SUPABASE_ANON_KEY=<your-supabase-anon-key> SUPABASE_SERVICE_ROLE_KEY=<your-supabase-service-role-key> # OpenRouter — server-side only; powers Brain Chat (/api/brain/chat) OPENROUTER_API_KEY=<your-openrouter-key> # Optional: sent as the OpenRouter HTTP-Referer / attribution header OPENROUTER_SITE_URL=http://localhost:3000 # Embeddings — enables the dense (pgvector) half of RAG retrieval. # Without it, RAG falls back to lexical (full-text) search only. OPENAI_API_KEY=<your-openai-key> # or: # BRAIN_EMBEDDING_API_KEY=<embedding-key> ``` Never commit `.env.local`. Never commit real keys. The service role key and `OPENROUTER_API_KEY` are server-only and must not be exposed to browsers or public repositories. If a key ever leaks, rotate it at the provider immediately. ### Local Supabase in one command A full local Supabase stack ships with the repo (`supabase/config.toml`, `supabase/migrations/`). If you have the [Supabase CLI](https://supabase.com/docs/guides/cli): ```bash supabase start # boots Postgres + Studio locally, applies migrations ``` `supabase start` prints the local URL, anon key, and service-role key — paste those three into `.env.local`. Studio runs at `http://localhost:54323`. The database is only needed for the graph `database`/`hybrid` layers and RAG-grounded chat; the app is otherwise fully local-first. ### Minimal Supabase Tables Create these tables in Supabase SQL editor or through migrations: ```sql create table if not exists brains ( id text primary key, name text not null, created_by_actor_id text not null, created_by_actor_name text not null, created_by_actor_type text not null check (created_by_actor_type in ('user', 'agent')), updated_at timestamptz not null default now() ); create table if not exists brain_memberships ( brain_id text not null references brains(id) on delete cascade, actor_id text not null, actor_name text not null, actor_type text not null check (actor_type in ('user', 'agent')), role text not null check (role in ('owner', 'admin', 'editor', 'commenter', 'viewer', 'agent')), invited_by_actor_id text, created_at timestamptz not null default now(), primary key (brain_id, actor_id) ); create table if not exists brain_invites ( id uuid primary key default gen_random_uuid(), brain_id text not null references brains(id) on delete cascade, brain_name text not null, invite_code_hash text not null unique, invite_code_hint text, email text not null default 'pending collaborator', role text not null check (role in ('admin', 'editor', 'commenter', 'viewer', 'agent')), invited_by_actor_id text not null, invited_by_actor_name text not null, status text not null default 'pending' check (status in ('pending', 'accepted', 'revoked')), accepted_by_actor_id text, accepted_by_actor_name text, accepted_at timestamptz, created_at timestamptz not null default now() ); create index if not exists brain_invites_code_hash_idx on brain_invites(invite_code_hash); create index if not exists brain_invites_brain_id_idx on brain_invites(brain_id); ``` The app currently uses a service role key in server routes. Before production deployment, add proper auth, row-level security, ownership checks, invite expiration, and audit logging. The full, current schema — including the server-layer brain tables described below — lives in [`supabase/schema.sql`](supabase/schema.sql). Run that file in the Supabase SQL editor to provision everything at once. ## Server Layer (Hybrid Brain) The server layer is optional and activates only when Supabase is configured. It mirrors the brain module that ships inside Connect, adapted from Prisma to the Supabase service client. ### Data model `supabase/schema.sql` adds six tables: - `brain_spaces` — scope-based knowledge containers (`user`, `base`, `group`, `agent`, `council`), unique per `(user_id, scope_type, scope_id)`. - `brain_documents` — human-readable markdown documents with frontmatter, tags, content hash, and actor audit fields, unique per `(space_id, slug)`. - `brain_sources` — provenance for imported/ingested content. - `brain_entities` — normalized entity nodes (people, orgs, concepts). - `brain_relations` — graph edges, including `wiki_link` relations derived automatically from `[[links]]` on every document write. - `brain_change_events` — append-only audit log of create/update actions. PostgreSQL is the operational source of truth; the markdown documents remain the durable, human-readable layer. ### API routes | Route | Purpose | | --- | --- | | `GET/POST /api/brain` | Read documents by scope, or upsert a space/document. | | `GET/PATCH /api/brain/[documentId]` | Read or update a single document. | | `GET /api/brain/schema-graph` | Hybrid-graph payload: tables, foreign keys, raw sources, ownership. | | `POST /api/brain/import` | Import markdown files into a scoped space (frontmatter + type inference). | | `GET/POST /api/brain/agents/ensure` | Ensure agent brain spaces and refresh their scaffolding pages. | All routes operate under a single local user (`local-user`) and degrade gracefully: if Supabase is not configured they return a `*_FAILED` error rather than crashing the local-first app. ### Hybrid graph The graph view has a `markdown` / `database` / `hybrid` toggle (top-right of the canvas): - `markdown` — the original local wiki-link graph (default; unchanged behaviour). - `database` — the brain schema graph from `/api/brain/schema-graph`. - `hybrid` — both, side by side. An anatomical brain silhouette is rendered faintly behind the graph in every mode. ### Agent-brain ingestion `syncAgentBrainKnowledgeSpaces` (exposed via `/api/brain/agents/ensure`) creates an `agent`-scoped space per profile and generates the standard durable-knowledge pages — `AGENT_START`, `log`, dataset inventory, projection policy, and ingestion gaps. A stable-diff check means unchanged pages are never rewritten. The built-in `brain` profile reflects the brain's own tables (counts and recent rows). ## No Supabase Data In This Repository This repository should contain only placeholders such as `.env.local.example`. Real values belong in local environment files or deployment secrets. Also note: - `.env.local` is ignored. - `.env*.local` is ignored. - `demo-wikis/` is ignored and should not be pushed to GitHub. If a real key was ever committed or shared, rotate it in Supabase immediately. ## Continuing A Brain With An Agent When an agent continues a Brain, always start the local app first: ```bash npm run dev ``` Then give the agent a concrete task and the Brain folder context. The agent should: 1. Read the Brain file. 2. Read `AGENT_START.md`. 3. Read `log.md`. 4. Inspect the graph or use MCP tools. 5. Identify the ownership boundary. 6. Make small, source-backed edits. 7. Update links, frontmatter, and log entries when structure changes. 8. Run typecheck/build when app code changes. ### Ingest Workflow Use ingest when adding new material. 1. Preserve the unprocessed source first under `raw/`. 2. Create a structured source note under `sources/`. 3. Link the structured source note back to the raw note with `raw_source`. 4. Extract durable claims, entities, concepts, contradictions, and open questions. 5. Link to concepts, entities, projects, and syntheses. 6. Update the Brain file when navigation changes. 7. Append a `log.md` entry. The MCP `ingest_text` tool enforces this by writing a raw note before it writes the source note. ### Lint Workflow Use lint when maintaining Brain quality. Check for: - Orphan pages. - Missing backlinks. - Broken or unresolved wiki links. - Stale frontmatter. - Duplicate concepts. - Claims without sources. - Contradictions that are not marked. - Agent instructions that conflict with the Brain file. - Old `protocol` terminology where Brain terminology should be used. ### Relation Workflow Relations should help future readers and agents navigate. Use links when the relationship is durable: - `supports` - `contradicts` - `depends_on` - `implements` - `source_for` - `handoff_to` - `related` Avoid dense decorative linking. Every important link should be explainable in one sentence. ## MCP Server Big Brain DB includes a local stdio MCP server: ```bash npm run mcp:brain-nodes -- --vault "/path/to/your/brain" ``` The MCP server can: - list notes - read notes - write notes - patch notes - delete notes with confirmation - move notes - search notes - build the graph - inspect a note - lint a Brain - suggest relations - add relations - append to `log.md` - ingest pasted text Writes go through canonical actor stamping with two ports from the Connect brain module: - **Document-type inference** — when a note has no explicit `type` in frontmatter, the type is inferred from its path (`sources/` → `source`, `*-brain.md` → `brain`, `AGENT_START.md` → `agent-start`, and so on). - **Change detection** — overwriting a note with unchanged meaningful content (body + non-volatile frontmatter) is skipped and returns `skipped: true`, so repeated/idempotent ingests don't churn timestamps or history. The server only reads and writes below the configured `--vault` path. It rejects path traversal with `..`. The MCP server is local and file-based; the database schema graph is exposed through the web API (`/api/brain/schema-graph`), not the stdio MCP. ## Claude Code MCP Connection Claude Code and other MCP clients do not automatically run a server just because a repository contains one. This is a security feature. You must explicitly register the server in the MCP client config. Use an absolute path. Paths with spaces are fine when they are passed as JSON array arguments: ```json { "mcpServers": { "big-brain-db": { "command": "node", "args": [ "/absolute/path/to/big-brain-db/mcp/server.mjs", "--vault", "/absolute/path/to/your/brain", "--actor", "agent:claude-code", "--actor-name", "Claude Code" ] } } } ``` For read-only inspection: ```json { "mcpServers": { "big-brain-db-readonly": { "command": "node", "args": [ "/absolute/path/to/big-brain-db/mcp/server.mjs", "--vault", "/absolute/path/to/your/brain", "--actor", "agent:claude-code", "--actor-name", "Claude Code", "--readonly" ] } } } ``` ## Agent Start Prompt Copy this prompt into a new agent session when you want an agent to continue a Brain. ```text You are working inside Big Brain DB, an agent-first Markdown Brain database. Your job is to continue the selected Brain as durable knowledge infrastructure, not as disposable chat context. Preserve structure, links, sources, decisions, contradictions, open questions, and handoff notes so future humans and agents can continue without rediscovering the same context. Before doing any work: 1. Confirm that the Big Brain DB web app is running locally, usually at http://localhost:3000. 2. Confirm the Brain folder you are responsible for. 3. Read the Brain file completely. It is usually named <brain-name>-brain.md. 4. Read AGENT_START.md completely. 5. Read log.md for recent work, decisions, lint findings, and unresolved tasks. 6. If MCP is available, call get_server_info, list_notes, and lint_brain before editing. 7. Identify whether the task is query, ingest, lint, maintenance, refactor, collaboration, or release work. 8. Identify the ownership boundary: what this Brain owns, what belongs in another Brain, and what should remain only as a source. Operating rules: - Prefer durable Markdown pages over hidden chat memory. - Use wiki links for semantic relationships. - Keep important nodes reachable from the Brain file or a meaningful hub page. - Preserve raw sources and source-backed claims. - Mark uncertainty as Open Questions. - Record contradictions instead of smoothing them over. - Avoid decorative link spam. - Do not invent facts. - Do not delete or rewrite user content unless the task explicitly requires it. - Do not expose secrets, private env values, service role keys, tokens, credentials, or private data. - Keep file names stable and link-friendly. - Do not create folder README files automatically. When answering a question: 1. Search the Brain first. 2. Read relevant sources and linked pages. 3. Answer with citations to the pages or files used. 4. If the answer creates durable synthesis, save it as a page and link it from the Brain file, a hub, or log.md. When ingesting: 1. Read one source at a time when possible. 2. Preserve the unprocessed source first under raw/. 3. Create a structured source note and link it back to the raw note with raw_source. 4. Extract durable claims, entities, concepts, contradictions, decisions, and open questions. 5. Create or update concept/entity/project/synthesis notes only when they add durable value. 6. Link the new material semantically. 7. Update the Brain file if the navigation map or ownership changed. 8. Append a concise log.md entry with summary, files touched, decisions, sources, and open questions. When linting: 1. Check unresolved links, orphan pages, duplicate titles, weak source coverage, stale metadata, contradictions, missing backlinks, and old protocol terminology. 2. Separate findings from fixes. 3. Fix clear local issues when authorized. 4. Record unresolved findings in log.md. When editing app code: 1. Read the relevant code before changing it. 2. Follow existing patterns. 3. Keep edits scoped. 4. Avoid unrelated refactors. 5. Run npm run typecheck after substantive changes. 6. Run npm run build before release or GitHub publishing. When using collaboration: 1. Treat humans and agents as actors. 2. Use canonical actor IDs such as user:local-owner or agent:claude-code. 3. Do not duplicate actor name/type frontmatter when the actor ID already encodes the identity. 4. Use invite codes for joining. 5. Let the invited person choose their own collaborator name when joining. When using MCP: 1. Prefer read_note, search_notes, get_graph, inspect_note, and lint_brain before writing. 2. Prefer patch_note for precise edits. 3. Use write_note only when creating a new note or intentionally overwriting. 4. Use append_log for durable maintenance history. 5. Use ingest_text for pasted source material; it always writes raw/ first and then creates the linked source note. 6. Never write outside the configured vault. End every meaningful work session by: 1. Updating log.md if durable work was done. 2. Updating the Brain file if navigation, ownership, or workflows changed. 3. Reporting what changed, what was validated, and what remains risky or unresolved. ``` ## Development Commands ```bash npm run dev npm run typecheck npm run build npm run mcp:brain-nodes -- --vault "/path/to/your/brain" ``` ## GitHub Publishing Checklist Before publishing: - Confirm `.env.local` is ignored. - Confirm `.env*.local` is ignored. - Confirm no real Supabase keys appear in tracked files. - Confirm `demo-wikis/` is ignored or removed from the commit. - Run `npm run typecheck`. - Run `npm run build`. - Review `README.md`, `AGENT_START.md`, and `big-brain-db.md` for stale protocol terminology. - Do not publish local user Brains unless they are intentionally public. ## License Released under the [MIT License](LICENSE) — you may use, modify, and redistribute it, including commercially, provided the copyright notice and license text are retained. The software is provided "as is", without warranty.