knowledge_base_qa_bot

# Knowledge Base Q&A Bot **English** · [](https://ask-wiki-rag.paynepew.dev)    A grounded Q&A bot over a Markdown knowledge base. Every answer is assembled only from cited records. When the sources don't back a claim, the bot says _"I cannot confirm from the knowledge base"_ rather than guessing. The same question can run against **three retrieval stacks**, switchable from one toggle: | Stack | How it retrieves | Corpus | | --- | --- | --- | | **Wiki** (A) | BM25 keyword search over a curated `wiki/` layer ([Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) pattern) | `wiki/` Sections | | **RAG** (B) | Dense vector search (FAISS) over raw chunks | `docs/` Chunks | | **Hybrid** (C) | BM25 keyword **and** dense vectors (the same technique RAG uses), both over the curated `wiki/` layer, fused by Reciprocal Rank Fusion | `wiki/` Sections | > **"Hybrid" fuses two retrieval _methods_, not the two other _stacks_.** Its dense > arm is the same vector-search technique the RAG stack uses > (`text-embedding-3-small` + similarity), pointed at the curated `wiki/` Sections > instead of `docs/` chunks. Both Hybrid arms return wiki Sections with aligned ids, > so the fusion stays clean and the wiki citation is kept. Merging the Wiki and RAG > stacks directly (different corpora, different units) was the cross-corpus reading > explicitly rejected in [ADR-0018](project-docs/adr/0018-hybrid-retrieval-third-stack-rrf-over-wiki.md). **Live demo: <https://ask-wiki-rag.paynepew.dev>.** Reader at `/`, Operator Console at `/console`. ## Architecture One question enters from any of three interfaces, a stack is selected, and all three stacks hand off to **one shared grounded-answer pipeline**. Adding the Hybrid stack was a single dispatch entry plus a dense index and an RRF merge. The grounding check, citation, page expansion, and "I cannot confirm" gate were already written once and reused.  The Browser talks to a FastAPI **Gateway** that owns multi-turn query rewriting, the session store, and a production guard (daily budget + concurrency caps). The **CLI** (`kb`) and the **MCP server** (`kb_mcp`) skip the Gateway and call the same stacks directly, single-turn. The three interfaces share no state except the files on disk. They meet at the corpus. ### How knowledge gets in Sources in `docs/` are immutable. **Ingest** synthesises them once into a curated `wiki/` layer (with a Grounding Check on every page it writes), and three indexes are built from that corpus. This is the write side of the Operator Console.  > Diagram sources live next to the images as editable Mermaid: > [`runtime.mmd`](project-docs/architecture/runtime.mmd), > [`lifecycle.mmd`](project-docs/architecture/lifecycle.mmd). ## Technical highlights - **Grounded by construction, not by prompt.** A pre-LLM relevance gate decides whether the retrieved context is strong enough to answer at all; after the LLM drafts, a second model pass (the **Grounding Check**) extracts each atomic claim and checks it against the cited Sections. One unsupported claim discards the whole draft and returns _"I cannot confirm."_ The system fails closed. - **Three retrieval architectures behind one toggle**, compared with an honest, statistically-tested benchmark (Cochran's Q omnibus → post-hoc McNemar with Holm correction), not a hand-picked headline number. - **Same-corpus Hybrid fusion.** Both Hybrid arms index the _same_ wiki Sections, so their ids align 1:1 and Reciprocal Rank Fusion (K=60) needs no score reconciliation between BM25 magnitude and FAISS distance. The Cannot-Confirm gate is a per-arm **OR** over each arm's own calibrated threshold: recall-union without inventing a new threshold. - **Deep-module reuse.** Every stack returns a `Section` or `Chunk` that satisfies one `CitableContent` protocol, so grounding, citation, page expansion, and the refusal gate are written once and shared across all three. - **Bilingual end to end.** Files and questions work in English or Traditional/Simplified Chinese; the keyword stacks tokenise CJK by character n-gram and apply a per-language score threshold, and Chinese questions get Chinese answers. - **Three interfaces over one corpus.** Browser, CLI, and MCP server each drive the full Upload → Import → Ingest → Index → Lint lifecycle; they share state only through the filesystem. - **Production posture.** Dockerised, with image-pull CD to a VPS on every push to `main` (GHCR → SSH → `/healthz` smoke). A `/healthz/shed` probe sheds reader traffic at the edge under load; a daily USD budget guard and read/admin concurrency caps protect the box. The retrieval indexes ship committed, so a fresh clone answers on the first run. ## Agentic workflow orchestrator The product above is also **built by agents** — feature work runs through a version-controlled orchestrator ([`project-docs/agents/orchestrator.js`](project-docs/agents/orchestrator.js)) that drives GitHub issues to merged PRs in four phases:  The design follows the workflow/agent distinction from Anthropic's [Building Effective Agents](https://www.anthropic.com/engineering/building-effective-agents): the **outer layer is a deterministic workflow** — plain JavaScript control flow that fans out one pipeline per issue, applies the severity gate, and reports — so a run is inspectable, resumable (completed agent calls replay from cache), and cheap to re-run. **Autonomy is delegated only inside the nodes**: each role agent works to a version-controlled prompt contract in [`project-docs/agents/`](project-docs/agents/). - **Separation of duties, enforced by the platform.** The builder never certifies its own work: an adversarial verifier (read-only; every blocker needs evidence, and only `critical`/`high` findings block) gates each branch, and after the PR opens an **independent Verdict agent** re-verifies the pushed SHA and posts the `verify/verdict` commit status it just formed itself — the poster is the verifier, so it is never self-approval. Branch protection (CI on Windows + Ubuntu, plus that status) is the hard gate no agent can bypass. - **An autonomy ladder, not a switch.** Rung 0 stops at verified local branches; the default rung opens PRs and posts the verdict but leaves the merge button to a human; Rung 3 (opt-in per run) arms `gh pr merge --auto`, and GitHub itself merges only when every required check is green. Trust is widened one rung at a time, and any incident demotes a rung. - **Model tiering per role.** Mechanical steps (plan, open-PR) run on a small model, implementation on a mid-tier one, and both verification passes on the strongest — reasoning is spent where being wrong is expensive. - **Guardrails as part of the prompt contract.** Every file-touching agent carries hard anti-pollution rules (never initialise another tracker, never touch other slices' work or repo config, never merge) — each rule traceable to a real incident — and every build runs in an isolated git worktree. The manual one-role-at-a-time version of the same loop, its hand-off contract, and its stop conditions are documented in [`project-docs/orchestration-plan.md`](project-docs/orchestration-plan.md). > Diagram source: > [`orchestrator.mmd`](project-docs/architecture/orchestrator.mmd). ## Tech stack Every choice below earns its place; the note next to it says why it won over the obvious alternative. | Layer | Choice | Why this one | | --- | --- | --- | | Language & tooling | **Python 3.11+**, **uv** workspace (one root `.venv`, 7 members) | uv resolves the whole workspace once; package members (`kb_cli`, `kb_mcp`) get console-script entry points, library members stay namespace-only. | | Web framework | **FastAPI** + **Uvicorn** (ASGI) | Async streaming, typed request models, and a generated OpenAPI schema the eval can fuzz. | | Streaming | **Server-Sent Events**, hand-written parser | Answers stream token-by-token, sources first; SSE is a better fit than WebSockets for one-way server push and needs no client library. | | Reader / Console UI | Plain **HTML + vanilla JS**, no framework | Zero build step and nothing to bundle; the page source is as inspectable as the index it drives, the same "keep it readable" bet the retrieval side makes. | | Keyword retrieval | **BM25**, hand-implemented over heading Sections | A ~120-line scorer over `.kb/index.json` is one `cat` from a diagnosis; no `rank-bm25` black box, and the index is plain JSON. | | Dense retrieval | **OpenAI `text-embedding-3-small`** + **FAISS** (`faiss-cpu`) | The RAG and Hybrid arms; FAISS persists to `.kb/faiss_index/` and `.kb/hybrid_dense/`, both committed so a clone answers offline-of-index. | | Fusion | **Reciprocal Rank Fusion** (K=60), hand-implemented | Rank-only fusion needs no score reconciliation between BM25 magnitude and FAISS distance, which is why both Hybrid arms index the same wiki Sections. | | LLM orchestration | **OpenAI SDK** via **LangChain-OpenAI**, structured output | Templates and the Grounding Check use `with_structured_output`; LangChain is borrowed at the leaf (loaders, splitters) and kept out of the domain core (ADR-0005). | | Import / convert | **markdownify** (`.html`), **MarkItDown** (`.pdf` text layer), a vision model (scanned-PDF transcribe) | Digital-native PDFs stay on the free mechanical path; only text-less scans pay for a model (ADR-0031 / ADR-0032). | | Optional reranker | **`bge-reranker-v2-m3`** cross-encoder via **sentence-transformers** | Eval-only and default-off, imported lazily so `torch` and the 2.3 GB model never touch the production image or the VPS tenant (ADR-0019). | | CLI / MCP | **Typer** (`kb`), the **MCP** SDK (`kb_mcp`) | Two more interfaces over the same on-disk corpus; no retrieval logic is re-implemented. | | Evaluation | **DeepEval** Synthesizer + a custom `BaseMetric`, **matplotlib** charts, hand-written **Cochran's Q / McNemar / Holm** | The framework's runner and dataset are borrowed; the opinionated hit metric and the stats are written by hand so the verdict is auditable. | | Delivery | **Docker** → **GHCR** → SSH image-pull CD, **Caddy** edge, **pytest** on a Windows + Ubuntu matrix | Every push to `main` rebuilds, ships, and smoke-tests `/healthz`; the edge sheds load and fronts three tenants. | ## Quick start The retrieval indexes ship **pre-built and committed**, so a fresh clone answers questions immediately, with no build step on the first run. ```bash # 1. Clone git clone https://github.com/PaynePew/knowledge_base_qa_bot.git cd knowledge_base_qa_bot # 2. Install (single .venv at the repo root) uv sync --all-packages # 3. Add your OpenAI key cp .env.example .env # then edit .env and set OPENAI_API_KEY=sk-... # 4. Launch the Gateway (serves the UI and all three stacks on one origin) uv run uvicorn gateway.app.main:app --port 8000 ``` Open <http://localhost:8000/> for the Reader and <http://localhost:8000/console> for the Console. > **Keys.** `OPENAI_API_KEY` is **required**: every stack calls OpenAI to write > the final answer and to embed (RAG and Hybrid). `ANTHROPIC_API_KEY` is > **optional**, used only by the evaluation's cross-family Claude judge. See > [`.env.example`](.env.example). ## Using the Reader **Reader.** Ask a question, pick a stack, and read a grounded answer. <img src="project-docs/screenshots/reader.png" /> - **Ask anything** in the box (English _or_ Chinese) and press Enter. - **Toggle Wiki / RAG / Hybrid** to run the same question against each retrieval stack and compare them side by side. - **Sources appear first** (the records the answer is grounded in), then the answer streams in below them. - A **grounding badge** confirms every claim traces back to a cited source; if it can't, you get _"I cannot confirm"_ instead of a hallucination. - **Follow-up questions** continue the same conversation (multi-turn on the web path). Switching stacks keeps the conversation history. ## Using the Console **Operator Console.** The admin interface: upload documents, run the build pipeline, and keep a company or personal knowledge base healthy. <img src="project-docs/screenshots/console.png" /> Before the Wiki stack can retrieve a file, it passes through the pipeline stepper, each step with its own **Run** button. **A file's journey, Upload → Import → Ingest → Index:** | Step | What it does | | --- | --- | | **Upload** | Drag-drop files. `.html` / `.txt` / `.pdf` land in `raw/`; `.md` goes straight to `docs/`. | | **Import** | Convert `raw/` sources into clean Markdown in `docs/` (with provenance). Digital-native PDFs use a text-layer extractor; text-less scans are **Transcribed** by a vision model instead (ADR-0031 / 0032). | | **Ingest** | An LLM synthesises `docs/` Sources into curated `wiki/` pages, grounding-checked as they are written. | | **Index** | Build the BM25 index (and the dense-over-wiki index) so new content is answerable. | The **RAG track** (separate panel) rebuilds the FAISS vector index from `docs/`, independent of the Wiki chain. Click **Rebuild**. **Long documents get a different shape.** A Source with no usable heading structure — a transcribed book, a long report — is detected as **longform** and routed specially: an LLM proposes chapter boundaries (**Structure Enrichment**), then Ingest writes one **Hub Page** ("what is this document about?") plus a concept page per chapter, instead of collapsing the whole thing into a single page (ADR-0033). **Lint and curation.** Run **Lint** after editing the corpus, or periodically, to audit knowledge-base health across the **C1–C12** lint taxonomy: coverage gaps and broken `[[wikilinks]]`, ungrounded pages, slug collisions, cross-page contradictions, stale and orphan pages, alias collisions, and the health of the bot's own filed Q&A. Every finding carries an **executable remediation**, sorted into four tiers by how much judgement it needs: | Tier | What you do | Examples | | --- | --- | --- | | **Direct** | One click, no LLM | **Promote** a filed Q&A to a permanent page, **Demote** a schema-invalid live answer to draft, assign an **Alias**, re-ingest a stale page, **Discard** a draft | | **Authored** | The bot drafts, you edit and apply | **Reconcile** two contradicting pages, **Merge / Differentiate** a slug collision | | **Confirmed** | You confirm an irreversible action | Delete an orphan page whose Source is gone | | **Routed** | The fix lives outside the wiki layer, so the console routes you there | **fix-the-Source** for an ungrounded page or a *source-rooted* contradiction, **fill-via-import** for a query with no coverage | The **Reconcile** modal is two-view: it shows the wiki pages side by side when the contradiction is fixable in the wiki layer, and drops to a **Source comparison** when the two Sources themselves disagree — so you fix the root, not the symptom. Promoted Q&A folds good answers back into the knowledge base over time. The **Resource Browser** at the bottom reads through `docs/`, `raw/`, and `wiki/` without leaving the page. ## Project structure ``` knowledge_base_qa_bot/ ├── gateway/ # FastAPI Gateway + browser UI (Reader at /, Console at /console); start here ├── markdown_kb/ # Wiki stack (A): BM25 over the curated wiki/ layer + the shared grounding pipeline ├── vector_rag/ # RAG stack (B): chunk + embed docs/ into FAISS ├── hybrid_kb/ # Hybrid stack (C): BM25 + dense over wiki/, fused by RRF ├── kb_cli/ # The `kb` command-line interface ├── kb_mcp/ # The kb_mcp MCP server (Claude Desktop compatible) ├── docs/ # Sources: the bot's immutable runtime knowledge base ├── wiki/ # Curated layer written by Ingest (concepts, entities, filed Q&A) ├── raw/ # Local inbox for uploaded .html/.txt before Import ├── .kb/ # Committed indexes: index.json (BM25), faiss_index/ (RAG), hybrid_dense/ (Hybrid) ├── eval/ # Three-stack paraphrase comparison harness + report ├── project-docs/ # ADRs, roadmap, coding standard, architecture diagrams, screenshots ├── CONTEXT.md # Shared vocabulary (glossary) └── .env.example # Copy to .env and add your OpenAI key ``` ## Evaluation: three stacks, measured Does a curated Wiki + BM25 actually out-retrieve plain Vector RAG on the same raw corpus, and where does Hybrid land? The benchmark answers it with real numbers rather than assertion. **Test set.** 260 queries over one 20-Source / ~51-Gold-Section corpus: **250 Core paraphrases** (5 LLM-generated rewrite types × 50) plus **10 hand-written structural probes** (2 types × 5). A "hit" requires the retrieved unit to match the gold section _and_ share content key-tokens, so a right-document-wrong-content result is a miss. **Method.** Each arm overfetches a deep candidate pool once per query, decoupled from the cutoff; hit-rate is reported across a sweep (hit@{1,3,5,10}) plus MRR. A three-way **Cochran's Q** omnibus gates **post-hoc pairwise McNemar** (with Holm correction), so no pairwise claim is made unless the omnibus first says some arm differs. The dense arms use real OpenAI `text-embedding-3-small`. **Core results (macro-average hit@3, real embeddings):** | | Wiki (A) | RAG (B) | Hybrid (C) | | --- | --- | --- | --- | | hit@3 | 0.880 | **0.936** | 0.924 | | MRR | 0.807 | **0.863** | 0.849 | The omnibus is significant (Cochran's Q = 7.95, p = 0.019). After Holm correction, the **only** pairwise gap that survives is **Hybrid > Wiki** (p = 0.010); Wiki↔RAG (p = 0.077) and Hybrid↔RAG (p = 0.71) are statistically indistinguishable. On natural paraphrases the three stacks land close together. The structural probes are where they separate. **What the per-type data shows:** - **Synonyms and unseen jargon are Wiki's weak spot.** When a query uses vocabulary absent from the source, keyword BM25 can miss where vector similarity matches: synonym_swap Wiki 0.84 vs RAG 0.94, with the industry-jargon probe the extreme (Wiki 0.40 vs RAG 1.00, Hybrid 0.60). Hybrid recovers part of RAG's edge while keeping wiki-Section citations. - **Cost is asymmetric, and the headline numbers hide it.** Wiki pays a one-shot LLM synthesis cost at ingest and then retrieves for free; RAG and Hybrid pay a per-query embedding cost forever. The retrieval scores don't capture this; the report's cost log does. - **Citation quality is Wiki's structural edge.** Wiki and Hybrid cite a stable `filename#heading` with `sources:` provenance that is grounding-checked at ingest; RAG cites raw chunks by similarity, usually losing section boundaries.   The honest reading: at this corpus scale (curated FAQ / policy, well under ~1000 pages), the differences on natural questions are small and mostly don't survive correction. Wiki trades a few points of recall for zero per-query cost, an inspectable index, and structured provenance; Hybrid buys back most of that recall while keeping the wiki-Section citation. The full methodology, statistical tests, cost log, and disclosed limitations are in [`eval/paraphrase_comparison/report.md`](eval/paraphrase_comparison/report.md); the design rationale is in [`why-wiki.md`](project-docs/why-wiki.md). To regenerate the corpus and re-run the comparison, see the maintainer runbook at [`eval/paraphrase_comparison/README.md`](eval/paraphrase_comparison/README.md). ### A fourth arm we measured but ship turned off: the cross-encoder reranker RRF is a recall step: it rescues a relevant Section that one method ranked low. It cannot fix the opposite failure, where a wrong Section sits high in the fused pool. That precision job belongs to a cross-encoder reranker, which re-scores the fused pool by reading the query and each Section together in one pass. We measured its full ceiling with the strongest open multilingual model (`bge-reranker-v2-m3`) as a fourth arm, **Stack C + rerank**, behind a flag that is off everywhere ([ADR-0019](project-docs/adr/0019-hybrid-reranker-eval-only-cross-encoder-default-off.md)). This project runs without it, but the number is still worth showing: it answers "would reranking help here at all?" with data instead of a guess.  The two halves of the chart tell different stories. On the adversarial probes the reranker earns its keep: `typo_fatfinger` climbs 0.40 → 1.00 and `industry_jargon` 0.60 → 0.80, because those are ranking failures and reranking is what fixes them. On natural Core paraphrases the lift is marginal. The macro-average hit@3 moves 0.924 → 0.960 (paired McNemar p = 0.004), which reads as already near the ceiling. So the ADR-0019 ship gate, a probe lift with no Core regression, is technically met. So why does the live tenant run without it? The reason is cost, not capability. The cross-encoder adds about 5.75 s per query on the dev box, and its `torch` plus 2.3 GB model will not fit the 512 MB per-tenant bulkhead on the shared VPS. At this corpus scale (~51 Sections, where the fused pool almost always already holds the gold Section), the marginal Core gain does not pay for a heavy dependency and a new failure surface. So it stays measured, flag-gated, and imported lazily: the production image never installs `torch`, and turning it on later would be a decision against these numbers rather than a rewrite. The full paired test, per-type tables, and dev-box latency live in [`report.md`](eval/paraphrase_comparison/report.md#reranker-evaluation-stack-c--stack-c--rerank). ## CLI and MCP reference The CLI (`kb`) and the MCP server (`kb_mcp`) drive the same lifecycle as the Console, over the same on-disk corpus. ### CLI subcommands (`uv run kb <subcommand>`) | Command | Description | | --- | --- | | `kb ask <question> [--stack wiki\|rag\|hybrid]` | Ask a question and print a grounded answer (LLM synthesis + Grounding Check). | | `kb index` | Rebuild the BM25 Section Index from the wiki corpus and persist it to `.kb/index.json`. | | `kb import <path>` | Import a local file (`.html`, `.txt`, `.pdf`, `.md`) into `docs/` via format conversion. | | `kb transcribe <path>` | Transcribe a scanned / text-less PDF into a `docs/` Markdown Source via a vision model. | | `kb ingest [source]` | Synthesise one named `docs/` Source (or all Sources when omitted) into `wiki/` pages. | | `kb lint` | Run the Lint Pass health check (the C1–C12 taxonomy: orphans, contradictions, stale pages, coverage gaps, …). | | `kb qa <list\|show\|promote\|discard>` | Inspect and curate the bot's filed Q&A: list drafts, show one, promote a good one to a permanent page, or discard it. | | `kb alias add <slug> <alias>` | Add a curator alias so a `[[wikilink]]` resolves to a page. | | `kb` (bare) | Enter the interactive REPL with a warm index; supports `:stack <wiki\|rag\|hybrid>` and `quit`. | ### MCP tools (`kb_mcp` server) | Tool | Description | | --- | --- | | `kb_ask_v1` | Ask a question and receive a grounded answer with citations and a Grounding Check result. Accepts `stack = wiki \| rag \| hybrid`. | | `kb_search_v1` | Retrieve raw Sections or Chunks with no LLM synthesis; returns BM25 scores for the keyword stacks. Accepts `stack = wiki \| rag \| hybrid`. | | `kb_read_hot_v1` | Read the working-memory hot cache (`wiki/hot.md`); returns `""` on the first session. | | `kb_save_hot_v1` | Persist a working-memory summary (composed by the host) to the hot cache. | | `kb_capture_v1` | Write a Markdown Source directly from conversation to `docs/`; stamps provenance frontmatter. | | `kb_import_v1` | Import a local file by absolute path into `docs/` via the same conversion pipeline as `kb import`. | | `kb_transcribe_v1` | Transcribe a scanned / text-less PDF (by absolute path) into a `docs/` Markdown Source via a vision model. | | `kb_ingest_v1` | Ingest a single named `docs/` Source into `wiki/` pages, auto-routing large Sources to a background job. | | `kb_ingest_start_v1` | Submit a Source to the background ingest job and return immediately. | | `kb_ingest_status_v1` | Poll the status of a background ingest job. | | `kb_index_v1` | Rebuild the BM25 Section Index and return `{files_indexed, sections_indexed}`. | | `kb_lint_v1` | Run the Lint Pass and return structured findings; can skip the LLM-backed contradiction check. | > **Read surface stays stable.** `kb ask`, `kb_ask_v1`, `kb_search_v1`, and the > Hot Cache pair keep their contracts; the Hybrid stack was added as a new `stack` > value, not a new read API. ### Concurrency recovery Concurrent writes from two interfaces (e.g. `kb_ingest_v1` from MCP while `kb ingest` runs in a terminal) can leave `.kb/index.json` stale. The index is fully regenerable: re-run `kb index` (CLI) or call `kb_index_v1` (MCP) to rebuild it from the wiki corpus. ## Deep dive - [`CONTEXT.md`](CONTEXT.md): the project's shared vocabulary. - [`PROMPT.md`](PROMPT.md): the exercise spec and design answers. - [`project-docs/adr/`](project-docs/adr/): the architectural decisions (ADR-0018 covers Hybrid; the curation layer — Reconcile, fix-source, Demote, aliases, PDF and longform ingest — spans ADR-0027–0038). - [`project-docs/roadmap.md`](project-docs/roadmap.md): the full implementation sequence. --- # Tool List [English](#knowledge-base-qa-bot) · **English** [](https://ask-wiki-rag.paynepew.dev)    A Markdown knowledge base-based, grounded question-answering bot. Each answer consists only of cited passages; if the sources can't support a claim, the bot replies _"I cannot confirm from the knowledge base"_, rather than guessing. The same question can be fed into **three retrieval engines**, with a toggle to compare their results side-by-side: | Engine | How it retrieves | Corpus | | --- | --- | --- | | **Wiki**(A) | BM25 keyword search on a curated `wiki/` layer (à la [Karpathy LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)) | `wiki/` Sections | | **RAG**(B) | Dense vector search (FAISS) on raw chunks | `docs/` Chunks | | **Hybrid**(C) | Simultaneous BM25 keyword and dense vector search (using the same tech as RAG) on the same curated `wiki/` layer, fused with Reciprocal Rank Fusion | `wiki/` Sections | > **"Hybrid" fuses two retrieval **methods**, not the other two "engines".** Its dense > arm uses the same vector retrieval tech as RAG (`text-embedding-3-small` + similarity), > but targets the curated `wiki/` Sections, not `docs/` chunks. Both arms return wiki > Sections with aligned IDs, so the fusion stays clean and wiki citations are preserved. > Directly merging the Wiki and RAG engines (different corpora, different units) is > explicitly rejected by [ADR-0018](project-docs/adr/0018-hybrid-retrieval-third-stack-rrf-over-wiki.md). **Live Demo: <https://ask-wiki-rag.paynepew.dev>.** Reader is at `/`, Operator Console is at `/console`. ## System Architecture A question enters from one of three interfaces, selects an engine, and all three eventually feed into the **same grounded answer pipeline**. Adding the Hybrid engine only required changing a dispatch entry point and adding a dense index and RRF fusion — the grounding check, citation, page expansion, and "I cannot confirm" gate were already written and shared among the three.  The browser connects to a FastAPI **Gateway**, which handles multi-turn query rewriting, session store, and production environment protection (daily budget + concurrency limit). The **CLI** (`kb`) and **MCP Server** (`kb_mcp`) bypass the Gateway and call the same three engines directly (single-turn). The only shared state between the three interfaces is files on disk — they intersect at the corpus layer. ### How Knowledge Enters Sources in `docs/` are immutable. **Ingest** combines them into a curated `wiki/` layer (with a Grounding Check for each page), and builds three indexes from this corpus. This is the write side of the Operator Console.  > The image files have editable Mermaid source code next to them: > [`runtime.mmd`](project-docs/architecture/runtime.mmd), > [`lifecycle.mmd`](project-docs/architecture/lifecycle.mmd). ## Technical Highlights - **Groundedness comes from architecture, not prompt engineering.** Before the LLM, there's a relevance gate that judges if the retrieved context is strong enough to answer; after the LLM drafts an answer, a second model call (**Grounding Check**) extracts atomic claims and checks them against cited Sections. If any claim isn't supported, the draft is discarded and the bot replies _"I cannot confirm"_. The system fails closed. - **Three retrieval architectures share a toggle**, with an honest, statistically validated benchmark (Cochran's Q omnibus → post-hoc McNemar + Holm correction), rather than picking a flashy headline number. - **Hybrid fusion on the same corpus.** Hybrid's two arms index the **same** wiki Sections, with 1:1 aligned IDs, so Reciprocal Rank Fusion (K=60) doesn't need to reconcile BM25 scores and FAISS distances across scales. The "Cannot-Confirm" gate applies an OR across each arm's corrected threshold — achieving recall-union without adding another threshold. - **Deep module reuse.** Each engine returns `Section` or `Chunk` objects that conform to the same `CitableContent` protocol, so grounding, citation, page expansion, and reject gates are written once and shared among the three. - **Bilingual support.** Files and questions can be in English or Traditional Chinese; the keyword engine uses character n-grams for CJK tokenization and language-specific scoring thresholds, giving Chinese questions Chinese answers. - **Three interfaces, one corpus.** Browser, CLI, and MCP Server can drive the full Upload → Import → Ingest → Index → Lint lifecycle; they only share state through the file system. - **Production-ready.** Containerized, with a daily image pull to a VPS (GHCR → SSH → `/healthz` smoke test). The `/healthz/shed` probe sheds reader traffic when overloaded; a daily USD budget gate and read/manage concurrency limit protect the machine. Retrieval indexes are submitted with the repo, so the first clone can answer questions. ## Agentic Workflow Orchestrator The product itself was **built by an agent** — feature development follows a controlled orchestrator ([`project-docs/agents/orchestrator.js`](project-docs/agents/orchestrator.js)), which pushes GitHub issues through to merged PRs in four stages:  The design follows Anthropic's [Building Effective Agents](https://www.anthropic.com/engineering/building-effective-agents) workflow/agent distinction: **the outer workflow is deterministic** — pure JavaScript control flow that expands a pipeline for each issue, applies severity gates, and reports results — so each run is inspectable, re-runnable (completed agents replay from cache), and cheap to re-run. **Autonomy is delegated to nodes**: each role-based agent works against a controlled prompt contract ([`project-docs/agents/`](project-docs/agents/)). - **Separation of duties, enforced by the platform.** Builders never endorse their own output: a contrasting **Verifier** (read-only; each blocker must be backed by evidence, only `critical`/`high` block) guards the branch; after a PR is opened, an **independent Verdict agent** re-verifies the pushed SHA and posts a `verify/verdict` commit status — the poster is the verifier, never constituting self-approval. Branch protection (Windows + Ubuntu dual-platform CI, plus this status) is a hard gate that no agent can bypass. - **Autonomy is a ladder, not a switch.** Rung 0 stops at a verified local branch; the default config opens a PR, posts a verdict, but leaves the merge button to humans; Rung 3 (opt-in progressively) pre-equips `gh pr merge --auto`, with required checks all green before GitHub auto-merges. Trust is incrementally increased; if issues arise, it's decreased. - **Model tiers are assigned by role.** Mechanical steps (planning, PR opening) use smaller models; execution uses medium-tier models; both verification steps use the strongest — with inference focused on the most costly errors. - **Safeguards are written into prompt contracts.** Each file-changing agent carries hard anti-pollution rules (never init another tracker, never mutate another slice's output or repo settings, never self-merge) — each tracing back to a real incident — and each build runs in an isolated git worktree. The same loop's manual version (one role at a time), handoff contracts, and stop conditions are written in [`project-docs/orchestration-plan.md`](project-docs/orchestration-plan.md). > The image files have editable Mermaid source code next to them: > [`orchestrator.mmd`](project-docs/architecture/orchestrator.mmd). ## Technical Choices Each choice below justifies its position — the notes explain "why this one, not that more obvious alternative", not just "what it is". | Layer | Chosen | Why this one | | --- | --- | --- | | Language & Tooling | **Python 3.11+**, **uv** workspace (one `.venv`, 7 members) | uv resolves the entire workspace at once; package members (`kb_cli`, `kb_mcp`) get console-script entry points, library members keep namespace-only. | | Web Framework | **FastAPI** + **Uvicorn** (ASGI) | Async streaming, typed request models, and an OpenAPI schema that can be fuzzed with an eval. | | Streaming | **Server-Sent Events**, custom parser | Answers stream character-by-character, sources first; one-way server push uses SSE over WebSocket, and the frontend doesn't need any library. | | Reader / Console UI | Pure **HTML + native JS**, no framework | Zero build steps, nothing to bundle, and inspectable page source like its driving indexes — a bet on "readability" like the retrieval side. | | Keyword Retrieval | **BM25**, custom implementation on heading Sections | A ~120-line scorer running on `.kb/index.json`, diagnosable with a `cat`; no `rank-bm25` black box, index is pure JSON. | | Dense Retrieval | **OpenAI `text-embedding-3-small`** + **FAISS** (`faiss-cpu`) | RAG and Hybrid's dense arm; FAISS lands in `.kb/faiss_index/` and `.kb/hybrid_dense/`, both submitted to repo, so cloning doesn't require prior index builds. | | Fusion | **Reciprocal Rank Fusion** (K=60), custom implementation | Rank-only fusion doesn't require reconciling BM25 scores and FAISS distances across scales — that's why Hybrid's two arms index the same wiki Sections. | | LLM Orchestration | **OpenAI SDK** via **LangChain-OpenAI**, structured output | Templates and Grounding Check use `with_structured_output`; LangChain only borrows at leaf nodes (loader, splitter), keeping it out of the domain core (ADR-0005). | | Import / Conversion | **markdownify** (`.html`), **MarkItDown** (`.pdf` text layer), visual models (scans PDF text) | Digital-native PDFs use the free mechanical path; only non-text scanned documents pay for the model (ADR-0031 / ADR-0032). | | Optional Reranker | **`bge-reranker-v2-m3`** cross-encoder via **sentence-transformers** | Only for evaluation, disabled by default, loaded with delay — `torch` and the 2.3 GB model never touch production images or VPS tenants (ADR-0019). | | CLI / MCP | **Typer** (`kb`), **MCP** SDK (`kb_mcp`) | Two more interfaces on the same disk corpus; no retrieval logic rewritten. | | Evaluation | **DeepEval** Synthesizer + custom `BaseMetric`, **matplotlib** plots, custom **Cochran's Q / McNemar / Holm** | Borrows framework's runner and dataset; claim-based metrics and statistical tests written for auditability. | | Delivery | **Docker** → **GHCR** → SSH image-pull CD, **Caddy** edge, **pytest** on Windows + Ubuntu matrix | Every `main` push rebuilds, deploys, and smoke-tests `/healthz`; edge sheds traffic and provisions three tenants. | ## Quick Start The retrieval indexes are **pre-built and submitted**, so cloning the repo lets you answer questions right away — no build steps required. ```bash # 1. Clone git clone https://github.com/PaynePew/knowledge_base_qa_bot.git cd knowledge_base_qa_bot # 2. Install (shared `.venv` for repo root) uv sync --all-packages # 3. Set your OpenAI key cp .env.example .env # then edit .env and fill in OPENAI_API_KEY=sk-... # 4. Start the Gateway (serves UI and three engines from the same origin) uv run uvicorn gateway.app.main:app --port 8000 ``` Open <http://localhost:8000/> to access the Reader, or <http://localhost:8000/console> to access the Console. > **About keys.** `OPENAI_API_KEY` is **required** — all three engines call OpenAI > to generate final answers, and RAG and Hybrid also use it for embedding. > `ANTHROPIC_API_KEY` is **optional**, only used by the cross-family Claude judge > for evaluation. See [`.env.example`](.env.example). ## Operating the Reader **Reader** — input questions, select engines, and read grounded answers.  - Input **any question** (in English **or** Chinese) in the input box and press Enter. - **Toggle between Wiki / RAG / Hybrid**, asking the same question to compare results side-by-side. - **Sources are shown first** — the cited passages that back up the answer — then streamed character-by-character below. - **Grounding badges** confirm each claim can be traced to a cited source; if not, you'll see _"I cannot confirm"_, rather than hallucinations. - **Follow-up questions** continue the same conversation (with multi-turn support in the URL); switching engines preserves chat history. ## Operator Console **Operator Console** — Backend interface: upload files, execute build processes, and maintain the health of company or personal knowledge bases.  The Wiki engine must go through a process flowchart to retrieve a file, with each step having its own **Run** button. **A file's journey — Upload → Import → Ingest → Index:** | Step | What it does | | --- | --- | | **Upload** | Drag and drop files. `.html` / `.txt` / `.pdf` into `raw/`; `.md` directly into `docs/`. | | **Import** | Convert sources in `raw/` to clean Markdown in `docs/` (including origin). Digital native PDFs use text layer extraction; non-text layer scanned documents use visual model **transcription** (ADR-0031 / 0032). | | **Ingest** | The LLM synthesizes sources in `docs/` into curated `wiki/` pages, performing grounding checks during writing. | | **Index** | Create BM25 indexes (and dense-over-wiki indexes) to enable new content to be queried. | **RAG track** (independent panel) rebuilds the FAISS vector index from `docs/`, unrelated to the Wiki process — just click **Rebuild**. **Long documents take a different shape.** A source without a usable title structure — a transcribed whole book, long reports — is identified as **longform** and specially routed: first, the LLM proposes chapter boundaries (**Structure Enrichment**), then Ingest writes a **Hub Page** ("What does this document talk about?") and one concept page per chapter, rather than collapsing the whole document into a single page (ADR-0033). **Linting and curation.** After editing the corpus or running **Lint** periodically, use **C1–C12** lint categories to audit the knowledge base's health: covering gaps and broken `[[wikilink]]`, unreferenced pages, slug collisions, contradictions between pages, outdated and isolated pages, alias collisions, and the health of Q&As archived by the robot itself. Each finding comes with an **executable fix**, categorized into four levels based on "how much human judgment is required": | Level | What you do | Examples | | --- | --- | --- | | **Direct** | One-click, no LLM call | Promote archived Q&A to permanent pages, demote live answers with illegal schema to drafts, assign **Alias**, re-ingest outdated pages, **Discard** drafts | | **Authored** | Robot drafts, you edit and apply | **Reconcile** two conflicting pages, **Merge / Differentiate** a slug collision | | **Confirmed** | You confirm an irreversible action | Delete an isolated page with a missing source | | **Routed** | The fix is outside the wiki layer, so the console directs you | Fix-the-Source when two sources really contradict each other (**source-rooted**) or a page cannot be grounded; **fill-via-import** when a query is completely uncovered | **Reconcile** is two-view: when contradictions can be fixed at the wiki layer, display two wiki pages side-by-side; when two **Source documents** themselves contradict, switch to **Source comparison** — allowing you to fix the root rather than the symptoms. Upgraded Q&As let high-quality answers gradually be incorporated into the knowledge base. The **Resource Browser** at the page bottom lets you browse `docs/`, `raw/`, and `wiki/` without leaving the page. ## Project Structure ``` knowledge_base_qa_bot/ ├── gateway/ # FastAPI Gateway + browser UI (Reader at /, Console at /console) — start here ├── markdown_kb/ # Wiki engine (A): BM25 + shared grounded answer pipeline on wiki/ layer ├── vector_rag/ # RAG engine (B): slice docs/ and embed into FAISS ├── hybrid_kb/ # Hybrid engine (C): BM25 on wiki/ + dense vector, RRF fusion ├── kb_cli/ # `kb` command-line interface ├── kb_mcp/ # kb_mcp MCP server (compatible with Claude Desktop) ├── docs/ # Sources — robot-executed, immutable knowledge base ├── wiki/ # Curated layer written by Ingest (concepts, entities, archived Q&As) ├── raw/ # Uploaded .html/.txt in Import's local inbox ├── .kb/ # submitted index: index.json (BM25), faiss_index/ (RAG), hybrid_dense/ (Hybrid) ├── eval/ # rewriting comparison tools + reports ├── project-docs/ # ADR, roadmap, coding standard, architecture diagrams, screenshots ├── CONTEXT.md # shared glossary └── .env.example # copy to .env and fill in your OpenAI key ``` ## Evaluation: Three Engines, Numbers Talk Can a curated Wiki + BM25 really outperform a general Vector RAG on the same raw corpus — and where does Hybrid stand? This benchmark answers with real numbers, not just claims. **Test data.** On a 20-Source / ~51 Gold Section corpus, with 260 queries: **250 Core rewrites** (5 types of LLM-generated rewrites × 50) plus **10 hand-written structural probes** (2 types × 5). A "hit" requires the retrieval result's source to match the gold section **and** share keywords, so "wrong document, right content" counts as a miss. **Methodology.** Each engine overfetches once per query on a deep candidate pool, decoupled from final cutoff; hit rates are reported across cutoff (hit@{1,3,5,10}) with MRR. **Cochran's Q** omnibus test ensures the pairwise **McNemar** (with Holm correction) — omnibus first says "some engine is different," then any pairwise claims. Dense arms use real OpenAI `text-embedding-3-small`. **Core results (macro average hit@3, real embedding):** | | Wiki(A) | RAG(B) | Hybrid(C) | | --- | --- | --- | --- | | hit@3 | 0.880 | **0.936** | 0.924 | | MRR | 0.807 | **0.863** | 0.849 | Omnibus significant (Cochran's Q = 7.95, p = 0.019). After Holm correction, the **only** surviving pairwise difference is **Hybrid > Wiki** (p = 0.010); Wiki↔RAG (p = 0.077) and Hybrid↔RAG (p = 0.71) are statistically indistinguishable. On natural rewrites, the three engines are close — structural probes really drive the gap. **Type-by-type data shows:** - **Synonyms and unseen jargon are Wiki's weaknesses.** When queries use vocabulary not seen in sources, keyword BM25 might miss — while vector similarity hits: synonym_swap Wiki 0.84 vs RAG 0.94, and "jargon" probes are extreme examples (Wiki 0.40 vs RAG 1.00, Hybrid 0.60). Hybrid retains wiki-Section references while recovering some RAG advantages. - **Costs are asymmetric, and headline numbers hide it.** Wiki pays a one-time LLM synthesis cost during ingest, then free retrieval; RAG and Hybrid pay embedding costs per query. Retrieval scores don't show this — cost logs in reports do. - **Citation quality is Wiki's structural strength.** Wiki and Hybrid cite stable `filename#heading`, with `sources:` provenance checked during ingest; RAG cites raw chunks by similarity, often losing section boundaries.   Honest interpretation: on this corpus scale (curated FAQ / policy, far below ~1000 pages), differences on natural questions are small and mostly pass correction. Wiki trades a few percent recall for zero-cost, inspectable indexes, and structured provenance; Hybrid buys back most recall while retaining wiki-Section references. Complete methodology, statistical tests, cost records, and honest limitations are in [`eval/paraphrase_comparison/report.md`](eval/paraphrase_comparison/report.md); design trade-offs are in [`why-wiki.md`](project-docs/why-wiki.md). To re-generate the corpus and re-run comparisons, see maintainer's guide [`eval/paraphrase_comparison/README.md`](eval/paraphrase_comparison/README.md). ### The fourth arm we measured but chose to turn off: cross-encoder reranker RRF is a recall step — it recovers sections down-ranked by one method, but can't fix the converse problem: a wrong section stuck in the fusion pool's top ranks. That's the precision job of the **cross-encoder reranker**: it pairs the query with each section, reads them together, and re-scores. We measured its complete potential using the strongest open multilingual model (`bge-reranker-v2-m3`) as a fourth arm — **Stack C + rerank** — hidden behind a **generally disabled** flag ([ADR-0019](project-docs/adr/0019-hybrid-reranker-eval-only-cross-encoder-default-off.md)). Even if this project doesn't enable it, these numbers are worth sharing because they data-drivenly answer "to rerank or not here".  The two halves of the plot tell different stories. On adversarial probes, the reranker justifies its cost — `typo_fatfinger` climbs from 0.40 to 1.00, `industry_jargon` from 0.60 to 0.80, because those are ranking errors and rerank directly addresses them. On **natural Core rewrites**, gains are marginal: macro average hit@3 from 0.924 to 0.960 (paired McNemar p = 0.004), which reads as "already near ceiling". ADR-0019's gating criteria — probes improve **and** Core doesn't degrade — are technically **met**. So why don't we enable it for online tenants? **It's cost, not capability.** This cross-encoder adds ~5.75 seconds per query on our dev box, and its `torch` + 2.3 GB model doesn't fit in each tenant's 512 MB container on shared VPS. At this corpus scale (~51 sections, fusion pool usually already contains the gold section), that marginal Core gain doesn't justify the added dependency and new failure mode. So this feature stays in "measured, flagged off, and lazy-loaded" state: production images never install `torch`, and any future enablement is a data-driven decision, not a rewrite. Complete pairwise tests, type-by-type tables, and dev-box latency are in [`report.md`](eval/paraphrase_comparison/report.md#reranker-evaluation-stack-c--stack-c--rerank). ## CLI and MCP Command Reference The CLI (`kb`) and MCP server (`kb_mcp`) drive the same lifecycle as the Console and operate on the same disk corpus. ### CLI Subcommands (`uv run kb <subcommand>`) | Command | Description | | --- | --- | | `kb ask <question> [--stack wiki\|rag\|hybrid]` | Ask a question and output a grounded answer (LLM synthesis + Grounding Check). | | `kb index` | Rebuild the BM25 Section Index from wiki corpus and write to `.kb/index.json`. | | `kb import <path>` | Import local files (`.html`, `.txt`, `.pdf`, `.md`) into `docs/` through format conversion. | | `kb transcribe <path>` | Transcribe scanned / non-text layer PDFs into Markdown Sources in `docs/` using visual models. | | `kb ingest [source]` | Synthesize specified `docs/` Source(s) (or all Sources if omitted) into `wiki/` pages. | | `kb lint` | Run Lint Pass health checks (C1–C12 categories: isolated pages, contradictions, outdated pages, coverage gaps…). | | `kb qa <list\|show\|promote\|discard>` | View and curate robot-archived Q&As: list drafts, show individual entries, promote good ones to permanent pages, or discard. | | `kb alias add <slug> <alias>` | Add a curated alias so `[[wikilink]]` can resolve to a page. | | `kb` (run directly) | Enter an interactive REPL with warm index; supports `:stack <wiki\|rag\|hybrid>` and `quit`. | ### MCP Tools (`kb_mcp` server) | Tool | Description | | --- | --- | | `kb_ask_v1` | Ask a question and get a grounded answer with citation and Grounding Check. Optional `stack = wiki \| rag \| hybrid`. | | `kb_search_v1` | Retrieve raw Sections or Chunks from index without LLM call; keyword engine returns BM25 scores. Optional `stack = wiki \| rag \| hybrid`. | | `kb_read_hot_v1` | Read the working memory hot cache (`wiki/hot.md`); returns `""` on first call. | | `kb_save_hot_v1` | Write the host-composed working memory summary to the hot cache. | | `kb_capture_v1` | Directly write Markdown Source to `docs/` from conversation; automatically add provenance frontmatter. | | `kb_import_v1` | Import local files into `docs/` through the same conversion process as `kb import`. | | `kb_transcribe_v1` | Transcribe scanned / non-text layer PDFs (absolute path) into Markdown Sources in `docs/` using visual models. | | `kb_ingest_v1` | Synthesize a single specified `docs/` Source into a `wiki/` page; large Sources are automatically offloaded to background jobs. | | `kb_ingest_start_v1` | Put a Source into a background ingest job and return immediately. | | `kb_ingest_status_v1` | Query the status of a background ingest job. | | `kb_index_v1` | Rebuild the BM25 Section Index and return `{files_indexed, sections_indexed}`. | | `kb_lint_v1` | Run Lint Pass and return structured results; can skip LLM-backed contradiction checks. | > **Read interface remains stable.** `kb ask`, `kb_ask_v1`, `kb_search_v1`, and hot cache combos retain existing contracts; Hybrid joins with a new `stack` value, not new read APIs. ### Concurrent Recovery If two interfaces write simultaneously (e.g., MCP's `kb_ingest_v1` and CLI's `kb ingest` run concurrently), the worst case leaves `.kb/index.json` in an outdated state. The index can be completely re-generated: re-run `kb index` (CLI) or call `kb_index_v1` (MCP) to rebuild from the wiki corpus. ## Further Reading - [`CONTEXT.md`](CONTEXT.md) — shared glossary for the project. - [`PROMPT.md`](PROMPT.md) — prompt specifications and design rationale. - [`project-docs/adr/`](project-docs/adr/) — Architecture Decisions (ADR-0018 covers Hybrid; curation layer — Reconcile, fix-source, Demote, alias, PDF and longform ingest — in ADR-0027–0038). - [`project-docs/roadmap.md`](project-docs/roadmap.md) — complete implementation sequence.