noema-ai

# NOEMA-AI An end-to-end Retrieval-Augmented Generation pipeline that ingests documents and answers queries grounded in their content. It supports **three retrieval paths** side by side — **vector-based** (embeddings + vector store), **vector-less** (keyword/BM25 structured retrieval), and **hybrid** (Reciprocal Rank Fusion of both) — orchestrated with **LangChain** and **LangGraph**, traced with **LangSmith**, and exposed to external clients via **MCP** (Model Context Protocol). It also supports **multi-turn chat with session memory** and **streaming answers** over Server-Sent Events. ``` Document ─▶ Ingestion ─▶ Chunking ─▶ ┬─▶ Embeddings ─▶ Vector Store ─┐ └─▶ BM25 Index (vector-less) ─┤─▶ LangGraph ─▶ LLM ─▶ Answer │ (retrieve node) Query ───────────────────────────────────────────────────────────────────┘ ``` ## Screenshots <!-- Drop your screenshot images into docs/screenshots/ using these exact filenames and they'll render automatically on GitHub — no other changes needed. Recommended: 1280px+ wide PNGs, dark mode (matches the UI theme). --> **Home page — landing view of the NOEMA UI**  **Console — ask a question, watch retrieval + generation trace live**  **Hybrid retrieval mode — fused vector + BM25 results with match source**  **Chat — multi-turn conversation with session memory**  **API reference — full endpoint interface view**  **Streaming answer — tokens arriving live over SSE**  ## Features - **Vector RAG** — chunking → embeddings → persistent vector store (Chroma) → similarity search. Ships with a zero-dependency offline hash embedding by default, and a real local semantic embedding option (`EMBEDDING_PROVIDER=sentence-transformers`) for actual semantic search - **Vector-less RAG** — BM25 keyword ranking over raw chunks, no embedding model required - **Hybrid RAG** — fuses the vector and BM25 result lists with Reciprocal Rank Fusion (RRF), so a chunk that ranks well in *either* retriever surfaces, and one that ranks well in *both* surfaces first - **Multi-turn chat with memory** — `POST /chat` keeps a rolling window of prior turns per session, so follow-up questions ("what about the second one?") resolve against earlier turns instead of being answered in a vacuum - **Streaming answers** — `GET /query/stream` (Server-Sent Events) streams the answer token-by-token instead of waiting for the full response - **Document management** — `GET /documents` lists every indexed source; `DELETE /documents/{source}` removes it from every retrieval path at once - **Query result caching** — a short-lived in-memory TTL cache serves repeated identical questions without re-running retrieval + the LLM; automatically invalidated on ingest/delete - **Persistence across restarts** — the BM25 corpus and the ingested-document manifest are saved to `PERSIST_DIR` and reloaded on startup, so a restart no longer silently wipes what was ingested - **LangGraph pipeline** — ingestion, retrieval (vector / vectorless / hybrid), and generation modeled as composable graph nodes, with automatic fallback from vector to vector-less - **LangSmith tracing** — every LLM call and retrieval step is traceable/evaluatable when configured - **LLM Gateway / Middleware** — a single layer that routes requests, logs, and rate-limits before calls reach the LLM - **Guardrails** — input/output checks with normalized (whitespace/zero-width-character-resistant) prompt-injection pattern matching; still pattern-based, not a full classifier — see the note below - **API key auth** — optional `X-API-Key` header enforcement on `/ingest`, `/query/*`, `/documents/*`, and `/chat/*`, off by default so the app still runs with zero setup - **MCP server** — exposes `ingest_document`, `delete_document`, `list_documents`, `query_vector_rag`, `query_vectorless_rag`, `query_hybrid_rag`, and `chat_with_memory` as MCP tools for external LLM clients - **Docker support** — a `Dockerfile` and `docker-compose.yml` for one-command containerized deployment with a persistent volume - **52 unit/API tests** covering the chunker, guardrails (including evasion attempts), the BM25 store, hybrid RRF fusion, the query cache, chat memory, and the FastAPI endpoints (including auth enforcement, document deletion, and chat sessions) ## Project layout ``` noema/ ├── app/ │ ├── config.py # environment / settings │ ├── main.py # FastAPI entrypoint (REST API) │ ├── middleware.py # LLM gateway / logging / rate-limit middleware │ ├── guardrails.py # input & output guardrails │ ├── cache.py # TTL query-result cache │ ├── chat.py # session-based conversational memory │ ├── ingestion/ │ │ ├── loader.py # load .txt / .pdf / .docx into raw text │ │ └── chunker.py # sliding-window chunking with overlap │ ├── retrieval/ │ │ ├── embeddings.py # embedding provider (API or local fallback) │ │ ├── vector_store.py # Chroma-backed Vector RAG │ │ ├── vectorless_store.py # BM25-backed Vector-less RAG (+ persistence) │ │ └── hybrid.py # Reciprocal Rank Fusion of vector + BM25 │ ├── llm/ │ │ └── client.py # LLM client (Groq / LLaMA 3.3 70B, mockable, streaming + history-aware) │ ├── graph/ │ │ └── pipeline.py # LangGraph state machine wiring it all together │ ├── mcp/ │ │ └── server.py # MCP server exposing the pipeline as tools │ └── static/ # web UI (served at "/" by app/main.py) │ ├── index.html │ └── assets/ │ ├── style.css │ └── app.js ├── docs/screenshots/ # drop screenshot images here (see Screenshots above) ├── data/sample_docs/sample.txt # example document to ingest ├── scripts/ingest.py # CLI: ingest a document from the command line ├── tests/ # 52 tests: chunker, guardrails, stores, hybrid, cache, chat, API ├── Dockerfile ├── docker-compose.yml ├── requirements.txt └── .env.example ``` ## Setup ```bash python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r requirements.txt cp .env.example .env # then fill in your keys ``` ### Environment variables (`.env`) | Variable | Purpose | |---|---| | `GROQ_API_KEY` | Key for Groq-hosted LLaMA 3.3 70B calls (leave blank to use the built-in mock LLM for local testing) | | `LANGCHAIN_TRACING_V2` | `true` to enable LangSmith tracing | | `LANGCHAIN_API_KEY` | LangSmith API key | | `LANGCHAIN_PROJECT` | LangSmith project name, e.g. `noema` | | `VECTOR_DB_PATH` | Local path for the persistent Chroma store, default `./chroma_db` | | `EMBEDDING_PROVIDER` | `hash` (default, deterministic offline bag-of-words embedding, zero deps) · `sentence-transformers` (real local semantic embeddings — `pip install sentence-transformers`, downloads a small model on first use) · `remote` (placeholder for a hosted embedding API you wire in) | | `EMBEDDING_MODEL_NAME` | Model name used when `EMBEDDING_PROVIDER=sentence-transformers`, default `all-MiniLM-L6-v2` | | `API_KEY` | Leave blank to run the API in open dev mode. Set a value to require a matching `X-API-Key` header on `/ingest` and `/query/*` | | `PERSIST_DIR` | Where the BM25 corpus and ingested-document manifest are saved, default `./storage`. Restoring on startup means a restart no longer wipes what was ingested | | `QUERY_CACHE_TTL_SECONDS` / `QUERY_CACHE_MAX_ENTRIES` | How long identical `(mode, query, top_k)` requests are served from cache, and how many entries the cache holds before evicting | | `CHAT_HISTORY_MAX_TURNS` / `CHAT_SESSION_TTL_SECONDS` | How many turns of conversation `/chat` keeps per session, and how long an idle session is kept before being swept | | `HYBRID_RRF_K` | Reciprocal Rank Fusion constant for `/query/hybrid`; higher values reduce how much rank position 1 vs 2 matters | ## Running it **1. Ingest a document** ```bash python scripts/ingest.py data/sample_docs/sample.txt ``` **2. Start the API (and the UI)** ```bash uvicorn app.main:app --reload ``` Open **http://127.0.0.1:8000** — this now serves the NOEMA web UI directly from the FastAPI app (`app/static/`). Drop a file on the ingest panel, pick a retrieval mode, and ask a question; the pipeline trace strip lights up as the request moves through guardrails → retrieval → generation, and results show which lane (vector / vector-less) answered along with the retrieved chunks and their scores. Endpoints: - `GET /health` — liveness + uptime - `GET /stats` — indexed document list, BM25 chunk count, vector store availability, active chat sessions, query cache stats - `GET /documents` — list every indexed source with its chunk count - `DELETE /documents/{source}` — remove a document from every retrieval path - `POST /ingest` — upload/ingest a document - `POST /query/vector` — answer a query using Vector RAG - `POST /query/vectorless` — answer a query using Vector-less (BM25) RAG - `POST /query/hybrid` — answer a query using Reciprocal Rank Fusion of both - `POST /query/graph` — run the full LangGraph pipeline (vector / vectorless / hybrid, with fallback) - `GET /query/stream` — Server-Sent Events: streams the answer token-by-token - `POST /chat` — multi-turn conversational endpoint with session memory - `GET /chat/{session_id}/history` — read back a session's turns - `DELETE /chat/{session_id}` — clear a session The UI is plain HTML/CSS/JS (no build step) so it runs from the same `uvicorn` process with no extra tooling. If you'd rather host it separately, set `window.__DOCUMIND_API_BASE__` in `app/static/index.html` before `app.js` loads, and point it at wherever the API is running — CORS is already open on the API side. **3. Run the MCP server** (so an MCP-compatible client can call this pipeline as tools) ```bash python -m app.mcp.server ``` **4. Or run it in Docker** ```bash docker compose up --build ``` This builds the image, mounts a named volume at `/data` for the BM25 corpus/Chroma store to persist across container restarts, and serves the API + UI on `http://localhost:8000`. Configuration still comes from `.env` (referenced via `env_file` in `docker-compose.yml`). ## Notes on the current state This is a working scaffold, not a black-box demo: the chunker, guardrails, BM25 retrieval, hybrid fusion, query cache, chat memory, and the FastAPI endpoints (52 tests total) run fully offline and are unit-tested. The vector store, LLM client, and MCP server are wired up with real integrations but need a `GROQ_API_KEY` (and, optionally, `chromadb`/LangSmith credentials) to hit real APIs. Without keys, the LLM client falls back to a mock responder so the pipeline still runs end-to-end for development and demos. Known, honestly-stated limitations: - **Embeddings**: the default `EMBEDDING_PROVIDER=hash` is a bag-of-words hash, not real semantic search — good for wiring/testing the pipeline offline, not for retrieval quality. Set `EMBEDDING_PROVIDER=sentence-transformers` for real semantic embeddings (needs an install + a one-time model download). - **Guardrails** are regex/pattern-based prompt-injection checks, not a trained classifier — they catch common phrasings and simple spacing/unicode evasions, not a determined adversary. Swap in a managed guardrails service for anything beyond a demo. - **Persistence is file-based, not a real database**: the BM25 corpus and ingested-document manifest are now saved to `PERSIST_DIR` and reloaded on startup (previously they were in-memory only and a restart silently wiped them), but this is still a JSON file on local disk, not a proper store — it won't behave correctly behind multiple worker processes or hosts. Chat sessions and the rate limiter are still in-memory-only by design (short-lived, session-scoped state that doesn't need to survive a restart). - **API auth** is a single shared `X-API-Key`, not per-user auth — fine for a demo or an internal tool, not a substitute for real auth in a multi-tenant deployment. - **The query cache is a simple in-memory TTL dict**, not a distributed cache — fine for a single-process deployment, and it's invalidated on every ingest/delete so it can't serve a stale answer past a document change, but it won't be shared across multiple worker processes. A dependency-free web UI sits in front of all of this (`app/static/`, served at `/`), and several functional bugs were fixed while wiring it up so the demo actually works out of the box: `python-multipart` was missing from `requirements.txt` (FastAPI needs it for file uploads — `/ingest` would 500 on boot without it), the BM25 vector-less path was silently dropping every hit on small corpora (with only one or two chunks indexed, BM25's IDF term can go negative for common words, so the old `score <= 0` filter discarded true matches — it now ranks by score but only discards chunks that share no terms with the query), and the frontend was reading `stats.vectorless_chunk_count` / `stats.ingested_documents` while `/stats` returned differently-named fields, so the document list and chunk count in the UI always silently showed zero regardless of what had actually been ingested.