Content
# Orchestrion
**A controlled LLM agent environment that provides safe and reliable responses based on verifiable sources.**
When integrating LLMs into services, what's most important? **Sometimes, you need trustworthy answers over fluent but incorrect ones.** A single sentence that seems plausible but lacks basis can lead to accidents.
Typically, a single agent model makes judgments, calls tools, and generates answers on its own. You can only hope it doesn't fabricate. Orchestrion inverts this structure. The code determines the sequence, and models make their own judgments at each stage. Therefore, all responses pass through an input gate, attach evidence to each sentence, and undergo output verification before reaching the screen.
This is a sample structure that can be run with Claude Code and a few dependencies. It includes controlled agent routing, guardrails, and knowledge base (KB) based responses bundled into a single scenario. The interface is in English, and a chemical safety document is provided as an example, but the engine is domain-agnostic and can be applied to any markdown document collection.
![Orchestrion Process Dashboard — A question passes through a node graph, citing evidence [S#]·[DB#] and undergoing output verification](docs/dashboard.jpg)
> A question goes through 6 stages: input gate → planning → evidence collection → review → output verification.
> Every factual sentence in the response is tied to a document chunk (`[S#]`) or a structured fact (`[DB#]`).
> The right panel shows executed SQL, queried fact tables, tool calls, and guardrail decisions in real-time.
## Core Principles
- **The bridge, not the model, orchestrates.** Each node is a single LLM call that performs one specific task (gate → routing → planning → execution → review → verification). The Python bridge determines which node comes next based on rules, not the model. This results in a controllable and reproducible process.
- **Responses must be verifiable.** Execution nodes provide evidence for each factual sentence — `[S#]` for document chunks or `[DB#]` for structured data. If there's no supporting evidence, it clearly states "not found in provided materials." A separate verification node cross-checks the draft with evidence.
- **All tool calls go through a single gate.** Every tool invocation passes through a policy hook, which reads the rules repository anew each time it's called. This allows dynamic rule updates.
- **The system doesn't break if components are missing.** If an embedding model isn't available, search falls back to BM25. Even without knowledge, the graph runs and honestly reports it doesn't know.
- **Identity is defined by content, not code.** An agent is defined by `scenario.md`, `knowledge/raw/`, `data/demo.json`, and guardrail policies. New tasks are created by updating content, not rewriting the engine.
## Architecture
### Execution Flow
A question flows through this node graph. **The bridge determines the path and sequence based on rules; each node makes its own judgment using a model.**
```mermaid
flowchart LR
Q([Question]) --> RMI["Input Gate<br>rm_input"]
RMI -. REJECT .-> DECL([Polite Rejection])
RMI -- ALLOW --> RT{"Routing<br>route"}
RT -- Simple --> SA["Simple Answer<br>simple_answer"]
RT -- Task --> PL["Planning<br>plan"]
PL --> WK["Evidence Collection & Draft<br>worker"]
WK --> RV["Review<br>review"]
RV -. "REDO ≤ 2" .-> WK
RV --> RMO["Output Verification<br>rm_output"]
SA --> RMO
RMO --> ANS([Verified Response])
classDef rm fill:#fde2e2,stroke:#c0392b,color:#7b241c;
classDef hyp fill:#e2ecfd,stroke:#2c5aa0,color:#1a3c6e;
classDef wrk fill:#e6f6ea,stroke:#27ae60,color:#186a3b;
class RMI,RMO rm;
class RT,SA,PL,RV hyp;
class WK wrk;
```
### Nodes — What, Which Model
| Node | Lane | Role | Model |
|---|---|---|---|
| `rm_input` | Risk Management | Determines if a question is within scope — rejects politely if not | haiku |
| `route` | Overview | Branches to simple answer or task | haiku |
| `simple_answer` | Overview | Short answers to simple queries like greetings | sonnet |
| `plan` | Overview | Plans search strategy based on task procedures | sonnet |
| `worker` | Execution | Gathers evidence, drafts response with `[S#]`/`[DB#]` citations | sonnet |
| `review` | Overview | Reviews draft for relevance and citations — up to 2 retries if inadequate | haiku |
| `rm_output` | Risk Management | Verifies draft against evidence, fact-checks, and produces JSON output | haiku |
The three **lanes** (`Risk Management` / `Overview` / `Execution`) are color-coded on the dashboard.
### Verification Contracts — `[S#]` and `[DB#]`
The most meticulous part of this project. Execution nodes can cite two types of evidence:
- **`[S#]`** — Document chunks. Sourced from hybrid search results, displayed in orange.
- **`[DB#]`** — Structured fact table rows. Accurate numbers or properties queried via SQL, shown with executed queries and result tables in yellow.
Sentences without evidence aren't allowed; instead, it reports "not found in provided materials." Finally, the `rm_output` node verifies each factual sentence against evidence.
### Knowledge Pipeline
```
knowledge/raw/*.md
│ ingest (make ingest)
▼
Chunking ─→ FTS5 Index (BM25) + Embedding (bge-m3) + LLM Fact Extraction
▼
SQLite (docs · chunks · chunks_fts · embeddings · facts)
▲
│ worker's 4 read-only tools have access (MCP)
│ search_knowledge · read_doc · query_facts · list_sources
```
Search combines BM25 and cosine similarity with reciprocal rank fusion (RRF). If an embedding model isn't available, it falls back to BM25.
### Components
| File | Role |
|---|---|
| `bridge/graph.py` | Node graph + model tiered warm session pool (maintained between runs, new session ID for history isolation per node) |
| `bridge/hooks.py` · `bridge/policy.py` | Single tool gate + editable first-match rule repository |
| `bridge/api.py` · `bridge/app.py` | REST API + WebSocket event feed, static front-end serving |
| `bridge/scenario.py` | Parses `scenario.md` to inject runtime prompts into graph |
| `core/db.py` · `core/search.py` · `core/embed.py` | SQLite schema · hybrid search · embedding (with downgrade support) |
| `ingest/` | Markdown → chunking → FTS → embedding → fact loading |
| `mcpserver/server.py` | FastMCP — exposes 4 knowledge tools to workers |
| `frontend/` | React dashboard showing the graph in real-time |
Detailed component descriptions and invariants are in [`CLAUDE.md`](./CLAUDE.md).
## Quick Start
Prerequisites: [uv](https://docs.astral.sh/uv/)(Python 3.11), Node 18+, Claude subscription or API key, and [ollama](https://ollama.com/) + `bge-m3`(optional).
```bash
cp .env.example .env # Optionally, set ANTHROPIC_API_KEY
make setup # uv sync
make ingest # knowledge/raw/ → kb.sqlite
./start.sh # FastMCP + bridge, front-end build, and auto-open browser
```
### Accessing the Interface
`./start.sh` builds the front-end and starts the bridge. The dashboard is accessible at **http://localhost:8787** (automatically opens; use `./start.sh --no-open` to prevent).
| Screen | What to See |
|---|---|
| **Process** | Main screen showing the node graph in real-time, citations, and verification decisions |
| **Knowledge** | Browse loaded documents, chunks, and facts; try hybrid search |
| **Guardrail** | Toggle tool policy rules, view allow/deny logs |
For front-end development, use `./start.sh --dev` for Vite's HMR server at **http://localhost:5173**, with `/ws` proxied to the bridge.
To test without the interface:
```bash
python scripts/graph_probe.py "What is the maximum daily handling amount and flash point of substance A?"
curl -s localhost:8787/api/kb/stats
```
## Adapting to Your Knowledge
1. Place well-organized `.md` files in `knowledge/raw/`. (PDF parser intentionally omitted; convert to markdown first.)
2. Rewrite `scenario.md` (summary/target/forbidden + `task procedure:` bullets). This content becomes the graph prompt at runtime.
3. Run `make ingest` and restart.
Created bundles are saved as snapshots in `scenarios/<name>/` and can be swapped out as needed. The included `scenarios/chemical/` example demonstrates this.
## License
MIT — see [LICENSE](./LICENSE).
