Content
# AI Note Backend
AI-powered todo list application backend service, also a hands-on project for learning AI Agent architecture.
Built with **FastAPI + LangGraph + LLM + lazy tool loading**, supporting voice input, smart task management, human approval (HITL), and DingTalk task synchronization.
---
## Features
| Feature | Description |
|---|---|
| 💬 **Conversational Task Management** | Natural language input, LLM automatically extracts/updates/deletes tasks |
| 🎤 **Speech-to-Text** | Upload audio → Groq Whisper → text → automatic task extraction |
| ✋ **Human-in-the-Loop (HITL)** | Task change proposals require human confirmation/editing/rejection to take effect |
| 📋 **Task CRUD** | REST API direct operation on tasks (query, modification, deletion) |
| 🔗 **DingTalk Synchronization** | One-click synchronization of approved tasks to DingTalk to-do list |
| 🧠 **Long-term Memory** | User profile, task history, and preference instructions persisted to PostgreSQL |
| 🔧 **MCP Tool Ecosystem** | Dynamically loads external tools (DingTalk, RAG, etc.) via MCP protocol |
---
## Technical Highlights
### Architecture
```
app/ FastAPI Web layer (routing, services, dependency injection)
packages/ Agent core layer (graph, tools, middleware, config)
```
Web layer and Agent layer are decoupled, allowing the Agent layer to evolve and test independently.
### LangGraph Orchestration Engine
- Not a simple LLM + tools loop, but a graph structure with **explicit topology control**
- Conditional routing: `START → transcription? → agent → tools → HITL? → agent → END`
- Transcription subgraph has independent private state and checkpoint namespace
- Supports `interrupt()` suspension → human approval → `Command(resume=True)` resumption
### Middleware Pipeline
```
ErrorHandlingMiddleware ← outermost: catches all exceptions
└── MemoryLoadMiddleware ← loads profile/tasks/instructions from store
└── SystemPromptMiddleware ← builds system prompt with memory
└── ToolBindingMiddleware ← binds tools to ChatOpenAI
└── LLM invocation
```
Each middleware handles a single concern, and through pipeline composition, can be tested and replaced independently.
### Tool Binding Strategy
- **Core tools** (8): always bound, including task CRUD and web search
- **MCP dynamic tools**: loaded at startup, promoted on demand via `tool_search`
- Solves GLM-5.1's ~55 tool limit, while DingTalk MCP provides ~98 tools
### Human-in-the-Loop (HITL)
- TrustCall deterministic extraction of task changes → generates proposals (`json_doc_id` for replay protection)
- `interrupt()` suspension, frontend displays diff → user approves/edits/rejects
- Resumes from checkpoint, does not re-call LLM, ensuring consistency
- Optional DingTalk synchronization: automatically pushes approved changes to DingTalk to-do list
### Persistence and Fault Tolerance
- PostgreSQL (Supabase) stores business data (profile/tasks/instructions)
- LangGraph checkpointing stores conversation state
- Database connection failure automatically falls back to in-memory storage (developer-friendly)
- Health check validates store read/write at startup
---
## What You Can Learn
### Agent Orchestration
- **LangGraph graph design**: building complex agent control flow with `StateGraph` + conditional edges
- **Subgraph isolation**: private state, independent checkpoint, namespace management
- **interrupt/resume pattern**: complete HITL implementation, including resumption and deterministic key strategy
- **Middleware vs Graph Nodes**: when to use middleware (cross-cutting concerns), when to use graph nodes (topology control)
### Tool System
- **MCP protocol integration**: loading external tools, dynamic registration, on-demand promotion
- **Tool selection strategy**: core tools vs lazy loading, solving LLM tool limit
- **TrustCall deterministic extraction**: preventing key drift during HITL resumption with `json_doc_id`
### Persistence and Configuration
- **Dual storage**: business data (PostgresStore) + conversation state (checkpointer) separation
- **Graceful degradation**: automatic fallback to in-memory storage when `DATABASE_URL` is not configured
- **App restart**: MCP tool loading and graph recompilation (hot reload)
### Speech and RAG
- **Groq Whisper integration**: end-to-end chain of audio → text → structured task extraction
- **Hybrid retrieval**: Qdrant dense+sparse hybrid search for knowledge base Q&A
### Engineering Practices
- **uv workspace monorepo**: independent packages for `app/` and `packages/harness/`, managed through workspace
- **FastAPI lifespan mode**: asynchronous resource initialization (MCP tool loading, graph recompilation)
- **Idempotent startup**: `store.setup()` automatically creates tables, safe for multiple startups
---
## Quick Start
```bash
# 1. Install dependencies
uv sync
# 2. Configure environment variables
cp .env.example .env
# Edit .env: fill in GLM_API_KEY, TAVILY_API_KEY, etc.
# 3. Start service
uv run uvicorn app.main:fastApi --reload --host 0.0.0.0 --port 8000
```
## Tech Stack
| Layer | Technology |
|---|---|
| Web Framework | FastAPI |
| Agent Orchestration | LangGraph (StateGraph + checkpointing) |
| LLM | GLM-5.1 (via OpenAI compatible API) |
| Tool Definition | LangChain tools + MCP |
| Structured Extraction | TrustCall |
| Database | PostgreSQL (Supabase), can run without database during development |
| Vector Retrieval | Qdrant + Qwen text-embedding-v4 |
| Speech Recognition | Groq Whisper |
| Web Search | Tavily |
| Package Management | uv (workspace monorepo) |
## Project Structure
See [AGENTS.md](AGENTS.md).
