agentforge

# AgentForge **Production-grade enterprise AI agent platform** — design, deploy, orchestrate, and operate multi-agent systems with the reliability, observability, and governance patterns teams expect in production. Built as a full-stack reference implementation covering the end-to-end AI engineering lifecycle: from LLM routing and agent runtime, to durable workflows, tool integration, memory, evaluation, and live observability. --- ## Why AgentForge? Most AI demos stop at a chatbot wrapper. AgentForge goes further — it models how **real teams ship agentic systems**: | Challenge | AgentForge approach | |-----------|---------------------| | Agents fail mid-run | **Temporal** durable workflows survive restarts; runs resume after crashes or human delays | | No visibility into agent behavior | **OpenTelemetry + Langfuse** tracing with live WebSocket streaming to the dashboard | | Unsafe tool execution | Permission tiers, human-in-the-loop approvals, schema validation, and audit trails | | Prompt/model drift | Versioned prompts with diff, labels, and A/B routing; runs pin exact versions | | Can't measure quality | Evaluation framework with datasets, LLM judges, regression gates, and model comparison | | Vendor lock-in | Unified **Model Gateway** across OpenAI, Anthropic, Gemini, and Ollama with fallbacks | | Integration sprawl | **MCP (Model Context Protocol)** support — discover and invoke tools from any compliant server | --- ## Core Capabilities ### Agent Runtime - Create, version, and publish agent definitions with model config, tools, memory, and guardrails - Compile agents into **LangGraph** ReAct loops with budget limits, max-step guards, and approval gates - Interactive **playground** for testing agents before deployment ### Multi-Agent Orchestration - **Planner → Worker → Aggregator** pattern for decomposing complex goals into task DAGs - Parallel fan-out, sequential pipelines, and supervisor routing - Human approval checkpoints that can pause runs for hours and resume via API signal ### Visual Workflow Builder - Drag-and-drop workflow designer (**React Flow**) for composing agent, LLM, tool, router, map, and approval nodes - Workflows compile to **Temporal** definitions for durable, long-running execution ### Tool Registry & MCP Integration - Unified invocation path for **native**, **HTTP**, and **MCP** tools - JSON Schema validation on inputs and outputs - MCP server registration with auto-discovery over stdio, SSE, and HTTP transports - Permission model: `safe`, `requires_approval`, `dangerous` ### Memory System - **Short-term** conversation buffer (Redis) - **Long-term** semantic memory (Qdrant vector store) - Pluggable embedders, reranking, token-budget recall, and async summarization ### Prompt Registry - Versioned prompt templates with Jinja rendering - Label-based routing (`production`, `challenger`) and weighted A/B testing - Diff view and render preview before publish ### Evaluation Framework - Dataset management with manual import or production trace harvesting - Evaluators: heuristic, LLM-judge, code-based, and human rating - Parallel eval runs via Celery with regression gates and model/agent comparison matrices ### Observability & Analytics - Distributed tracing with cost and latency per LLM call and tool invocation - Live run streaming over WebSocket (Redis pub/sub) - Analytics dashboard for run volume, cost, latency, and approval metrics ### Enterprise Foundations - Multi-tenant auth (JWT, orgs, API keys, role-based access) - Rate limiting, idempotency keys, and token/cost budgets - Scheduled runs (cron, interval, webhook triggers) - Docker Compose full-stack deployment --- ## Use Cases AgentForge is designed for scenarios where reliability, auditability, and human oversight matter: | Use case | How AgentForge helps | |----------|---------------------| | **Customer support automation** | Agents with memory recall prior interactions; dangerous actions (refunds, account changes) require human approval | | **Research & analysis pipelines** | Planner decomposes a research goal; worker agents search, summarize, and aggregate in parallel | | **Internal copilots** | Connect MCP servers (GitHub, Slack, databases) as tools; govern access with permission tiers | | **Document processing workflows** | Visual workflow builder chains extract → classify → route → approve steps with full traceability | | **Model/prompt experimentation** | Prompt versioning + eval framework compare quality, cost, and latency before promoting to production | | **Scheduled reporting agents** | Cron-triggered runs generate and deliver reports with durable execution and failure recovery | | **Compliance-sensitive automation** | Immutable version pinning, approval audit trails, and trace replay for every decision | --- ## Architecture AgentForge follows **hexagonal (ports & adapters) architecture** — domain logic is framework-agnostic, infrastructure is swappable, and each bounded context is a separable Python package. ``` ┌─────────────────────────────────────────────────────────────────┐ │ Next.js Dashboard (agents, workflows, runs, evals, analytics) │ └────────────────────────────┬────────────────────────────────────┘ │ REST + WebSocket ┌────────────────────────────▼────────────────────────────────────┐ │ API Gateway (FastAPI) — auth, rate-limit, routing, WS hub │ └──┬──────────┬──────────┬──────────┬──────────┬─────────────────┘ │ │ │ │ │ Agent Execution Workflow Tool Reg Prompt/Eval Runtime Engine Engine + MCP / Memory │ │ │ │ │ └──────────┴──── LangGraph + Model Gateway ─┘ │ ┌──────────────┼──────────────┐ │ │ │ Temporal Celery Postgres (durable) (async jobs) Qdrant + Redis ``` **Key design decisions:** - **Temporal** for durable, stateful agent runs (survives restarts, supports human-in-the-loop waits) - **Celery** for fire-and-forget background work (embeddings, eval fan-out, notifications) - **Modular monolith** — bounded contexts as packages, deployable as separate containers without code changes - **Async throughout** — async SQLAlchemy, httpx, and LangGraph nodes Full system design: [docs/01-ARCHITECTURE.md](docs/01-ARCHITECTURE.md) --- ## Tech Stack | Layer | Technologies | |-------|-------------| | **Backend** | Python 3.12+, FastAPI, Pydantic, SQLAlchemy (async), Alembic | | **Agent framework** | LangGraph, LangChain | | **Orchestration** | Temporal, Celery | | **LLM providers** | OpenAI, Anthropic, Gemini, Ollama (unified gateway) | | **Databases** | PostgreSQL, Redis, Qdrant | | **Frontend** | Next.js, React, TypeScript, TanStack Query, React Flow, Tailwind CSS | | **Observability** | OpenTelemetry, Langfuse | | **Tooling** | MCP SDK, Docker Compose, uv, pytest | --- ## Skills Demonstrated This project is a portfolio piece targeting **AI Engineer** and **ML Platform** roles. It demonstrates: - **LLM application architecture** — model routing, streaming, fallbacks, cost accounting, and budget enforcement - **Agent design patterns** — ReAct loops, multi-agent planner/worker orchestration, tool use, and memory integration - **Production engineering** — durable workflows, human-in-the-loop gates, multi-tenancy, auth, and rate limiting - **RAG & memory systems** — vector retrieval, embedding pipelines, conversation buffers, and summarization - **MCP & tool integration** — schema-validated tool registry with industry-standard protocol support - **Evaluation & quality** — LLM-as-judge, regression gates, and systematic model/prompt comparison - **Observability** — distributed tracing, live streaming, and cost/latency analytics - **Full-stack delivery** — API design, WebSocket real-time updates, and a polished operator dashboard - **Software architecture** — clean/hexagonal architecture, domain-driven bounded contexts, and async Python --- ## Quick Start ### Prerequisites Docker, Python 3.12+, Node.js 18+, [uv](https://github.com/astral-sh/uv) ### Run locally ```bash # 1. Environment cp .env.example .env # 2. Infrastructure docker compose up -d postgres redis qdrant temporal temporal-ui # 3. Python dependencies pip install uv && uv sync --all-packages # 4. Database migrations uv run alembic -c migrations/alembic.ini upgrade head # 5. API server cd apps/gateway && uv run uvicorn main:app --reload --port 8000 # 6. Frontend (new terminal) cd apps/web && npm install && npm run dev ``` Open **http://localhost:3000** — register an account, create an agent, and run it in the playground. ### Full stack (all services) ```bash docker compose up ``` ### Development commands ```bash make test # unit tests make migrate # alembic upgrade make seed # demo user + sample data ``` --- ## Services | Service | Port | Description | |---------|------|-------------| | API Gateway | 8000 | REST API + WebSocket streaming | | Frontend | 3000 | Next.js operator dashboard | | Temporal UI | 8080 | Workflow inspector | | PostgreSQL | 5432 | System of record | | Redis | 6379 | Cache, pub/sub, Celery broker | | Qdrant | 6333 | Vector memory store | --- ## Documentation | Doc | Topic | |-----|-------| | [01-ARCHITECTURE](docs/01-ARCHITECTURE.md) | System design, service boundaries, data flows | | [05-AGENTS](docs/05-AGENTS.md) | Agent lifecycle, ReAct loop, multi-agent patterns | | [06-WORKFLOW-ENGINE](docs/06-WORKFLOW-ENGINE.md) | Visual workflow builder and compilation | | [07-MEMORY](docs/07-MEMORY.md) | Short-term, long-term, and episodic memory | | [08-TOOLS](docs/08-TOOLS.md) | Tool registry, permissions, MCP integration | | [09-PROMPTS](docs/09-PROMPTS.md) | Prompt versioning, labels, A/B routing | | [10-EVALUATION](docs/10-EVALUATION.md) | Datasets, evaluators, regression gates | | [11-TRACING](docs/11-TRACING.md) | OpenTelemetry, Langfuse, live streaming | | [13-ROADMAP](docs/13-ROADMAP.md) | Implementation milestones and future work | --- ## Project Structure ``` agentforge/ ├── apps/ │ ├── gateway/ # FastAPI API gateway │ ├── web/ # Next.js dashboard │ ├── agent_worker/ # Temporal worker │ └── celery_worker/ # Background task worker ├── packages/ │ ├── agents/ # Agent runtime & LangGraph compiler │ ├── execution/ # Run orchestration engine │ ├── model_gateway/ # Unified LLM provider abstraction │ ├── tools/ # Tool registry + MCP client │ ├── memory/ # Redis + Qdrant memory service │ ├── prompts/ # Prompt versioning & routing │ ├── workflows/ # Visual workflow compiler │ ├── evaluation/ # Eval datasets & runners │ └── ... # auth, core, planner, scheduler ├── docs/ # Architecture & design docs ├── migrations/ # Alembic database migrations └── tests/ # Unit and e2e tests ``` --- ## License MIT