Content
# Acme Operations Copilot
Enterprise agentic assistant for **Acme Operations** — grounded customer support via LangGraph tool calls, PostgreSQL, Redis, Keycloak, and MCP.
## Quick start
```bash
cp .env.example .env # add OPENAI_API_KEY or use Ollama
make demo # stack + demo seed data
```
| Service | URL |
|---------|-----|
| **Command Centre** | http://localhost:5173/dashboard |
| **Web UI** | http://localhost:5173 |
| **API** | http://localhost:8000/docs |
| **Dashboard API** | http://localhost:8000/api/dashboard/summary |
| **Keycloak** | http://localhost:8080 (`bob` / `bob123`) |
| **Jaeger (traces)** | http://localhost:16686 |
| **Grafana (LGTM)** | http://localhost:3002 (admin / `GRAFANA_ADMIN_PASSWORD`) |
| **Prometheus** | http://localhost:9090 |
| **Langfuse** | http://localhost:3001 |
| **Arize Phoenix** | http://localhost:6006 |
See [docs/architecture.md](docs/architecture.md) and [docs/tradeoffs.md](docs/tradeoffs.md).
## Repository layout
```
acmeops-copilot/
├── apps/
│ ├── api/ # FastAPI + LangGraph agent
│ │ ├── main.py
│ │ ├── config.py
│ │ ├── auth/ # keycloak, rbac, dependencies
│ │ ├── agent/ # graph, tool_router, runner
│ │ ├── prompts/ # versioned YAML prompts (Git source of truth)
│ │ ├── services/ # chat_service, prompt_service
│ │ ├── routers/ # chat controller (HTTP layer)
│ │ ├── skills/ # customer_escalation_summary
│ │ ├── memory/ # redis_store, checkpoint, conversations
│ │ ├── telemetry/ # tracing, metrics, logging, audit
│ │ ├── schemas/ # chat, customer, issue
│ │ ├── support/ # db, middleware, exceptions, evidence
│ │ └── tests/
│ ├── mcp-server/ # Acme MCP tools (SSE)
│ │ ├── server.py
│ │ ├── tools/ # customers, issues, next_actions
│ │ ├── db.py
│ │ └── telemetry.py
│ └── web/ # React UI (demo)
├── infra/
│ ├── postgres/ # init.sql, seed.sql, migrations
│ ├── keycloak/ # realm-export.json
│ ├── redis/
│ ├── knowledge/ # SLA docs (filesystem MCP)
│ ├── mcp/ # filesystem, postgres, search MCP servers
│ ├── otel/
│ └── kubernetes/ # K8s manifests, Ingress LB, Headlamp — see infra/kubernetes/README.md
├── evals/
│ ├── dataset.json
│ ├── eval_questions.json
│ ├── run_eval.py
│ └── eval_results.md
├── docs/
│ ├── architecture.md
│ ├── architecture-diagram.mmd
│ ├── ai-usage-notes.md
│ ├── threat-model.md
│ └── tradeoffs.md
├── docker-compose.yml
├── Makefile
└── README.md
```
## Commands
```bash
make up # docker compose up --build
make demo # bootstrap + seed PostgreSQL demo data
make reseed-db # reload demo customers/issues
make eval # run evals/run_eval.py (10-question suite)
make logs # follow api logs
make logs-otel # collector + jaeger + phoenix + api trace logs
make otel-ui # print Jaeger URL
make phoenix-ui # print Phoenix URL
```
## Kubernetes (production infra)
Load-balanced deployment on Kubernetes with **NGINX Ingress**, **cert-manager TLS**, **HPA**, **NetworkPolicies**, and **Headlamp** (replaces deprecated Kubernetes Dashboard):
```bash
make k8s-bootstrap # kind cluster + ingress-nginx + cert-manager + Headlamp
make k8s-secrets # generate random DB/Redis passwords (gitignored)
make k8s-deploy # build images + apply local overlay (HTTPS via mkcert)
```
Full guide: [infra/kubernetes/README.md](infra/kubernetes/README.md).
## Testing
| Layer | Command | Notes |
|-------|---------|--------|
| **Ruff** (Python) | `make lint` | `ruff check` + `ruff format --check` on API, MCP, evals |
| **ESLint** (web) | `make lint` | `apps/web` — `npm run lint` |
| **API unit** | `make test` | FastAPI smoke + RBAC/skills (no database) |
| **API integration** | `make test-integration` | Requires Postgres on `127.0.0.1:5433` (from `make up` or `docker compose up postgres -d`) |
| **Web unit** | `make test-web` | Vitest + Testing Library |
| **E2E** | `make test-e2e` | `docker-compose.e2e.yml` + Playwright (Keycloak login) |
```bash
# One-time: Python dev deps + web packages
python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt
cd apps/web && npm install && npx playwright install chromium
make lint # Ruff + ESLint
make test # API unit + Vitest
make test-integration # API against seeded Postgres
make test-e2e # Full browser stack (then make test-e2e-down)
```
CI runs Ruff, API unit, API integration (GitHub Actions Postgres/Redis), web lint/typecheck/Vitest, and Playwright E2E.
## Observability
Four tracing backends are supported, each independently enable/disable via `.env`:
| Provider | Env flag | UI |
|----------|----------|-----|
| **OpenTelemetry** → Jaeger + Tempo | `ENABLE_OTEL=true` | http://localhost:16686 (Jaeger), Grafana → Tempo |
| **Grafana LGTM** | Docker stack (default) | http://localhost:3002 |
| **Langfuse** (self-hosted v3) | `ENABLE_LANGFUSE=true` | http://localhost:3001 |
| **LangSmith** (cloud) | `ENABLE_LANGSMITH=true` + `LANGSMITH_API_KEY` | https://smith.langchain.com |
| **Arize Phoenix** | `ENABLE_PHOENIX=true` | http://localhost:6006 |
`OBSERVABILITY_PROVIDER` filters which providers are active:
- `all` (default) — all enabled flags apply
- `both` — legacy: Langfuse + LangSmith only
- comma-separated, e.g. `langfuse,otel` or `phoenix,langsmith`
**What gets traced:** HTTP requests, outbound HTTP, Postgres, Redis, agent runs, tool calls (with latency), RBAC checks, and errors. Local JSONL logs are written to `evals/logs/traces-*.jsonl`.
**Disable examples:**
```bash
ENABLE_OTEL=false # no Jaeger spans
ENABLE_LANGFUSE=false # no Langfuse callbacks
ENABLE_LANGSMITH=false # no LangSmith export
ENABLE_PHOENIX=false # no Phoenix / OpenInference
OBSERVABILITY_PROVIDER=otel # only OpenTelemetry when enabled
```
## OpenTelemetry
Traces flow **API → OTLP collector → Jaeger + Tempo (+ Phoenix)**. Metrics and logs are available in **Grafana** (see [Grafana LGTM](#grafana-lgtm-stack)). Enabled by default in Docker (`ENABLE_OTEL=true`).
1. Start the stack: `make demo` (or `docker compose up -d jaeger otel-collector api`)
2. Open **Jaeger**: http://localhost:16686
3. Select service **`acme-assistant-api`**, click **Find Traces**
4. Send a chat message in the UI (or `curl /api/health`) to generate spans
**What gets traced:** HTTP requests (FastAPI), outbound HTTP (httpx), Postgres (asyncpg), Redis, agent runs (`agent.run`), and tool calls (`tool.*`).
## MCP (agent integration)
Four MCP servers run in Docker; the **chat agent loads their tools** when `ENABLE_MCP_AGENT_TOOLS=true` (default):
| Server | Port | Agent tool prefix (examples) |
|--------|------|------------------------------|
| Custom Acme | 8090 | `acme-custom_acme_get_customer_by_name`, `acme-custom_acme_list_open_issues`, … |
| Filesystem | 8091 | `filesystem_fs_read_file`, `filesystem_fs_list_directory` |
| PostgreSQL | 8092 | `postgres_postgres_query` (SELECT only) |
| Search (RAG) | 8093 | `search-mock_search_knowledge_semantic`, `search-mock_get_knowledge_chunk`, `search-mock_rag_index_status` |
Native tools (`get_open_issues`, `create_next_action`, …) remain for **RBAC, HITL, and audit**. MCP tools add policies, runbooks, SQL exploration, and **semantic RAG** (Qdrant-backed knowledge search).
### RAG (Qdrant)
| Step | Command / location |
|------|-------------------|
| Vector DB | Docker service `qdrant` (internal only); K8s `qdrant` StatefulSet |
| Ingest | `make rag-ingest` or automatic on `mcp-search` startup when index is empty |
| MCP tools | `search_knowledge_semantic`, `get_knowledge_chunk`, `rag_index_status` |
| Config | `QDRANT_URL`, `QDRANT_API_KEY`, `EMBEDDING_MODEL`, `EMBEDDING_DIMENSION` |
Without Azure/OpenAI embeddings, the ingest/search path uses a deterministic hash embedding for local demos.
- Status: `GET /api/mcp/status` (SSE reachability + agent tool load count)
- Disable MCP in agent: `ENABLE_MCP_AGENT_TOOLS=false`
**Correlate a request:** API responses include an `X-Trace-Id` header. Paste that trace ID into Jaeger’s **Trace ID** search.
**In the UI:** Assistant sidebar → **Trace status** shows OTel/Jaeger links. **Evaluations** page has the same observability panel.
**Disable:** set `ENABLE_OTEL=false` in `.env` and restart the API.
## Grafana (LGTM stack)
Unified **metrics, logs, and traces** for platform performance monitoring. LLM-specific debugging remains in Phoenix/Langfuse.
| Component | URL | Role |
|-----------|-----|------|
| **Grafana** | http://localhost:3002 | Dashboards (metrics + logs + traces) |
| **Prometheus** | http://localhost:9090 | Scrapes `/metrics` + OTel collector |
| **Loki** | internal `:3100` | Container log aggregation (Promtail) |
| **Tempo** | internal `:3200` | Distributed traces (OTLP from collector) |
### Quick start
```bash
make demo # starts prometheus, loki, promtail, tempo, grafana
make grafana-ui # print Grafana URL + credentials hint
```
1. Open **Grafana**: http://localhost:3002 (default `admin` / `admin`, override with `GRAFANA_ADMIN_PASSWORD`)
2. Open dashboard **Acme API — Performance** (folder: Acme Operations)
3. Send chat traffic — panels update for HTTP latency, agent runs, tool calls
4. Use **Explore → Loki** for JSON logs (`trace_id`, `request_id` labels)
5. Click **TraceID** in logs to jump to **Tempo** (or use Jaeger datasource)
### Configuration
```bash
GRAFANA_UI_URL=http://localhost:3002
GRAFANA_ADMIN_PASSWORD=admin
LOG_FORMAT=json # structured logs for Loki (default in Docker)
```
**Data flow:** API → OTLP collector → Jaeger + Tempo + Phoenix; Promtail → Loki; Prometheus scrapes API `/metrics` and OTel `:8888`.
**Kubernetes:** `kube-prometheus-stack` Grafana + Loki (`make k8s-bootstrap`) with Tempo in `acme-ops` namespace. Port-forward: `make k8s-grafana`.
## Arize Phoenix
Phoenix traces LLM/agent workflows using **`arize-phoenix-otel`** (`phoenix.otel.register`) and OpenInference auto-instrumentation. Enabled by default in Docker (`ENABLE_PHOENIX=true`).
### Configuration
```bash
ENABLE_PHOENIX=true
PHOENIX_COLLECTOR_ENDPOINT=http://phoenix:6006/v1/traces # Docker
# PHOENIX_COLLECTOR_ENDPOINT=https://phoenix.acme.local/v1/traces # local HTTPS ingress
PHOENIX_UI_URL=http://localhost:6006
PHOENIX_PROJECT_NAME=acme-ops
```
`PHOENIX_COLLECTOR_ENDPOINT` must be the **fully qualified HTTP OTLP URL** (`.../v1/traces`). If you pass only the host (e.g. `https://phoenix.acme.local`), the API appends `/v1/traces` automatically.
### How it works
1. On startup, `telemetry/phoenix_tracing.py` calls `phoenix.otel.register()`:
```python
from phoenix.otel import register
tracer_provider = register(
project_name=settings.phoenix_project_name,
endpoint=settings.phoenix_collector_endpoint,
auto_instrument=True,
batch=True,
)
tracer = tracer_provider.get_tracer("acme.agent")
```
2. Agent entry points are wrapped with `@tracer.chain` (via `phoenix_chain`):
- `run_agent` — full chat/agent run
- `invoke_agent_graph` — single ReAct LangGraph path
- `invoke_multi_agent` — supervisor multi-agent path
3. When **both** Phoenix and OpenTelemetry are enabled, Phoenix registers the global `TracerProvider` first; Jaeger export is attached to the same provider (dual export: Phoenix UI + Jaeger).
### View traces
1. Start Phoenix: `make demo` (or `docker compose up -d phoenix api`)
2. Open **Phoenix UI**: http://localhost:6006 (or `make phoenix-ui`)
3. Select project **`acme-ops`** (or your `PHOENIX_PROJECT_NAME`)
4. Send a chat message — spans appear for agent chains, LangChain/LangGraph calls, and tools
**Kubernetes (local HTTPS):** Phoenix UI at https://phoenix.acme.local — set `PHOENIX_UI_URL` and collector endpoint in `infra/kubernetes/base/config/app-config.yaml`.
**Kubernetes Grafana LGTM:** https://grafana.acme.local (or `make k8s-grafana`). Deploy runs `make k8s-sync-observability` automatically.
**Disable:** `ENABLE_PHOENIX=false` or `OBSERVABILITY_PROVIDER=otel,langfuse` (exclude `phoenix`).
## Evaluation
Run the 10-question assessment suite (tool selection, groundedness, RBAC, next-action quality):
```bash
make eval
# or: python evals/run_eval.py
```
Questions live in `evals/eval_questions.json`. Results are written to `evals/results.json` and `evals/eval_results.md`.
Metrics measured per question:
- **Tool selection** — expected tools invoked for the query
- **Groundedness** — response tied to database tool results
- **RBAC** — role-based access respected (deny / approval flows)
- **Next actions** — recommended follow-ups are reasonable when requested
**Local API (outside Docker):** point export at the collector on localhost:
```bash
ENABLE_OTEL=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
JAEGER_UI_URL=http://localhost:16686
```
## Demo prompts
Sign in as `bob` / `bob123` and try:
1. *"Show me open customer issues for AsteriaPay Global, summarise the latest status, and suggest the next action."*
2. *"Who is the account owner for AsteriaPay Global?"*
3. *"Run customer escalation summary for AsteriaPay Global."*
Demo data: **12+ customers**, **35+ issues**, and pending next actions across FinTech, healthcare, retail, travel, and utilities. Flagship account **AsteriaPay Global** (`ASTERIAPAY`) has **ISSUE-1001–1003** (`infra/postgres/seed.sql` + `07-extended-demo-seed.sql`). Schema design: [docs/database-schema.md](docs/database-schema.md).
**Azure:** Set `AZURE_OPENAI_*` in `.env` for chat (`DEFAULT_PROVIDER=azure`), voice (mic in assistant), and image generation. See `.env.example`.
## Assessment mapping
| Requirement | Location |
|-------------|----------|
| LLM agent + dynamic tools | `apps/api/agent/` |
| Reusable skills (5 workflows) | `apps/api/skills/` — escalation, SLA, triage, handoff, health |
| MCP server | `apps/mcp-server/` + `infra/mcp/` |
| PostgreSQL schema & seed | `infra/postgres/`, `docs/database-schema.md` |
| Keycloak RBAC | `infra/keycloak/`, `apps/api/auth/` |
| Evaluation | `evals/` (10-question suite) |
| Observability | Langfuse, LangSmith, Phoenix, OTel/Jaeger/Tempo, Grafana LGTM — `apps/api/telemetry/` |
| Architecture & AI notes | `docs/` |
