Content
# TraceAlign Lab — Agentic Eval & Alignment Platform
Demo URL: Fill in the public network address of Alibaba Cloud after deployment.
An automated evaluation and alignment optimization platform for complex reasoning tasks of large models. It completes GraphRAG retrieval, MCP tool calls, trajectory error analysis, DPO data generation, and comparison evaluation through a closed loop of "evaluation Agent + optimization Agent".
## Core Capabilities
- FastAPI + React trusted evaluation console: real-time SSE, Suite A/B, operation history, and export.
- 5 built-in complex Agent evaluation tasks: `data/samples/`.
- Real data upload entrance: supports JSON / JSONL / YAML / CSV / TSV / TXT / MD / images / PDF / DOCX / ZIP, preview before evaluation.
- DeepSeek `deepseek-chat` client (`OPENAI_API_KEY` + `OPENAI_BASE_URL`).
- Real MCP HTTP tool calls (`MCP_BASE_URL`), explicit fallback when unavailable.
- Hierarchical GraphRAG Local + Global Search.
- Self-developed error analyzer and failure trajectory DPO data generator (visible in UI + run-bound export).
- Naive baseline vs Full Supervisor same scorer comparison.
- QLoRA DPO fine-tuning script, reusing `Qwen/Qwen2.5-7B-Instruct`, without CPT.
- `python eval.py --mode sample` outputs baseline vs improved measurement report.
## Quick Start (Windows One-Click)
Default browser (夸克) or `.bat` association may be damaged, **prior double-click**:
1. `Open local website.vbs` (recommended, forced to use cmd to start, avoid "application not found")
2. Or right-click `start.bat` → Open with Command Prompt
- `start.bat` / `Open local website.vbs`: start **MCP(:8100) + backend(:8000) + frontend(:3000)** simultaneously, print MCP/backend health status, and open http://127.0.0.1:3000 with **Edge/Chrome**
- `stop.bat` / `Stop service.vbs`: stop MCP / backend / frontend (8100/8000/3000)
- If double-clicking `.bat` still prompts "application not found": right-click to run `scripts\fix_bat_association.bat` as administrator
Manual start:
```bash
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
python scripts/build_graphrag_index.py
python eval.py --mode sample
uvicorn backend.main:app --reload
```
Frontend:
```bash
cd frontend
npm install
npm run dev
```
Docker:
```bash
cp .env.example .env
docker-compose up --build
```
## Evaluation
```bash
python eval.py --mode sample
python eval.py --mode upload --path data/samples/multimodal/tasks.csv
python eval.py --mode upload --path data/samples/multimodal/auth_card.png
```
Output: `evals/METRICS.md`.
## Fine-Tuning
```bash
python finetune/train_dpo_qlora.py --dry-run
python finetune/train_dpo_qlora.py --data data/dpo_data/dpo_pairs.jsonl
```
## Documentation
- `docs/PRD.md`
- `docs/DESIGN.md`
- `docs/ARCHITECTURE.md`
- `docs/PIPELINE.md`
- `docs/LLM_STACK.md`
- `docs/IMPROVEMENTS.md`
- `docs/METRICS.md`
- `docs/DEPLOY.md`
## Metric Criteria
Unmeasured metrics are written as "target% + measurement method". Local sample evaluation will fill in reproducible results to `evals/METRICS.md`, without fabricating online DAU or irreproducible accuracy.
