cowork_bench

# Cowork Bench [](./LICENSE) **EN:** Cowork Bench is a multi-tool agent benchmark: **496 end-to-end tasks** that mirror real office workflows — pull data from a mock database, build Excel/Word/PPTX artifacts, schedule calendar events, send emails — executed through **MCP (Model Context Protocol) servers** (~120 tools visible per task). Evaluation is **fully deterministic** (programmatic checks + SQL side-effect verification, no LLM-as-judge) and runs entirely locally: one Docker Compose, a seeded PostgreSQL, zero external API calls at eval time. All task statements and system prompts are in **Russian** (identifiers stay Latin for exact matching), with Russia-specific domains (railways, MosBirzha, 1C HR, e-commerce). Requirements: Docker + bash; see the Quick Start below. Built on top of [Toolathlon](https://github.com/hkust-nlp/Toolathlon) (Apache 2.0). ## Tool List — A multi-tool benchmark for LLM agents: real-world workflow tasks, local PostgreSQL, no external APIs during evaluation. Training and evaluating LLM agents on real tool usage is challenging: existing datasets are either narrow in tool coverage, small in scale, or rely on live external APIs that change over time. **Cowork Bench** is a large, self-contained set of tasks with a rich mock database that runs entirely locally: no external calls are required during evaluation. All tasks and system prompts are in **Russian**; identifiers (file names, sheet names, column names, email addresses, ticker symbols, status slugs) are preserved in Latin for deterministic verification. Each task requires the agent to perform an end-to-end goal — e.g., extract data from a mock database, build an Excel report, schedule a calendar event, and send an email — using a fixed set of MCP servers (Model Context Protocol) as tools. Each task is fully automated: `preprocess/main.py` prepares the initial workspace state, the agent executes the task using available tools, and `evaluation/main.py` verifies the result against the ground truth (`groundtruth_workspace/`). No manual labeling or live external services are required. The benchmark stresses practically important abilities: multi-step planning across heterogeneous tools, reading and writing structured formats, synchronizing data between systems, and completing long tasks under a fixed step budget. ## Runner The repository contains two runners: - **CAMEL** (`main.py`) — default: single pass agent + evaluation, minimal boilerplate. - **Strands** (`main_strands.py`) — phase-aware runner with ground truth isolation (agent does not see `groundtruth_workspace/` and `evaluation/`), robust streaming, and retries; used to obtain solo results in the table below. Enable with `AGENT_ENTRY=main_strands.py`, `AGENT_PHASE_AWARE=1` + `LLM_*` variables — see [`docs/STRANDS.md`](./docs/STRANDS.md). The default runner is the **CAMEL agent** (`main.py`). Task completion is determined by the model finishing its turn (stopping tool calls) — **no `claim_done` tool is required**: the `SUCCESS` status is set when the agent completes its turn without new tool calls (see `utils/roles/task_agent.py`). Evaluation is performed by the task evaluator based on side effects, not by calling a signaling tool. The model and provider for the CAMEL runner are specified (in order of priority): 1. environment variables `MODEL_NAME` and `MODEL_PROVIDER` (read by `main.py`); 2. command-line arguments `--model_name` / `--provider`; 3. `scripts/eval_config.json` file (fallback). The API key and endpoint are taken from `MODEL_API_KEY` / `MODEL_API_URL`. Allowed values for `MODEL_PROVIDER`: `openai`, `anthropic`, `gemini`, `deepseek`, `aihubmix`, `openai_compatible`. ## Quick Start Only **Docker** and **bash** are required. Everything else (Python, MCP servers, migrations) is inside containers. ```bash cp .env.example .env # fill in MODEL_* (see below) docker build -t cowork-pack:latest . # build image (once, ~10–15 min) ./cowork_bench up # start Postgres (cowork_pg) + cowork_net ./cowork_bench test-db # smoke test db/* migrations on fresh PG ./cowork_bench run <task-name> # single task run (agent + evaluation) ./cowork_bench stability <task> 5 # 5 consecutive runs, PASS/FAIL summary ./cowork_bench bench 3 # parallel run of all tasks (concurrency=3) ./cowork_bench tasks [pattern] # task list (optionally by substring) ./cowork_bench stop # emergency cleanup of containers + lock ./cowork_bench help # full command list ``` `./cowork_bench up` runs `docker compose up -d` (Postgres `cowork_pg`, network `cowork_net`, and optional dev-shell container `cowork`). For a single task run (`run`), it's sufficient for `cowork_pg` to be healthy and the `cowork_net` network to exist — `scripts/run_containerized.sh` automatically starts ephemeral agent containers for each task. `cowork_bench` sources `.env` and uses the CAMEL runner by default (`AGENT_FRAMEWORK=camel`, `IMAGE=cowork-pack:latest`); any defaults can be overridden via `.env`/env. Environment quirks are documented in [`docs/ENVIRONMENT_QUIRKS.md`](./docs/ENVIRONMENT_QUIRKS.md). ### Minimal `.env` for CAMEL Runner ```bash MODEL_PROVIDER=openai_compatible # openai | anthropic | gemini | deepseek | aihubmix | openai_compatible MODEL_NAME=your-model-name MODEL_API_KEY=sk-REPLACE_ME MODEL_API_URL=https://your-llm-endpoint/v1 # required for openai_compatible / aihubmix ``` ### Evaluation With the default CAMEL runner, evaluation is performed **automatically in the same run**: `main.py` calls the evaluator and writes the verdict to `eval_res.json` after the agent finishes. No separate command is required. The standalone grader `scripts/run_eval.py` (and separate eval container in `run_containerized.sh`) is used only for phase-aware runners with ground truth isolation (see "Connecting Your Runner"). ### Where to View Results ``` dumps/<task>/<timestamp>/.../SingleUserTurn-<task>/ ├── traj_log.json # config + status log ├── traj.json # full trajectory (messages + tool calls) ├── eval_res.json # {"pass": true|false, "details": "..."} └── workspace/ # files created by the agent ``` Aggregate parallel run output: `benchmark_logs/fully_parallel_<timestamp>/summary.csv`. ## Connecting Your Runner `scripts/run_containerized.sh` selects the agent entrypoint as follows: - `AGENT_ENTRY=<path>` — explicit path to your runner script (highest priority); - otherwise `AGENT_FRAMEWORK=strands` → `main_strands.py`; - otherwise default → `main.py` (CAMEL). Engine contract: - accepts `--task_dir <task>` and `--max_steps <N>`; - writes `traj_log.json` and `workspace/` directory under `/workspace/dumps` — found by the grader `scripts/run_eval.py`. **Ground truth isolation (optional).** To ensure the agent does not see `groundtruth_workspace/` and `evaluation/`, the run is split into two phases (agent in a sanitized container → separate evaluation). This is enabled if the runner is `main_strands.py` or if `AGENT_PHASE_AWARE=1` is set; the agent must support the `--phase agent` flag. Without this, a single pass is performed (agent + evaluation together, full mount, **without** isolation). ```bash # custom runner with ground truth isolation AGENT_ENTRY=my_runner.py AGENT_PHASE_AWARE=1 \ bash scripts/run_containerized.sh <task-name> 100 ``` The runner and grader code are baked into the image (`Dockerfile: COPY . .`), so the built image works without host mounts. For rapid iteration on the runner code, use `DEV_MOUNTS=1` (overlay host code). ## Architecture: MCP Servers `configs/mcp_servers/*.yaml` registers an MCP server, and `local_servers/<name>/` contains its code. The backend for "SQL-like" MCPs is the shared PostgreSQL `cowork_pg`, with a schema per MCP (e.g., `canvas`, `moex`, `wc`, `hr1c_data`). Data is loaded deterministically: **`db/init.sql.gz`** — a full dump (all schemas, including Russian data `rzd`, `teamly`, `moex`, `hr1c`, `gform`, canvas relabeling, `wc`), automatically applied by the Postgres docker-entrypoint on the first start. To add a new domain, fork the MCP in `local_servers/`, write a post-init migration `db/zzz_<name>_after_init.sql` (prefix `zzz_` guarantees loading **after** `init.sql.gz` — entrypoint sorts files alphabetically), register it in `docker-compose.yml` (volume in `/docker-entrypoint-initdb.d/`) and the server in `configs/mcp_servers/<name>.yaml`. A ready-to-use recipe is in [`docs/RUSSIFICATION.md`](./docs/RUSSIFICATION.md). ## Task Structure All tasks are located in `tasks/finalpool/`. Each task directory has a unified layout: ``` <task-name>/ ├── task_config.json # MCP servers and local tools available to the agent ├── docs/ │ ├── task.md # task description (displayed to the agent) │ └── agent_system_prompt.md ├── evaluation/main.py # automatic evaluator ├── preprocess/main.py # preparation of database state (runs before the task) ├── initial_workspace/ # input files in the agent's workspace └── groundtruth_workspace/ # reference outputs for verification ``` Task descriptions (`task.md`) are written **without branded tool names** — services like knowledge base, common calendar, or LMS are described in a generalized manner. This is the same obfuscation convention as in the original Toolathlon project: it prevents the agent from cutting corners by recognizing key words and forces it to think about the actual use of tools. ## Mock Database **Connection**: `cowork_gym` database on the `cowork_pg` container (within the network — `cowork_pg:5432`, from the host — `localhost:15433`; user `eigent`, password `camel`). All data is provided by a local PostgreSQL, initialized from a compressed dump (`db/init.sql.gz`) and post-init migrations. There are no external calls at runtime — the environment is fully controlled, without API limits, schema changes, or data drift. Data is derived or simulated from real sources: **Kaggle OULAD** (Open University Learning Analytics) for LMS, **Kaggle HR Analytics** for corporate HR, stock quotes for finance, a combination of **Kaggle Amazon** and **DummyJSON** for e-commerce, and Russian forks (RZD, knowledge base, forms). ### Data-Rich Schemas | Domain | Description | Schema | |-------|----------|-------| | **canvas** | LMS — courses, users, records, assignments, submissions, tests, categories, announcements (rusified names/realities) | `canvas` | | **moex** | Exchange — quotes, financial statements, news, options, holders | `moex` | | **woocommerce/insales** | E-commerce — products, orders, customers, coupons, reviews | `wc` | | **rzd** | Railway — stations, trains, routes, places | `rzd` | | **teamly** | Knowledge base (Russian analog of Confluence) | `teamly` | | **forms** | Surveys/forms on PG backend | `gform` | | **hr1c** | Corporate HR | `hr1c_data` | | **youtube** | Video platform — channels, playlists, videos, transcripts | `youtube` | ## Results Runs from June to July 2026, all 496 tasks, metric **pass@1**. Method **solo** — single-threaded runs with Strands runner (concurrency 1): clean pass@1 with zero infra stalls. Method **parallel** — earlier multi-threaded runs with ex-stall correction (see footnotes). | Model | Window | PASS@1 | Method | |---|---|---|---| | **MiniMax-M3** (428B-A23B, FP8) | 400k | 323 / 496 — **65.1%** | parallel | | **Qwen 3.5 397B-A17B** (BF16, MoE) | 262k | 312 / 496 — **62.9%** | solo | | **Qwen 3.6 27B** (Dense, FP8) | 262k | 309 / 496 — **62.3%** | solo | | **Qwen 3.6 35B-A3B** (FP8, MoE) | 262k | 300 / 496 — **60.5%** | solo | | **GigaCode Agentic v1.6** \* (80B-A3B, FP8) | 262k | 120 / 421 — **28.5%** | parallel, ex-stall | | **GigaChat-3.5** (432B-A28B) \*\*\* | 130k | 119 / 496 — **24.0%** | solo | | **GigaChat-3-Max** \*\* (90B Dense, FP8) | 130k | 41 / 375 — **10.9%** | parallel, ex-stall | | **GigaChat-3-Pro** \*\*\* | 130k | 24 / 496 — **4.8%** | solo | > \* **GigaCode Agentic v1.6** — infra failures excluded: 75 stream-stall. Metric **28.5%** = 120/421 reached eval (raw pass@1 = 120/496 = 24.2%). > > \*\* **GigaChat-3-Max** — infra failures excluded: 59 stream-stall + 22 × HTTP 500 + 32 task_timeout. Metric **10.9%** = 41/375 reached eval (raw pass@1 = 41/496 = 8.3%). > > \*\*\* **GigaChat-3.5 / 3-Pro** — run through legacy `functions`-API with translating proxy (**1 tool per turn** — structural handicap); actual window ~130k. 3-Pro — without reasoning. The benchmark shows a large spread in model strength (65% → 5%); the Qwen family clusters at 60–63% regardless of size (27B ≈ 35B ≈ 397B in honest single-thread). ## What Distinguishes Cowork Bench - **Scale and diversity.** Hundreds of tasks across dozens of MCP servers and multiple data domains; tasks require actual cross-system coordination, not just single lookups. - **Fully local and reproducible.** The entire environment is set up from a single Docker Compose. No external service keys are needed during evaluation. The PostgreSQL dump is versioned and deterministic — results are reproducible across machines and time. - **Realistic complexity.** Tasks are based on real workflows: HR analytics in Excel, verifying LMS submissions against calendar deadlines, generating presentations from collected data, and similar multi-step goals. Most tasks require 4–7 tools. ## Acknowledgements Cowork Bench is built on the infrastructure and data pipelines of the project: > **Toolathlon: Benchmarking LLM Agents on Real-World Tool-Use Tasks** > HKUST-NLP — https://github.com/hkust-nlp/Toolathlon The mock database schema design, MCP server interfaces, and evaluation framework are from the Toolathlon project (Apache License 2.0, see [`LICENSE`](./LICENSE)). Cowork Bench localizes the task pool to Russian, replaces some external services with Russian forks, and expands the dataset. The `local_servers/` catalog contains vendored forks of third-party open-source MCP servers (filesystem, excel, word, powerpoint, playwright, fetch, etc.) — authorship and licenses are specified in their own manifests within the corresponding subdirectories.