gofer

# Tool List Put configurable **CLI Agent** (`codex` / `claude` / `opencode` / arbitrary commands) with registered **projects**, bridge to a unified **asynchronous job control surface**: submit `{project, agent, prompt/command, cwd}`, gofer executes in the project's real working directory (local / remote worker / peer container), unify status, logs, exit code, results back, and can be submitted and observed via **CLI / HTTP / MCP / Web Console** four interfaces. > gofer = "a person who runs errands": send tasks to agents → execute in target project directory → return logs/results. With a double entendre of Go `gofer`↔`gopher`. ## Overview of Capabilities - **Multi-Entry Control Surface**: CLI (`gofer job ...`) / HTTP (`/v1/jobs`) / MCP (stdio tools) / Web Console (including submission form), same `job.Service`. - **Multi-Agent**: `type: cli-agent` (template rendering `--prompt`, e.g., codex/claude/opencode) and built-in `type: exec` (run argv as is). Uninstalled agents are marked `unavailable`, not affecting startup. - **Multi-Project**: each project registers `host_path`/`container_path` + allowed agents/runners + `allow_exec` + concurrency limit. - **Three Execution Locations (runners)**: `local` (this process) / `peer-http` (forward to another gofer) / `worker` (WS remote executor); remote logs/interactions are transparently returned via "mirroring" mechanism, read path unchanged. - **WS Remote Worker + Label Scheduling**: worker connects to hub via WebSocket, automatically selected by `worker_labels` (or explicit `worker_id`); actual executor recorded in result for auditing. - **Synchronous / Asynchronous Submission**: default asynchronous (return `id` immediately); `sync` makes server wait for final state before returning (capped at 30s/60s, timeout falls back to asynchronous). - **Multiple Submission Formats**: JSON, `-- argv` (exec), **md+yaml** (frontmatter defines parameters + body is prompt). - **Runtime Interaction**: agent can ask questions during execution → job set to `pending_interaction` → user answers → continue running. - **Observable / Auditable**: `/v1/runners` health roster, `/v1/jobs` status/logs/SSE real-time stream, `caller_id` (who submitted) / `worker_id` (where executed) stored, retention period cleanup. ## Architecture Overview ```txt Submission/Observation Entrance Control Surface (serve) Execution Location (runner) agent ┌───────────────┐ ┌────────────────────────┐ ┌─ local (this process) ─┐ │ CLI gofer job │──HTTP────▶│ /v1/jobs job.Service │─────▶│ peer-http (another gofer) │──▶ cli-agent │ HTTP /v1/* │ │ registry: project/agent │ │ worker (WS remote executor) │ (codex/claude/…) │ MCP stdio │ │ runners + ws hub │ └────────┬───────────┘ or exec(argv) │ Web Console │ │ jobstore(SQLite)+logs │◀────mirror logs/status/interactions────┘ └───────────────┘ └────────────────────────┘ Authorization: Bearer <token>（except /health） Result: <host_path>/tmp/gofer/<job_id>/ + DB ``` ## Installation / Build Requires Go 1.25+ (and Node + pnpm for Web Console). ```bash make build # current platform → dist/gofer (default upx compression; use go build without upx) go build -o dist/gofer ./cmd/gofer make web build # build frontend and embed into binary (= make web copy dist into internal/webui/dist + make build) make build-all # cross-platform compilation (linux/darwin/windows × amd64/arm64) ``` ## Quick Start ```bash # 1. Register project (host-path must be existing absolute directory) gofer project add workspace \ --host-path /work/projects/workspace --container-path /workspace \ --default-agent codex --allow-agent codex --allow-agent claude --allow-agent exec \ --allow-runner local --allow-exec # 2. Start service (token via env) export GOFER_TOKEN=dev-token gofer serve --addr 0.0.0.0:8765 # 3. Run an exec job (--sync makes server wait for final state) gofer job run -p workspace -a exec --sync -- go version # → status=done exit_code=0 # 4. Run a cli-agent job; check logs gofer job run -p workspace -a codex --prompt "summarize test failures in this directory" --wait gofer job logs <id> --stream stdout # 5. Open http://<addr>/ in browser, paste token to access, see dashboard/details/real-time logs/new job ``` ## Core Concepts - **project**: a real directory where tasks can be executed. `host_path` (host path) / `container_path` (container path) / `allowed_agents` / `allowed_runners` / `allow_exec` / `max_concurrent_jobs`. - **agent**: how to execute. `cli-agent` uses `command`+`args` template rendering (placeholders `{{prompt}}`/`{{cwd}}`/`{{job_id}}`/`{{result_dir}}`, replaced element-wise, not through shell); `exec` runs `cmd` argv as is (requires project `allow_exec`). - **runner**: where to execute. `local` (built-in, this process's subprocess) / `peer-http` (forward to another gofer) / `worker` (WS connected remote executor). - **job lifecycle**: `queued → running → done|failed|cancelled|timeout`; during execution, questions asked lead to `running → pending_interaction → running`. ## Ways to Submit Jobs Same `JobRequest`, four interfaces, two sequences: ```bash # CLI —— cli-agent uses --prompt; exec uses -- argv gofer job run -p workspace -a codex --prompt "review changes and give risk points" gofer job run -p workspace -a exec --sync -- mvn -q test # --sync: server waits for final state # md+yaml file —— frontmatter defines parameters, body is prompt (only cli-agent) gofer job run -f task.md ``` ```markdown <!-- task.md --> --- project_key: workspace agent: codex runner: worker worker_labels: [gpu] # or worker_id: w-01 sync: false --- generate batch script under scripts/, reading task list from config.yaml…… ``` - **HTTP**: `POST /v1/jobs` (JSON). Add `"sync": true` or `?wait=1` for synchronous (return `200`+complete result; exceed server limit `202`+`X-Gofer-Async:1`+id). md submission uses `Content-Type: text/markdown`. - **Web Console**: top bar "+ New Job" form, select project/agent/runner (worker can explicitly select id or fill labels), tick sync, submit and jump to details. ## Remote Execution (peer-http / ws-worker / Label Scheduling) Remote job logs, status, and interactions are transparently returned to local job via "mirroring", **dashboard/details/logs read path remains unchanged**. ```yaml # Method A: peer-http —— forward to another gofer (e.g., in a container) runners: docker-peer: { type: peer-http, base_url: http://127.0.0.1:8766, token_env: PEER_TOKEN } # Method B: ws-worker —— remote executor actively connects to this hub server: workers: # registered workers (worker_id ↔ token binding + scheduling labels) w-gpu: { token_env: WTOK_GPU, labels: [gpu, linux] } w-cpu: { token_env: WTOK_CPU, labels: [cpu, linux] } runners: worker: { type: worker } # worker type runner (dynamically dispatch based on worker_id) ``` worker side uses independent configuration to connect and execute locally (`gofer worker --worker-config worker.yaml`): ```yaml worker_id: w-gpu server_link: { urls: [ws://hub:8765/v1/workers/connect], token_env: WTOK_GPU } labels: [gpu, linux] projects: { workspace: { host_path: /abs, container_path: /abs, allowed_agents: [exec], allow_exec: true } } ``` - **Explicit Routing**: submit `{"runner":"worker","worker_id":"w-gpu"}`. - **Label Auto-Scheduling**: submit `{"runner":"worker","worker_labels":["gpu"]}` → select one from **connected and label-inclusive** workers based on `in_flight↑ → heartbeat freshness↑`; no eligible candidate returns `503`. - Actual `worker_id` recorded in `JobResult`, visible in dashboard runner column and details meta. ### Configure a Worker (init → validation → start) ```bash gofer init worker -o worker.yaml # generate worker configuration template (init uses -o/--output to specify output path) # edit worker.yaml (see below "three alignments") and self-check, focus on token resolution / host_path existence: gofer -c worker.yaml config validate worker # global -c must be before subcommand (app-level flag) gofer worker --worker-config worker.yaml # start; worker.yaml has independent semantics, dedicated --worker-config ``` **Three Alignments** (same `worker_id`, missing one will cause connection failure or Web visibility): | Location | Field | must equal | |---|---|---| | Server `server.workers` | segment's KEY (auth/binding) | worker side `worker_id` | | Server `runners.<name>` | `worker_id` (roster/dispatch) | same `worker_id` —— **missing this segment = connected but not visible in `/v1/runners`, Web** | | Worker side `server_link` | `token` / `token_env` | `server.workers.<worker_id>`'s same token | > To see a worker as `connected` in Web "Runners", must declare a **named worker runner with `worker_id`** (e.g., `w-gpu: {type: worker, worker_id: w-gpu}`); unnamed `worker` runner only for dynamic dispatch by label, not corresponding to a specific worker. **Dispatch Key Points**: - **Submission side** project `allowed_runners` must include that **worker-runner's name** (e.g., `w-gpu`, not literal `worker`) to dispatch to it; **worker side** project `allowed_runners` includes `local` (worker receives and actually executes with local runner). - job's `project` key must **exist in both configurations** (worker uses its own config to reparse same-name project, mismatch will reject as "unknown project"). **Connection Logs**: worker / serve output structured logs at key points (`worker registered with hub` / `hub accepted worker` / `registration rejected … reason=…`), issues can be located at a glance. `GOFER_LOG_LEVEL=debug|info|warn|error` adjusts log level (default `info`, writes to stderr). ## Runtime Interaction 1. Running agent asks questions via `POST /v1/jobs/{id}/interactions` → job set to `pending_interaction`. 2. User `GET /v1/jobs/{id}/interactions` sees pending items → `POST .../answer` (`{"answer":"..."}`). 3. Without other pending items, job automatically returns to `running`, agent reads answer and continues. ## Observation and Auditing - **Execution Location Visible**: `GET /v1/runners` lists each runner's health status (local always up; peer-http periodic probe up/down; worker heartbeat status connected/disconnected + in-flight jobs + labels + heartbeat age). Web `/runners` dashboard consumes it. - **Job Status/Logs**: `GET /v1/jobs` (filtered by project/status/caller), `/v1/jobs/{id}`, `/logs/{stdout,stderr}` (last 256KB), `/stream` (SSE real-time logs+status+interactions). - **Auditing Fields**: `caller_id` (who submitted, parsed from token, server-side coverage for anti-fraud) + `worker_id` (where executed) stored with job, returned in response, and filterable. - **Retention**: `storage.retention` periodically cleans up expired/over quota final jobs and their log directories. ## Configuration Reference ### Lookup Chain (Hit and Use) 1. `--config <path>` → 2. Environment variable `GOFER_CONFIG` → 3. `./.gofer.local.yaml` → `./.gofer.yaml` → 4. `<config-dir>/config.yaml` (default `~/.config/gofer/config.yaml`, can be changed with `GOFER_CONFIG_DIR`). Complete example see [`config/gofer.example.yaml`](config/gofer.example.yaml). Key sections: ```yaml server: addr: 0.0.0.0:8765 # Default accessible from container; security relies on mandatory token + private network admission token_env: GOFER_TOKEN # bearer token source: token_env > inline token > --token allow_empty_token: false # Must explicitly set to true to allow token-less startup # callers: # Optional: multiple caller identities (each token → caller_id, into job audit) # - { id: docker, token_env: DOCKER_CALLER_TOKEN } # workers: # Optional: registered workers (worker_id↔token binding + scheduling labels) # w-gpu: { token_env: WTOK_GPU, labels: [gpu, linux] } # web_enabled: true # Built-in Web console switch (default on; serve --no-web off) storage: default_exchange_subdir: tmp default_result_subdir: gofer # root: /var/lib/gofer # Optional: centralized store (set and container side can only read back via HTTP) # db_path: "" # SQLite metadata database; empty means <root>/gofer.db > <config-dir>/gofer.db # retention: { max_age_days: 30, max_count: 5000, prune_interval_minutes: 60 } projects: my-project1: host_path: /work/projects/my-project1 container_path: /work/my-project1 default_agent: codex allowed_agents: [codex, claude, exec] allowed_runners: [local, worker] # Including worker for remote dispatch allow_exec: true max_concurrent_jobs: 4 agents: # Placeholders: {{prompt}} {{cwd}} {{job_id}} {{result_dir}} codex: { type: cli-agent, command: codex, args: [exec, "{{prompt}}"], detect: { command: codex, args: [--version] } } claude: { type: cli-agent, command: claude, args: ["-p", "{{prompt}}"], detect: { command: claude, args: [--version] } } exec: { type: exec, detect: { command: sh, args: [-c, "true"] } } runners: local: { type: local } # docker-peer: { type: peer-http, base_url: http://127.0.0.1:8766, token_env: PEER_TOKEN } # worker: { type: worker } # WS remote executor (label scheduling, see above) ``` - **`.env` Auto-loading**: Early startup loads `<config-dir>/.env` (global) → `./.env` (current directory, overriding global); already exported OS env always has the highest priority; files not found are skipped. Only for local convenience, **do not commit real tokens** (`.gitignore` already ignores `.env`, see [`.env.example`](.env.example)). - **Result Directory**: Default `<host_path>/tmp/gofer/<job_id>/` (if `storage.root` is set, then `<root>/<project_key>/<job_id>/`). Each job directory only keeps `stdout.log`/`stderr.log`; status/metadata/interactions are in SQLite, readable via HTTP. ## CLI Reference Global `-c/--config` (default `${GOFER_CONFIG}`, otherwise follows the discovery chain: cwd `.gofer[.local].yaml` → `~/.config/gofer/config.yaml`) is an **app-level flag, must be placed before subcommands**: `gofer -c <path> <command> [sub] [--options]`. `init` uses `-o/--output` to specify the output path; `worker` uses independent `--worker-config` (worker.yaml has different semantics from gofer config). ```bash gofer init [server|worker] [-o path] # Generate config templates (default server→.gofer.yaml; worker→worker.yaml; -o specifies output path) gofer serve --addr 0.0.0.0:8765 [--token …] [--allow-empty-token] [--no-web] gofer worker --worker-config worker.yaml # Connect as WS remote executor to hub (worker.yaml has independent config) gofer [-c path] config validate [server|worker] # Validate config (default server; worker checks token/host_path etc.) gofer project list | show <k> | add <k> … | remove <k> | validate <k> gofer agent list | detect | show <k> # detect reports unavailable if not installed, does not exit with non-zero gofer job run … | list … | show <id> | watch <id> | logs <id> --stream … | rerun <id> | cancel <id> gofer workflow run <file.yaml> [--watch] | show <id> | list | cancel <id> # Multi-step job chain (previous step output feeds next step; supports step retry/failure strategy/parallel fan-out/workflow nesting type=workflow) gofer mcp # stdio MCP server (config follows global -c/discovery chain) gofer --gen-completion bash|zsh > ~/.gofer.completion.sh # Completion script ``` `job run` key parameters: `-p/--project`, `-a/--agent` (mandatory), `--runner` (default local), `--cwd` (default `.`, project-internal), `--prompt` (cli-agent), `--argv` (exec), `-f/--file` (md+yaml), `--sync` + `--wait-timeout` (synchronous wait), `--wait` (client-side polling to terminal state), `--worker-id` / `--worker-labels` (worker routing), `--tags`, `--timeout`, `--title`, `-s/--server`, `--token`. > ⚠️ **Workflow Cross-Project / Cross-Machine Value Passing (`${steps.N.result_dir}`)**: `result_dir` is an **absolute path**. Each step can point to a different project (development project output → test project read), as long as these steps are executed on the **same file system** (local machine / same container local runner), the next step can directly read the `result_dir` of the previous step without copying. **However**, when a step is executed **remotely on a different machine** using a worker, `result_dir` is on that machine and **cannot be directly read** across machines - use `${steps.N.result}` (inline result.json, ≤32KB) / `${steps.N.stdout}` to pass values, or store artifacts in a **shared disk**. Remote artifact automatic pull channel is reserved for future implementation. > `GOFER_LOG_LEVEL=debug|info|warn|error` (default `info`, written to stderr) adjusts structured log detail levels - worker/serve connection lifecycles are output here. ## Recommended Deployment (Single Machine) One machine runs one server, project mappings converge to a single global file: ```bash export GOFER_CONFIG=~/.config/gofer/config.yaml # Write into shell profile gofer init server --global # Generate global skeleton # Edit: fill server.token_env / agents / runners gofer project add demo-api --host-path /abs/demo-api --container-path /work/demo-api gofer serve # One process # Any directory: gofer job run -p demo-api -a claude "..." # CLI connects to serve ``` `GOFER_CONFIG` takes priority over `.gofer.yaml` in the current directory; commands in any directory use the global configuration. Project-specific preferences are placed in the project directory `.gofer.project.yaml` (lightweight config, to be implemented in future stages). ## MCP Integration `gofer mcp` exposes the same control interface as a **stdio MCP server** (reuses serve's `job.Service`/registry). stdout is the protocol channel, without logging output. ```json { "mcpServers": { "gofer": { "command": "/abs/path/to/gofer", "args": ["-c", "/abs/path/to/config.yaml", "mcp"] } } } ``` Exposes 8 tools (fields in snake_case, aligned with HTTP): `bridge_list_projects` / `bridge_list_agents` / `bridge_run_job` / `bridge_get_job` / `bridge_tail_log` / `bridge_cancel_job` / `bridge_get_interactions` / `bridge_answer_interaction`. ## Web Console `serve` has a built-in static SPA (mounted at the root path, page opening without authentication, `/v1/*` still requires a token), with resources **embedded in binary**. `make web build` builds and embeds; bare `go build` (without running `make web`) displays a placeholder page, without affecting API. Development mode `pnpm -C web dev` (vite proxy `/v1`). Provides deep/light themes (following system + persistent). Dashboard/details/real-time logs/Workers dashboard/new job form. Disable Web: `serve --no-web` or `server.web_enabled: false`. ## HTTP API Based on `gookit/rux`. `/health` does not require authentication; `/v1/*` all require `Authorization: Bearer <token>`. Error response `{"error":"…","detail":"…"}`. | Method | Path | Description | |---|---|---| | GET | `/health` | Process alive (no authentication) | | GET | `/v1/projects` · `/v1/projects/{key}` | Project list/details | | GET | `/v1/agents` | Agent list + detection status | | GET | `/v1/runners` | Runner health roster (local/peer-http probe/worker heartbeat) | | GET | `/v1/meta` | Form option aggregation (projects/agents/runners/workers) | | POST | `/v1/jobs` | Create job (JSON or `text/markdown`; `sync`/`?wait=1` synchronous) | | GET | `/v1/jobs` · `/v1/jobs/{id}` | Job list (project/status/caller filtering)/details | | GET | `/v1/jobs/{id}/logs/{stdout,stderr}` | Log tail (256KB) | | GET | `/v1/jobs/{id}/stream` | SSE real-time log + status + interaction | | POST | `/v1/jobs/{id}/cancel` | Cancel | | POST/GET | `/v1/jobs/{id}/interactions` | Running interaction: initiate (question)/list | | POST | `/v1/jobs/{id}/interactions/{iid}/answer` | Answer ({"answer":"…"}) | | GET | `/v1/workers/connect` | Worker WS upgrade entry (bare Bearer, failure with 401) | `POST /v1/jobs` body (snake_case): `project_key`, `agent`, `runner`, `prompt`(cli-agent)/`cmd`(exec argv), `cwd`, `timeout_sec`, `title`, `worker_id`/`worker_labels`(worker), `sync`/`wait_timeout_sec`, `request_id`(idempotent key). ## Security Considerations - **Listening on `0.0.0.0:8765`**: Allows container access via `host.docker.internal`; security relies on **mandatory token + private network admission**, not default binding to loopback. Pure local usage can tighten to `127.0.0.1`. - **Mandatory token**: Default rejects token-less startup; empty token requires explicit `--allow-empty-token`. Multiple caller tokens use `crypto/subtle` constant time comparison, `caller_id` stored to prevent forgery. - **Worker binding**: `worker_id` must be bound to its token (per-worker token MVP mandatory, `allow_empty_token` does not exempt). - **exec double approval** + **cwd limited to project-internal safeJoin** (prevent `../` escaping) + **do not concatenate shell** (argv array) + **token/secret not logged** + **log interface only tail 256KB**. ## Docker Container ↔ Host (Main Usage in This Workspace) Claude is inside the container, tasks that require host cooperation (host `codex`, commands needing host environment, external dependencies, etc.) are relayed through host gofer: ```bash # Submit from container (via host.docker.internal) curl -s -X POST http://host.docker.internal:8765/v1/jobs \ -H "Authorization: Bearer $GOFER_TOKEN" -H "Content-Type: application/json" \ -d '{"project_key":"workspace","agent":"exec","runner":"local", "cmd":["mvn","-q","test"],"cwd":"services/demo-api","timeout_sec":1200}' # Read logs via shared disk / or read status via HTTP tail -n 50 tmp/gofer/<id>/stdout.log curl -s -H "Authorization: Bearer $GOFER_TOKEN" http://host.docker.internal:8765/v1/jobs/<id> ``` ## History Codename evolution: `codex-bridge` (single codex+exec direct run) → `dev-agent-bridge` (multi agent/project registry + `/v1` async job) → **`gofer`** (+ ws remote worker / label scheduling / synchronous and md submission / Web console / MCP / SQLite storage). > Design and implementation plans see [`docs/`](docs/) (`design/` design, `plans/` implementation plan, `TODO.md` todo and roadmap).