ao-runtime

# AO Runtime **English** | [日本語](./docs/ja/README.md) | [简体中文](./docs/zh-Hans/README.md) | [繁體中文](./docs/zh-Hant/README.md) | [한국어](./docs/ko/README.md) | [Español](./docs/es/README.md) | [Русский](./docs/ru/README.md) | [Français](./docs/fr/README.md) | [Deutsch](./docs/de/README.md) | [Português](./docs/pt/README.md)  **Engine repo for the AO public set. Product users should start with [`ao-operator`](https://github.com/uesugitorachiyo/ao-operator).** AO Runtime is the Rust execution engine under AO Operator. In the product line, **AO** is defined as **AI Orchestration Operation**: AO Operator is the operation layer, and AO Runtime is the engine underneath it. AO Runtime runs local-first, policy-governed agent work for Codex CLI, Claude Code CLI, OpenClaw chat sessions, Hermes plugin/queue flows, MCP tools, and A2A-compatible remote agents. It turns a heterogeneous fleet of AI coding agents into a single executable surface: one DAG, one event log, one policy seam. AO Runtime is public because the engine contract matters. If AO Operator is the product you run first, AO Runtime is the part you inspect when you need to know how DAG execution, policy decisions, approvals, artifacts, remote workers, MCP, A2A, and provider adapters are actually implemented. ## Public Repo Set The repos are public together, but the advertisement should still name one product: **AO Operator**. | Repo | What it is | How to read it | | --- | --- | --- | | [`ao-operator`](https://github.com/uesugitorachiyo/ao-operator) | The product: role contracts, RunSpecs, provider routing, evidence packs, release gates | **Start here. Clone this first.** | | `ao-runtime` | The Rust execution engine: DAG scheduler, policy seam, event log, artifacts, workers, OpenClaw/Hermes/MCP/A2A adapters | You are here. Read this to understand or embed the engine. | | [`financial-services-profile`](https://github.com/uesugitorachiyo/financial-services-profile) | Flagship demo profile: citation-sensitive financial workflows over public/synthetic data | Run after AO Operator to see the regulated-workflow story. | | [`secure-agent-profile`](https://github.com/uesugitorachiyo/secure-agent-profile) | Reusable secure coding-agent profile: guarded patching, dependency review, PR evidence | Run after AO Operator to see policy-gated software work. | | [`ao-control-plane`](https://github.com/uesugitorachiyo/ao-control-plane) | Future management layer: typed run state, evidence aggregation, release-train gates | Read last. It is not required for the first product trial. | Use this repo directly only if you are working on the engine, SDKs, adapters, policy, artifacts, workers, or event replay. If you are trying the product, begin with AO Operator. ## Adapter Boundary AO Runtime intentionally keeps adapter execution below the product workflow layer: ```text OpenClaw: chat, cron, webhooks, observe, cancel Hermes: allocate queues, saturate Codex-backed factories, synthesize closure AO Operator: role contracts, gates, routing, evaluation, reports AO Runtime: RunSpec DAGs, provider launch, policy, events, artifacts ``` The OpenClaw adapter and Hermes plugin bridge are runtime surfaces. AO Operator uses them as front doors for repeatable workflows without giving those front doors ownership of SDD ingestion, role contracts, provider routing, evaluator closure, or signed evidence packs. ## Overview AO Runtime is a control plane for running agent work as auditable, policy-governed jobs. Users submit a `RunSpec`; the runtime validates and plans the DAG, dispatches ready tasks to local or remote adapters, gates risky actions through `ao-policy`, persists every `RuntimeEvent`, and stores outputs as content-addressed artifacts. It is designed for teams that want multiple agent front ends without giving each agent direct access to the machine, the network, or every tool in the environment. The core stays provider-neutral; adapters translate provider behavior at the edge. ## Product Boundary AO Runtime is standalone infrastructure, but it is not the first product surface. It should remain usable without AO Operator, ai-teams, or any one agent framework while AO Operator gives users the first complete workflow. - **AO Runtime** owns the execution contract: `RunSpec` schemas, DAG planning, scheduler behavior, policy decisions, approvals, adapters, event replay, artifacts, observability, and remote worker transport. - **AO Operator** (formal name: AO Runtime Operator; repo slug `ao-operator`; legacy compatibility slug `factory-v3`) is a flagship production system built on AO Runtime. It adds SDD discipline, role contracts, operator slices, release readiness, and evaluator closure for high-throughput software and knowledge work. It is the public launch surface. - **AO Control Plane** is the future management layer for typed run state, evidence aggregation, and train evidence packs. It is not required to try AO Operator. - **Other products** can use AO Runtime directly by submitting `RunSpec` jobs and consuming events, artifacts, approvals, and policy decisions. ## Product Ideas Built On AO Runtime AO Runtime is useful anywhere work can be decomposed into auditable, policy-governed agent tasks: - secure agent CI runner for test repair, code review, dependency audit, and release checks - enterprise agent control plane for Claude, Codex, MCP tools, local scripts, and remote workers - security review factory for SAST, DAST prep, threat modeling, prompt-injection tests, and pen-test evidence - regulated evidence automation for healthcare, finance, defense, legal, and compliance workflows - research production line for literature review, claim extraction, experiment planning, and report drafting - data/ML pipeline orchestrator for dataset cleanup, labeling review, eval generation, red-team analysis, and model-card release - incident-response automation for log triage, timeline generation, containment, patch proposal, and postmortems - infrastructure migration factory for inventory, Terraform review, dry-run, rollout, validation, and rollback - documentation and knowledge factory with source-verified updates and durable review artifacts - remote worker and edge orchestration for workspace bundles, isolated task execution, signed returned artifacts, and policy-gated transport ## Status Feature-complete per the AO Runtime prompt pack (S00 → S26), including five post-pack follow-ups for shared agent context, daemon artifact persistence, CLI artifact retrieval/export, workflow-friendly artifact metadata, and TUI artifact visibility. Workspace validation for the S26 artifact pipeline passed with the full Rust workspace test suite, `cargo fmt --all -- --check`, and `cargo clippy --workspace --all-targets -- -D warnings`; GitHub Actions CI matrix runs on macOS + Linux. Slice reports for every slice (S00 — S26) live under [`progress/slice-reports/`](./progress/slice-reports/). Future product-maturity guidance is tracked in [`docs/PRODUCT-MATURITY-ROADMAP.md`](./docs/PRODUCT-MATURITY-ROADMAP.md). ## Visual guide   ## Agent Team Examples Manual five-terminal Codex team scaffolds are available in [`example/agent-team/`](./example/agent-team/). They provide ready-to-paste role prompts and coordination files for one Planner, two Implementers, and two Reviewers. - [`single-pc-5-codex`](./example/agent-team/single-pc-5-codex/) runs all five Codex terminals against one local checkout. - [`distributed-mac-ubuntu-wsl`](./example/agent-team/distributed-mac-ubuntu-wsl/) shows the same team shape across macOS, Ubuntu, and Windows via WSL without exposing AO Runtime services over the network. Validate the scaffold with `bash scripts/agent_team_check.sh`. ## Runtime Example Packs Runnable end-to-end runtime examples live under [`example/`](./example/): - [`a2a-policy-gateway`](./example/a2a-policy-gateway/) starts the real `ao-a2a` HTTP gateway, denies an inbound `remote-a2a` message under a high-security policy, then allows the same request under `trusted-loopback` and verifies `message/send`, `tasks/get`, and `tasks/cancel`. - [`artifact-store-flow`](./example/artifact-store-flow/) drives the `ao-artifacts` content-addressed store end-to-end: PUT/GET round-trip, dedup of identical bytes across distinct `(run, task, name)` triples, on-disk SHA-256 fanout layout, tamper detection (`HashMismatch`), `NotFound` and `InvalidHash` error paths, `list_by_task` filtering, and persistence across reopen. - [`distributed-worker-workspace-bundle`](./example/distributed-worker-workspace-bundle/) starts the real `ao-node` gRPC coordinator, enrolls a local worker, packs a workspace bundle, dispatches a fake remote task, and verifies returned `task.artifact` events. - [`coordinator-wal-replica`](./example/coordinator-wal-replica/) starts an `ao-node` primary with a JSONL WAL, drives register/heartbeat/dispatch gRPC mutations, starts a read-only replica, and verifies replay consistency, duplicate-entry idempotency, and tail-following new entries after startup. - [`observability-trace-metrics-flow`](./example/observability-trace-metrics-flow/) drives `ao-otel` end-to-end: mints a 32-hex `TraceId`, opens a Run span with two Task spans (one success, one denied) plus a nested Policy span, exports them through an in-memory exporter, exercises every canonical counter and both histograms, and round-trips a `RuntimeEvent` carrying `traceId` / `spanId`. - [`grpc-concurrent-dispatch`](./example/grpc-concurrent-dispatch/) starts the real `ao-node` gRPC coordinator, enrolls a local worker, then fires N parallel `RuntimeService.dispatch_task` RPCs from independent clients and asserts every stream stays bound to its own `(run_id, task_id)` with the expected event sequence — locking the parallel-RPC contract. - [`artifact-store-content-addressing`](./example/artifact-store-content-addressing/) drives the real `ao-artifacts` store through three PUTs across two distinct byte-strings, proves SHA-256 dedup (one on-disk blob backs two refs), then tampers with a stored blob and asserts `verify` flips to `false` and `get` returns `Error::HashMismatch` while the sibling blob stays clean — locking the content-addressed-store contract. - [`operator-tui-snapshots`](./example/operator-tui-snapshots/) submits a RunSpec through `ao`, files a pending approval ticket through `ao-policy`, then renders the three `ao-tui` snapshots (runs, run detail, approvals) against the resulting on-disk state. - [`mcp-gateway-policy-flow`](./example/mcp-gateway-policy-flow/) drives the TypeScript MCP gateway with deterministic tool-call fixtures, covering registry lookup, per-agent allowlists, policy allow/pending paths, local vs remote trust tags, upstream forwarding, and event emission. - [`openclaw-adapter-chat-flow`](./example/openclaw-adapter-chat-flow/) drives the TypeScript OpenClaw chat adapter with deterministic chat fixtures, covering paired/allowlisted/untrusted senders, privileged adapter rejection, policy allow/deny/pending outcomes, runtime submission, and chat replies. - [`scheduler-dag-flow`](./example/scheduler-dag-flow/) drives `ao-scheduler` end-to-end against an in-memory `MockExecutor`: sequential ordering, parallel wall-clock overlap, fan-in waiting, retry recovery + exhaustion, per-attempt timeouts, blocked-state propagation, DAG-validation negatives (cycle / missing dep / duplicate id), and the `RunReport` API. - [`event-replay-restart`](./example/event-replay-restart/) runs a real three-task `RunSpec`, reopens `AO_HOME` in later CLI invocations, and verifies ordered SQLite replay with trace/span context. - [`artifact-store-retrieval`](./example/artifact-store-retrieval/) exercises content-addressed artifact put, dedupe, task lookup, retrieval by hash, and corruption detection. - [`scheduler-failure-controls`](./example/scheduler-failure-controls/) demonstrates retries, timeout, dependency blocking, and fan-in using the scheduler's public API. - [`agent-team`](./example/agent-team/) contains manual and AO-managed Codex team examples, including shell RunSpecs and policy-gated Codex adapter smoke paths. Validate the A2A pack with `bash scripts/a2a_policy_gateway_validate.sh`. Validate the artifact-store flow pack with `bash scripts/artifact_store_flow_validate.sh`. Validate the distributed worker pack with `bash scripts/distributed_worker_workspace_validate.sh`. Validate the coordinator WAL replica pack with `bash scripts/coordinator_wal_replica_validate.sh`. Validate the observability trace+metrics pack with `bash scripts/observability_trace_metrics_validate.sh`. Validate the gRPC concurrent dispatch pack with `bash scripts/grpc_concurrent_dispatch_validate.sh`. Validate the artifact store content-addressing pack with `bash scripts/artifact_store_content_addressing_validate.sh`. Validate the operator TUI snapshots pack with `bash scripts/operator_tui_snapshots_validate.sh`. Validate the MCP gateway policy-flow pack with `bash scripts/mcp_gateway_policy_flow_validate.sh`. Validate the OpenClaw adapter chat-flow pack with `bash scripts/openclaw_adapter_chat_flow_validate.sh`. Validate the scheduler DAG flow pack with `bash scripts/scheduler_dag_flow_validate.sh`. Validate the event replay pack with `bash scripts/event_replay_restart_validate.sh`. Validate the artifact store pack with `bash scripts/artifact_store_retrieval_validate.sh`. Validate the scheduler pack with `bash scripts/scheduler_failure_controls_validate.sh`. Run every runnable runtime pack with `bash scripts/runtime_example_packs_validate.sh`. ## What it can do AO Runtime is the substrate underneath your coding agents — not another agent. It answers the operational questions that show up the moment more than one agent fronts real work: - **Run a multi-step coding task as a versioned spec.** `RunSpec` declares tasks, dependencies, inputs, and expected artifacts. The daemon plans the DAG, the scheduler executes ready tasks concurrently, and every state transition is appended to a durable event log. - **Pass runtime-managed context between agent tasks.** Downstream agent tasks can opt into upstream stdout handoff through declared dependencies, with an explicit untrusted-data boundary before provider launch. - **Mix local CLIs and remote agents in one DAG.** The same spec can pin one task to Claude Code, another to Codex CLI, a third to a Hermes plugin workflow, and a fourth to an A2A-compatible remote agent — adapters normalize them behind a single execution contract. - **Fail closed at the policy seam.** Every external action — shell command, network call, file write outside the workspace, MCP tool invocation — passes through `ao-policy` before it runs. Decisions are `allow`, `require_approval` (queued for an operator), or `deny`, evaluated against bundled YAML profiles like `high-security.yaml`. - **Approve sensitive actions out-of-band.** When the policy engine returns `require_approval`, the action is parked on the approval queue. A human reviews it via the `ao-policy` CLI or the `ao-tui` operator surface; the task resumes only after an explicit decision is recorded. - **Inspect foreground daemon lifecycle.** `ao daemon start --foreground` records the latest supervisor session under `AO_HOME/daemon/lifecycle.json`, and `ao daemon status` surfaces that state for installed-service validation without exposing bearer token values. - **Replay any run from durable state.** `ao-daemon` persists every `RuntimeEvent` to SQLite with W3C-shape trace and span ids, so a run can be reconstructed end-to-end after restart, crash, or migration. Schema changes ship as idempotent `ALTER TABLE` migrations. - **Replay failed or interrupted task closures as new runs.** `ao runs retry-plan <run-id>` identifies failed, blocked, running, and never-started candidates; `ao runs replay <run-id> --task <task-id> --dry-run` previews the selected targets plus their transitive dependencies before executing the closure as a separate run. - **Summarize operator attention.** `ao runs summary [--json]` aggregates run health, pending approvals, recoverable interrupted runs, retryable run/task counts, artifact GC preview counts, and one recommended next action without printing task output bytes or mutating state. - **Bundle a workspace and ship it to a remote node.** `ao-workspace` packs the working tree into a content-addressed bundle; `ao-node` carries it over gRPC to a worker, runs the assigned slice there, and streams events back. Artifacts flow through `ao-artifacts`, a content-addressed store. - **Enroll remote nodes with a trust contract.** `ao-node` requires an enrollment token at registration, tracks revocation, and tags each peer with a `TrustLevel` so the coordinator can reject or downgrade work bound for less-trusted workers. - **Speak A2A to peer agents.** `ao-a2a` exposes a JSON-RPC + SSE gateway with an Agent Card at `/.well-known/agent-card.json`, supports `message/send`, `message/stream`, `tasks/get`, and `tasks/cancel`, and routes every inbound message through the same policy gate as local work. - **Bridge MCP tools per-agent.** `mcp-gateway` enforces per-agent tool allowlists and forwards every call through the policy bridge — no agent silently picks up a tool the operator did not grant it. - **Drive chat-shaped agents through the same DAG.** `openclaw-adapter` pairs OpenClaw chat sessions with `RunSpec` execution, so conversational agents inherit the same approval queue, event log, and artifact store. - **Observe everything.** `ao-otel` mints W3C trace ids per run and span ids per task, exposes a metrics registry with canonical counters and histograms, and stamps every emitted `RuntimeEvent` with its trace + span context for downstream collectors. - **Operate from a terminal.** `ao-tui` renders the runs list, run detail, artifact metadata, and approval queue as a model + view pair — usable as a snapshot binary today, ready to host a full TUI front-end without rewriting the model. - **Inspect, export, and clean up task artifacts.** Non-empty task stdout/stderr is stored in `ao-artifacts`; operators can list artifact metadata with `ao run <run-id> artifacts --json`, export bytes explicitly with `ao run <run-id> artifact <task-id> <name> --output <path>`, remove logical refs with `ao artifacts delete-ref`, and preview storage cleanup with `ao artifacts gc --dry-run` before running `ao artifacts gc`. - **Validate the system end-to-end.** `evals/` ships golden, chaos, and security scenarios; `bash evals/run_evals.sh` is the same harness CI runs. `scripts/quickstart_validate.sh` exercises the install → run → policy-deny path in an isolated tempdir on every CI build. ## Architecture at a glance The runtime has four main planes: - **Submission and operation:** `ao-cli`, `ao-tui`, and `ao-policy` give humans a way to submit runs, inspect state, and approve gated actions. - **Core execution:** `ao-daemon` validates `RunSpec` input, owns run records, emits events, and coordinates `ao-scheduler`. - **Execution adapters:** local CLIs, Hermes, OpenClaw, A2A peers, gRPC nodes, and MCP tools are normalized behind adapter contracts. - **Governed state:** `ao-policy`, `ao-artifacts`, SQLite-backed event persistence, and `ao-otel` keep actions bounded, outputs addressable, and runs observable. ## Layout - [`crates/`](./crates/) — Rust workspace - `ao-core` — versioned schemas (`RunSpec`, `RuntimeEvent`, ids). - `ao-daemon` — synchronous DAG executor + event store + run records. - `ao-cli` — `ao` CLI (`init`, `doctor`, `run`, `run <id> events`, `run <id> artifacts`, `run <id> artifact <task> <name>`, `runs list`, `runs summary`). - `ao-scheduler` — concurrent task scheduler over the daemon's plan. - `ao-workspace` — workspace bundling for remote nodes. - `ao-artifacts` — content-addressed artifact store. - `ao-policy` — policy evaluator + approval queue + `ao-policy` CLI. - `ao-a2a` — A2A gateway server + client. - `ao-grpc` — tonic-generated gRPC stubs from `proto/ao/v1/runtime.proto`. - `ao-node` — coordinator + worker over gRPC, with enrollment + workspace sync. - `ao-otel` — observability primitives (TraceId/SpanId/MetricsRegistry). - `ao-tui` — operator TUI primitives (model+view) plus the `ao-tui` snapshot binary, including run, approval, and artifact metadata snapshots. - [`adapters/`](./adapters/) — TypeScript adapters - `claude-adapter` — Claude Code CLI bridge with policy hooks. - `codex-adapter` — Codex CLI bridge with policy hooks. - `hermes-plugin` — Hermes plugin bridge (6 tools, all policy-gated). - [`packages/`](./packages/) — TypeScript packages - `mcp-gateway` — MCP tool gateway with per-agent allowlist + policy bridge. - `openclaw-adapter` — OpenClaw chat-to-run adapter with pairing + policy. - [`sdks/`](./sdks/) — no-bloat generated client surfaces. - `python` — developer-preview package `ao-runtime-sdk`, with generated `ao_runtime_proto` bindings from `proto/ao/v1/runtime.proto` and a thin async `ao_runtime_sdk` client. Generate it with `scripts/generate_python_sdk.sh`. It is built locally today and is not yet published to PyPI. Do not document `pip install ao`; that package name is not AO Runtime. - [`evals/`](./evals/) — golden + chaos + security scenarios + `run_evals.sh`. - [`tests/{e2e,security}/`](./tests/) — focused end-to-end + security canaries. - [`scripts/`](./scripts/) — `install.sh`, `release.sh`, `release_preflight.sh`, `quickstart_validate.sh`, `package_python_sdk.sh`, `python_sdk_local_daemon_smoke.sh`, `docker_persistence_validate.sh`, `verify_install.sh`. - [`docs/`](./docs/) — `INSTALL.md`, `QUICKSTART.md`, `RELEASE.md`, `SUPPORTABILITY.md`. - [`config/policy/`](./config/policy/) — bundled YAML policy profiles. - [`proto/`](./proto/) — protobuf service + message definitions. ## Quickstart ```bash # 1. Install bash scripts/install.sh # 2. Run the golden spec ao run evals/golden/hello.yaml ao runs list ao runs summary # If a process was interrupted mid-run, inspect it and close it explicitly: # ao runs diagnose <run-id> # ao runs retry-plan <run-id> # ao run <run-id> events # ao runs recover <run-id> # 3. Verify the policy engine fails closed ao-policy --json --db /tmp/ao-policy.db \ --profile config/policy/high-security.yaml \ evaluate --task-id t1 --action-type network.connect \ --command "curl evil.example" # 4. End-to-end validation in an isolated tempdir bash scripts/quickstart_validate.sh # 5. Private-release supportability and persistence checks bash scripts/verify_install.sh bash scripts/python_sdk_local_daemon_smoke.sh bash scripts/docker_persistence_validate.sh ``` See [`docs/QUICKSTART.md`](./docs/QUICKSTART.md) for the walkthrough, [`docs/INSTALL.md`](./docs/INSTALL.md) for install paths, and [`docs/RELEASE.md`](./docs/RELEASE.md) for the release process and non-publishing preflight. See [`docs/PRODUCT-MATURITY-ROADMAP.md`](./docs/PRODUCT-MATURITY-ROADMAP.md) for future beta and production-readiness targets. ## CI [`.github/workflows/ci.yml`](./.github/workflows/ci.yml) runs on every push and pull request: - `test`: `cargo build/test --workspace`, `bash evals/run_evals.sh`, `bash scripts/quickstart_validate.sh` on `ubuntu-latest` + `macos-latest`. - `lint`: `cargo fmt --all -- --check` + `cargo clippy --workspace --all-targets -- -D warnings`. Both blocking. ## Engineering rules See [`AGENTS.md`](./AGENTS.md) (rules for non-Claude agents) and [`CLAUDE.md`](./CLAUDE.md) (rules for Claude agents). The original spec pack lives under [`pack/`](./pack/) (read-only — every slice has shipped). ## License AO Runtime is dual-licensed under either: - [Apache License, Version 2.0](./LICENSE-APACHE) - [MIT License](./LICENSE-MIT) at your option. See also [`NOTICE`](./NOTICE). Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual-licensed as above, without any additional terms or conditions.