Content
# Bionics
**An AI agent that controls Unreal Engine 5 with 199 tools.**
> One prompt → AnimGraph wired, Blueprints validated, assets spawned, PIE tested. The animation-pipeline layer no other UE5 AI tool ships.
<!-- TODO: replace with 30-second screen recording of BPDoctor scan + fix_all -->
<!--  -->
See **[docs/demos/](docs/demos/)** for the BPDoctor hero GIF (T1.A) and the 4-minute "One Prompt Full Locomotion" demo script (T1.B).
---
## What it does
- **199 tools** across 37 categories — UE5 actors, Blueprints, AnimGraphs, StateTrees, Control Rigs, Niagara, audio, materials, PIE, rigging, retargeting, EventGraph (K2), Linked Anim Layers, async tasks, session progress tracking, divine_powers NL→UE5 entry point
- **Native C++ bridge (BionicsBridge plugin)** — architecturally expected ~5-20ms (in-process JSON-RPC over loopback HTTP) vs ~100-400ms for Python multicast remote-exec; benchmark pending. See `plugins/BionicsBridge/README.md` for the technical breakdown.
- **MCP server** (FastMCP 2.11+ / 3.x) — drops straight into Claude Code, Cursor, Windsurf, or any MCP-aware client. Full MCP 2025-11-25 spec: annotations, `outputSchema` on query tools, async Tasks for long-running ops.
- **BPDoctor** — 34-check static analysis with auto-fix for Blueprint/AnimBP errors (missing MM schema, dead cached poses, unconnected slots, empty state machines, blend-weight sum violations, etc.).
- **Full AAA animation pipeline** — Motion Matching schema setup, IK Rig + IK Retargeter creation, batch retargeting of N animations, Linked Anim Layer AnimBPs, Control Rig asset creation and AnimBP binding.
- **Watch Mode** — read-only screen-analysis loop with TTS overlay that explains UE5 systems while you work.
- **Persistent memory with optional sqlite-vec semantic search** — cross-session SQLite memory and proven-sequence warm-starts. Opt-in vector search via `pip install bionics-agent[vector,embeddings_local]`.
- **Sub-agent fan-out** — named `AgentDefinition`s with tool subsets + `dispatch_parallel` for multi-perspective plans, critic ensembles, or parallel research.
- **OpenTelemetry** — opt-in `BIONICS_OTEL_ENABLE=1` emits OTLP spans per tool call.
- **3-tier safety + lifecycle hooks + guardrails** — safe / moderate / destructive, with confirmation gates and PreToolUse / PostToolUse / Stop hooks.
## Highlights (copy-paste ready)
### Bearer-token auth on the C++ bridge
```bash
# C++ plugin auto-generates a 256-bit token and writes it to:
# <ProjectDir>/.bionics-bridge/instance.json
# The Python side auto-reads that file — you don't have to configure anything.
# To force a known token (CI):
export BIONICS_BRIDGE_TOKEN=my-known-token
```
CORS on the `/bridge` endpoint is locked to `http://127.0.0.1` — browser-origin requests from another page cannot use a stolen token.
### Session resume after a crash
```python
from core.agent import AgentCore
agent = AgentCore(state_machine, safety, capture, executor)
sessions = agent._session.list_sessions() # all saved sessions
resumable = agent._session.list_running_sessions() # just the ones that crashed mid-run
if resumable:
agent.resume_from_session(resumable[0]["id"]) # continues exactly where it died
```
### Sub-agent fan-out (parallel multi-perspective)
```python
from core.agent_definitions import AgentDefinition, dispatch_parallel_sync
planner = AgentDefinition(name="planner", system_prompt="Decompose the task.", tools=["list_tools"])
critic = AgentDefinition(name="critic", system_prompt="Find risks in the plan.")
plan, critique = dispatch_parallel_sync(
[planner, critic],
["Build combat locomotion for Sworder:721"], # broadcast one prompt to both
)
```
### Force Claude to emit a tool call (never free-text)
```python
from core.agent_definitions import AgentDefinition
pinned = AgentDefinition(
name="structured",
tools=["my_schema_tool"],
tool_choice={"type": "tool", "name": "my_schema_tool"}, # or "any" / "required"
)
```
### OTel observability
```bash
# Opt-in, exports OTLP spans per tool call
export BIONICS_OTEL_ENABLE=1
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 # Tempo / Jaeger / any OTLP receiver
python mcp_server.py
```
Span attrs: `bionics.tool.{name,category,safety_tier,ok,elapsed_ms,arg_count}`.
### Vector memory (opt-in)
```bash
pip install -e ".[vector]" # sqlite-vec
pip install -e ".[embeddings_local]" # + sentence-transformers (~80 MB model)
```
```python
from core.memory import BionicsMemory
from core.embeddings import HashEmbedder # zero-dep, deterministic
# from core.embeddings import LocalEmbedder # quality upgrade, 80 MB model
store = BionicsMemory(embedder=HashEmbedder())
store.remember("task_outcome", "ANIMATION", "mm_setup_v1", {"worked": True})
hits = store.search("motion matching locomotion", mode="semantic", limit=5)
# each hit has .distance (lower = closer)
```
### Voyager-style self-verification (retry proven sequences)
```python
from core.tool_cache import get_tool_cache
cache = get_tool_cache()
hit = cache.replay_with_verification(
topic="ANIMATION",
prompt="build motion matching for the trooper",
execute_fn=lambda seq: my_executor.run(seq), # returns bool
max_attempts=3,
)
# Each failed sequence has its confidence decayed — stale sequences self-heal out.
```
### Async task manager (long-running UE5 ops)
```python
from core.bridge import ToolGate
gate = ToolGate(); gate.set_bypass_safety(True)
task_id = gate.execute("bionics_task_submit", {"tool_name": "ue5_live_coding", "args": {}}).data["task_id"]
# ...returns immediately, work runs on a thread pool...
status = gate.execute("bionics_task_status", {"task_id": task_id}).data
# When status.status == "completed":
result = gate.execute("bionics_task_result", {"task_id": task_id}).data
```
## Quick start
```bash
# 1. Clone
git clone https://github.com/itsribbZ/Bionics.git
cd Bionics
# 2. Install in editable mode (exposes `bionics`, `bionics-gui`, `bionics-mcp` CLI commands).
# Requires Python 3.12+ (3.14 tested). Optional extras unlock observability and vector memory.
pip install -e .
# ...or, for a dev contributor install with pytest/ruff/mypy:
# pip install -e ".[dev]"
# ...or with OTel + sqlite-vec enabled:
# pip install -e ".[otel,vector]"
# 3. Set your Anthropic API key
# Windows: setx ANTHROPIC_API_KEY sk-ant-...
# Linux/Mac: export ANTHROPIC_API_KEY=sk-ant-...
# 4. Copy the config template and set your paths
cp config.yaml.example config.yaml
# (edit config.yaml — at minimum set paths.ue5_project to your .uproject directory)
# 5. Run one of three modes
python main.py # PyQt6 GUI (Auto Mode + Watch Mode)
python mcp_server.py # MCP stdio/HTTP server (for Claude Code etc.)
python cli.py list # Command-line tool runner
```
**Optional (recommended for UE5 workflows):** build the BionicsBridge C++ plugin into your UE5 project to drop tool-call latency — architecturally expected ~5-20ms (in-process loopback HTTP) vs the ~100-400ms of Python remote-exec; benchmark pending. See [`plugins/BionicsBridge/README.md`](plugins/BionicsBridge/README.md).
## Requirements
| Requirement | Version |
|---|---|
| Python | 3.12+ (3.14 tested) |
| OS | Windows 10/11 (full support). Mac/Linux partial — `pywinauto` and `keyboard` are Windows-only; cross-platform port tracked. |
| UE5 | 5.4+ (5.7 tested). "Python Editor Script Plugin" + "Web Remote Control" must be enabled for the Python fallback path. |
| Anthropic API key | Sonnet 4.5 recommended |
| Visual Studio (optional) | Required only to build the BionicsBridge C++ plugin |
## Connecting to Claude Code
Add Bionics as an MCP server. At your project root (or globally in `~/.claude/.mcp.json`):
```json
{
"mcpServers": {
"bionics": {
"command": "python",
"args": ["<ABSOLUTE_PATH_TO_BIONICS>/mcp_server.py"],
"cwd": "<ABSOLUTE_PATH_TO_BIONICS>"
}
}
}
```
Restart Claude Code. All 199 tools show up as native tool-use.
## Tool categories
All 37 categories, live-measured from the registry (`register_all()` → 199 tools). Run `python cli.py list` for the full per-tool listing.
| Category | Tools | What it does |
|---|---:|---|
| `input` | 9 | click, type, hotkey, drag, scroll, mouse control |
| `capture` | 3 | screenshot, region capture, monitor listing |
| `vision` | 3 | template match, OCR, wait-for-image |
| `system` | 7 | windows, clipboard, processes, system info |
| `general` | 1 | wait |
| `meta` | 5 | list / describe tools, categories, config, version |
| `plans` | 4 | save / list / load / execute multi-step automation plans |
| `memory` | 7 | persistent cross-session + Voyager tool-cache |
| `tasks` | 6 | async task manager — submit / status / result / cancel / list / clear |
| `audit` | 5 | audit-log tail + session listing / resume / progress |
| `bionics` | 1 | `divine_powers` NL→UE5 entry point |
| `bp_doctor` | 2 | AnimBP Doctor CLI wrapper — locate + scan |
| `ue5_actor` | 17 | spawn / query / delete / modify actors, properties, components, CVars, console |
| `ue5_blueprint` | 15 | Blueprint graph CRUD + interfaces |
| `ue5_asset` | 11 | asset create / save / delete / query + DataAsset bulk-set |
| `ue5_animgraph` | 14 | AnimGraph node create / wire / query, state machines, Linked Anim Layer (C++ plugin) |
| `ue5_eventgraph` | 5 | EventGraph (K2) events, function calls, variable nodes, pin wiring (C++ plugin) |
| `ue5_bpdoctor` | 4 | 34-check Blueprint static analysis + auto-fix |
| `ue5_rigging` | 4 | IK Rig + IK Retargeter + batch retarget |
| `ue5_retarget` | 1 | native batch retarget (UE5.7 `duplicate_and_retarget`) |
| `ue5_autorig` | 2 | fail-closed humanoid autorig + skeleton bone validation |
| `ue5_controlrig` | 3 | Control Rig asset + AnimBP assignment |
| `ue5_niagara` | 2 | VFX emitter spawn + user-exposed param bind |
| `ue5_audio` | 2 | SoundWave import + SoundAttenuation configure |
| `ue5_statetree` | 2 | StateTree inspection + task add |
| `ue5_material` | 4 | material inspect / compile / scalar + vector set |
| `ue5_pie` | 5 | Play-In-Editor start / stop / pause / resume / state |
| `ue5_native` | 11 | C++ bridge status + native actor / asset / CVar / console ops |
| `ue5_runtime` | 7 | level save / load / new, viewport capture, logs, project info, class hierarchy |
| `ue5_python` | 2 | run Python code / file inside the editor |
| `ue5_build` | 2 | Live Coding compile (native bridge + fallback) |
| `ue5_profiling` | 3 | Insights trace start / stop + stat commands |
| `ue5_debug` | 1 | native log tail |
| `ue5_widget` | 2 | widget query + runtime widget listing |
| `ue5_uasvc` | 2 | UE5 Asset Service — preflight + native skeletal import |
| `watch` | 11 | Watch Mode start / stop / pause / resume / task / context / sessions |
| `market` | 14 | MarketBot: PDF parse + Claude-generated posts |
## Architecture
```
┌──────────────────────────┐
Claude Code / Cursor │ MCP (stdio / HTTP) │
Windsurf / CLI ─┤ mcp_server.py │
│ 199 tools registered │
└──────────┬───────────────┘
│
▼
┌──────────────────────────┐
│ core/bridge.py │
│ ToolRegistry, Gate, │
│ Safety tiers │
└──────────┬───────────────┘
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
┌───────────────┐ ┌──────────────────────┐ ┌────────────────────┐
│ core/agent.py │ │ UE5 Remote Control │ │ BionicsBridge │
│ Auto Mode │ │ HTTP :30010 (~400ms) │ │ C++ plugin │
│ Watch Mode │ │ Python RE socket │ │ :8090 JSON-RPC │
│ divine_powers│ │ :9998 (fallback) │ │ ~5-20ms native │
└───────────────┘ └──────────────────────┘ └────────────────────┘
```
> Latency figures above (~400ms RE / ~5-20ms native) are architectural estimates, not measured benchmarks — a benchmark is pending.
## Pre-built plans
The `plans/` directory contains automation scripts runnable via `python cli.py run <plan_name>` or the GUI's Plan Library. Most shipped plans reference the Sworder:721 project — use them as templates rather than running them verbatim. New project-agnostic plans are queued on the roadmap.
## Safety
Every destructive tool runs through a 3-tier gate:
| Tier | Examples | Default |
|---|---|---|
| **safe** | `mouse_move`, `screenshot`, `read_screen`, query tools | no confirm |
| **moderate** | `click`, `type_text`, `hotkey`, `open_file` | 1 confirm |
| **destructive** | `ue5_delete_asset`, `delete_actor`, `overwrite_file` | 2 confirms |
Under MCP (no GUI), destructive tools require explicitly setting `BIONICS_MCP_ALLOW_DESTRUCTIVE=1` in the environment.
## Testing
```bash
pytest tests/ # 580+ tests — core modules, tool registry (locked reads + summary), safety, UE5 rigging, memory, integration, OTel, task manager (DESTRUCTIVE gate + future-snapshot wait + auto-evict + clear tool), session (traversal guard), vector memory, sub-agent fan-out (DESTRUCTIVE gate + async-context-safe sync wrapper), Voyager verification
```
## Status
- **Version**: 0.8.4 (2026-05-29) — autorig now builds the 7-chain retarget canon (was 9) with `len(CHAINS)` as the single source of truth, live-verified against UE5 5.7 over the `:8090` bridge; see [CHANGELOG.md](CHANGELOG.md) for the full history. 582-test suite (collect-verified).
- **Stability**: v1.0 hardening in progress (see hellscape audit roadmap)
- **Active development**
## Roadmap
Short term: the internal hellscape audit (memory snapshot at `project_hellscape_audit_2026-04-23.md`) lays out T0 ship blockers, T1 WWZ-launch assets, T2 SOTA 2026 parity (native tool-use protocol, session resumability, lifecycle hooks, vector memory, subagent fan-out, MCP Tasks/Sampling/annotations), T3 moat deepening (Reflexion, zoom-before-click, Process Reward Models, internal eval harness), T4 ecosystem/community.
## Contributing
Contributions welcome. The codebase is split cleanly between:
- `core/` — agent loop, executor, capture, safety, state, memory, verification, bridge
- `bionics_tools/` — tool modules (grouped by category)
- `plugins/BionicsBridge/` — UE5 C++ plugin
- `gui/` — PyQt6 interfaces
- `plans/` — pre-built automation sequences
- `tests/` — pytest suite
Tool authoring is a single `@bionics_tool` decorator — see `bionics_tools/ue5_niagara.py` for a minimal example (2 tools, 192 lines).
## License
All rights reserved. Source available for portfolio review; not licensed for reuse. See [LICENSE](LICENSE).
## Links
- BionicsBridge (C++ plugin, native latency): [`plugins/BionicsBridge/README.md`](plugins/BionicsBridge/README.md)
- Anthropic Claude API: https://docs.anthropic.com
- Model Context Protocol: https://modelcontextprotocol.io
- FastMCP: https://gofastmcp.com
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available