Content
# bg-agent
*A keyframe from an autonomous run: the agent navigates Candlekeep through JSON world state alone.*
Play **Baldur's Gate** (and other Infinity Engine games) through a **structured text / MCP
interface** instead of a screen. An LLM agent reads JSON world state and issues verb actions
(`move_to`, `attack`, `talk`, `loot`, `choose_dialogue`, …) — no vision, no pixel coordinates.
It works by embedding a small bridge inside [GemRB](https://github.com/gemrb/gemrb) (the
open-source Infinity Engine reimplementation, which embeds CPython), plus a thin MCP server
the agent connects to. **The original game binary is never touched, and no game data is
included or acquired** — you point it at a copy you own.
> Status: working prototype. **Perception, navigation, looting, pause/camera control work**
> end-to-end on both GemRB's bundled demo game and classic Baldur's Gate 1. **Dialogue with
> NPCs is partially wired** (see `docs/AGENT_FAILURES.md`). True-headless rendering is a
> tracked follow-up; it currently runs in a window. Built/verified on macOS (Apple Silicon).
## How it works
```
LLM agent (e.g. Claude Code)
│ MCP tool calls: get_state, move_to, attack, talk, loot, choose_dialogue, ...
mcp/server.py ← thin, typed MCP wrapper (knows nothing about engine internals)
│ line-delimited JSON over local TCP {id,method,params} -> {id,ok,data|error}
bridge/bg_bridge.py ← runs INSIDE GemRB's embedded Python
├─ background thread: blocking socket I/O -> thread-safe queue (no engine calls here)
└─ GemRB.SetTimer pump on the engine's MAIN thread: drains queue, calls GemRB.*, replies
GemRB engine (C++, + small patches) ── your Infinity Engine game data
```
Design notes and the full reverse-engineered GemRB API surface are in `docs/FINDINGS.md`;
the architecture and milestone plan are in `docs/PLAN.md`.
## What's in here
```
bridge/ in-engine GemRB Python: bg_bridge.py (state+verbs+socket), bg_quickstart.py,
and the demo probes (bg_probe.py, bg_action.py)
engine-patches/ small C++/CMake/GUIScript patches applied to a GemRB checkout:
0001 — headless CLI build + GetAreaActors/GetAreaObjects/DialogStart/
TakeAllFromContainer/GetDialogState helpers
0002 — GUIScript auto-start hooks (demo + bg1 quickstart)
cpp/ readable copies of the C++ helpers (the patch is authoritative)
mcp/ server.py (FastMCP) + requirements.txt
context/ CLAUDE.md — verb API + AD&D 2e basics the agent reads before playing
test/ client.py (protocol smoke test), bg.py (one-shot CLI), play_demo.py
config/ demo.cfg.sample / bg1.cfg.sample (run.sh generates the real ones)
docs/ FINDINGS.md, PLAN.md, AGENT_FAILURES.md
setup.sh clone GemRB, apply patches, install bridge scripts, build
run.sh generate config, sign (macOS), launch engine + bridge
```
## Quick start (macOS, Apple Silicon)
```bash
brew install cmake sdl2 sdl2_mixer python@3.11
# 1) clone GemRB, apply patches, install bridge scripts, build
./setup.sh
# 2) run the bundled DEMO game (no owned data needed) + bridge
./run.sh
# 3) smoke-test the bridge from another shell (no LLM)
python3 test/bg.py get_state
python3 test/client.py
# 4) MCP server for an LLM agent
python3 -m venv mcp/.venv && mcp/.venv/bin/pip install -r mcp/requirements.txt
cp .mcp.json.sample .mcp.json # edit <PROJECT_ROOT>; registers the server for Claude Code
```
## Running your own game (e.g. Baldur's Gate 1)
You must own the game. Point `GAME_PATH` at the folder containing `chitin.key`:
```bash
GAME_TYPE=bg1 GAME_PATH=/path/to/your/bg1 ./run.sh
```
`GAME_TYPE` matches GemRB's (`bg1`, `bg2`, `iwd`, `pst`, …); the auto-start quickstart is
currently wired for `demo` and `bg1`.
## Security
The in-engine bridge is an unauthenticated control socket for the running game. It defaults
to `127.0.0.1` and refuses non-loopback binds unless `BG_AGENT_ALLOW_REMOTE=1` is set. Do not
expose it on a LAN or the internet without your own isolation/authentication layer.
## Build/runtime notes (macOS)
The patches and `run.sh` handle several macOS-specific gotchas, all documented in
`docs/FINDINGS.md §3`: building a headless CLI binary instead of the GUI Cocoa wrapper,
`-DDISABLE_WERROR=1` (newer clang/libc++), skipping the `ibtool` nib step, and **re-signing
Homebrew's SDL2** (GemRB's build invalidates its code signature, which otherwise causes
kernel `SIGKILL` at `dlopen`). Full Xcode is not required.
## Legal
GemRB is the engine only. You must supply Infinity Engine game data from a copy you legally
own. This project ships **no** game assets and never downloads them.
## License
MIT (this project's own code). GemRB itself is GPL-2.0-or-later; the files in
`engine-patches/` are diffs against GemRB and inherit its license when applied.
