Content
# Tool List
[](https://github.com/Ye-Zayne/canvas/actions/workflows/ci.yml)
[](./LICENSE)
A long-term fixed conversation canvas plugin in **codex / claude code**. Based on **React Flow + shadcn/ui**, it communicates bidirectionally with Agent through built-in **MCP Server**:
- **Agent → Canvas**: Agent pushes generated text, Markdown, images, videos, audio, and arbitrary files to the canvas in real-time as independent cards, supporting operations like dragging, zooming, stretching, multi-selecting, and node connections.
- **Canvas → Agent**: Select cards on the canvas → "Join Conversation", then use `/canvas-pull` in codex / claude code to bring these contents back into the conversation flow.
## Two Modes (Automatic Switching)
The same canvas automatically chooses rendering mode at runtime:
| Mode | Trigger Condition | Experience |
| --- | --- | --- |
| **Embedded Canvas** (MCP Apps) | Client supports [MCP Apps](https://modelcontextprotocol.io/extensions/apps/overview) (e.g., Claude / Claude Desktop) | Call `canvas_show` to render the canvas **directly within the conversation panel** |
| **Browser Canvas** (Fallback) | Clients that do not support MCP Apps (e.g., current Codex) | Call `canvas_open` to open `http://127.0.0.1:4399` |
> MCP Apps is an open extension of the MCP official protocol. In embedded mode, `ui://canvas/app.html` is just a shell page, and the actual canvas JS/CSS is loaded from the local `bridge-server` via CSP whitelist, so the embedded and browser versions share the same build product. The "Join Conversation" behavior is consistent across both ends (enqueue + `/canvas-pull`).
## Architecture
```
codex / claude code
│
├─ stdio(MCP) ──► bridge-server ──┬─ WebSocket ──► Browser Canvas (React Flow) [Browser Mode]
│ └─ ui://resources ──► Embedded iframe Canvas [MCP Apps Mode]
│ (express hosting frontend + proxying local files)
```
- `packages/canvas-web`: React + Vite + TypeScript + React Flow (`@xyflow/react`, MIT) + shadcn/ui canvas frontend.
- `packages/bridge-server`: Node service, simultaneously an MCP Server (connected by Agent) + WebSocket/REST service (connected by browser) + static hosting + local file proxy.
## Quick Start
### One-Click Installation (Recommended)
```bash
cd ai-canvas
./install.sh
```
The script will automatically complete: environment check → dependency installation → build → registration with Codex / Claude Code.
**No manual configuration file editing required.**
After installation:
1. Completely exit and restart Codex / Claude Code (ensure `Cmd + Q`, closing the window is ineffective)
2. Create a new conversation, input "Open Canvas"
3. Open the returned address in the browser to use
> The script can be re-executed (idempotent). Run it again after code updates, and it will not produce duplicate configurations;
> Before modification, it will automatically backup to `config.toml.bak` / `.claude.json.bak`.
---
### Manual Installation
If you want to control each step manually:
#### 1. Install Dependencies
```bash
cd ai-canvas
pnpm install
```
> If your directory path contains a colon `:` (like this project), pnpm cannot add `.bin` to PATH. The project has a built-in `scripts/run-bin.mjs` to bypass, no extra handling needed.
#### 2. Build
```bash
pnpm build
```
#### 3. Start
```bash
./start.sh
# or
pnpm start
```
The canvas address after startup: **http://127.0.0.1:4399** (port can be modified with environment variable `CANVAS_PORT`).
#### 4. Register with Client
Only register (no rebuild):
```bash
node scripts/register-mcp.mjs "$(pwd)/packages/bridge-server/dist/index.js"
```
Or refer to the manual configuration below to fill in the configuration.
### Development Mode (Hot Update)
Run in two terminals:
```bash
pnpm dev:server # Start bridge-server (HTTP+WS+MCP)
pnpm dev:web # Start Vite development server (http://localhost:5173, proxied /ws /assets /api)
```
## Accessing codex / claude code
> Using `./install.sh` has automatically completed the following configuration, this section is only for manual configuration or troubleshooting.
### Claude Code
Add to `.mcp.json` in the project root or globally:
```json
{
"mcpServers": {
"canvas": {
"command": "node",
"args": ["<absolute path>/ai-canvas/packages/bridge-server/dist/index.js"]
}
}
}
```
MCP's prompts will automatically expose as slash commands, e.g.:
- `/mcp__canvas__canvas-pull` — Pull content from canvas to join conversation
- `/mcp__canvas__canvas-open` — Get canvas address
Resources can be referenced with `@`: `@canvas://selection`, `@canvas://all`.
### Codex
Add to `~/.codex/config.toml`:
```toml
[mcp_servers.canvas]
command = "node"
args = ["<absolute path>/ai-canvas/packages/bridge-server/dist/index.js"]
```
> Note: Agent client will spawn bridge-server via stdio. This process also listens on port 4399 to provide the canvas UI, so **no need to run `start.sh` separately** (unless you want to debug the frontend in development mode).
## MCP Capability List
### Tools (Agent Active Call)
| Tool | Description |
| --- | --- |
| `canvas_show` | **Open canvas embedded** (clients supporting MCP Apps render within conversation panel) |
| `canvas_open` | Return browser canvas address (fallback) |
| `canvas_add_text` | Push text / Markdown card |
| `canvas_add_image` | Push image (local path or URL) |
| `canvas_add_media` | Push video / audio |
| `canvas_add_file` | Push arbitrary file card (downloadable) |
| `canvas_list` | List all card summaries on canvas |
| `canvas_pull` | Pull content from canvas to join conversation (dequeue) |
| `canvas_enqueue` | Internal call in embedded canvas: enqueue selected cards (usually not triggered directly by user) |
### Prompts (slash commands)
- `canvas-pull`, `canvas-open`
### Resources (@ reference)
- `canvas://selection`, `canvas://all`
## Typical Usage
1. Let Agent generate content and push it to the canvas:
> “Draw this plan on the canvas” → Agent calls `canvas_add_text`
> “Put /tmp/demo.mp4 on the canvas” → Agent calls `canvas_add_media`
2. Freely arrange and scale these cards in the browser canvas.
3. Select several cards, click "Join Conversation" on the card or toolbar "Select to Join".
4. Return to codex / claude code, execute `/canvas-pull`, and the selected content will enter the conversation as context.
## Directory Structure
```
ai-canvas/
├─ package.json # pnpm workspace root
├─ pnpm-workspace.yaml
├─ start.sh # One-click startup
├─ scripts/run-bin.mjs # Launcher to bypass bin path containing ":"
└─ packages/
├─ canvas-web/ # Frontend canvas
│ └─ src/
│ ├─ App.tsx
│ ├─ canvas/CanvasBoard.tsx
│ ├─ canvas/shapes/{CanvasCardShape,CardBody}.tsx
│ ├─ components/{Toolbar,QueueDrawer}.tsx
│ ├─ components/ui/{button,badge,sheet}.tsx
│ ├─ hooks/useBridge.ts
│ └─ lib/{types,utils}.ts
└─ bridge-server/ # MCP + WS + REST
└─ src/
├─ index.ts # Entry
├─ mcp.ts # MCP tools/prompts/resources
├─ ws.ts # WebSocket broadcast
├─ store.ts # State + pull queue
├─ assets.ts # Local file proxy
├─ config.ts
└─ types.ts
```
## Instructions and Constraints
- **Local File Access**: Browser cannot directly read disk, local media uniformly proxied by bridge-server's `/assets/:id` (supports Range, video can drag progress).
- **Large Files**: Prefer to send paths instead of base64 to avoid large MCP message bodies.
- **Single Canvas**: MVP is single-user single-canvas memory state; persistence (save/restore) can be extended later.
- **Port**: Default `4399`, can be overridden with `CANVAS_PORT`.
## Debugging (can test without Agent)
```bash
# Only start HTTP+WS, not MCP
node packages/bridge-server/dist/index.js --no-mcp
# Directly push a card to canvas using REST
curl -X POST http://127.0.0.1:4399/api/nodes \
-H 'Content-Type: application/json' \
-d '{"kind":"markdown","title":"Example","content":"# Hello\nFrom REST"}'
```
