Content
# tingly-computer-use
A macOS computer-use toolkit for AI agents. Exposes screen capture, accessibility-tree introspection, and input simulation as MCP tools, backed by a Swift native gRPC server.
```
┌─────────────────┐ stdio MCP ┌──────────────┐ gRPC/UDS ┌─────────────────────┐
│ AI agent host │ ────────────► │ tingly-cu │ ───────────► │ tingly-cu-native │
│ (Claude/Codex…) │ │ (Go CLI) │ │ (Swift, AppKit) │
└─────────────────┘ └──────────────┘ └─────────────────────┘
│
▼
macOS Accessibility · ScreenCaptureKit · CGEvent
```
## What it can do
The Go CLI registers eleven MCP tools:
| Tool | Purpose |
|---------------------------|------------------------------------------------------------------|
| `list_apps` | List running + recently used apps (last 14 days, deny-listed apps filtered) |
| `get_app_state` | Focus the app's key window, return PNG screenshot + accessibility tree (must be called once per turn) |
| `snapshot` | **Read-only** PNG + AX tree of an already-running app — never launches, activates, or reopens; fails if the app is not running |
| `click` | Click by `element_index` from the AX tree, or by screenshot pixel coords |
| `type_text` | Type literal Unicode text (per-grapheme keyDown/keyUp pairs) |
| `press_key` | Press a key combo in xdotool syntax (`Return`, `super+c`, …) |
| `scroll` | Scroll an element by direction + fractional pages |
| `drag` | Drag from/to screenshot pixel coords |
| `perform_secondary_action`| Invoke a non-default AX action (e.g. `AXShowMenu`) |
| `set_value` | Set the value of a settable AX element |
| `turn_ended` | Clear per-turn snapshot cache (host signals end of agent turn) |
### Why no `open_app` / dedicated `close_app` tools?
These actions are deliberately collapsed into the existing surface rather than exposed as separate tools:
- **Open / launch** is folded into `get_app_state`: passing an `app` that is not running causes the native side to launch and activate it, then snapshot — one round-trip instead of two, and no race between launch and capture.
- **Snapshot vs `get_app_state`**: use `snapshot` when you only want to *observe* an app the user already has open (no focus stealing, no launching). Use `get_app_state` when you intend to *act* on the app and need it focused.
- **Close / quit** is intentionally not exposed: terminating other processes is high-risk. The agent can dismiss windows or quit gracefully via `press_key` (`super+w` / `super+q`), which is auditable by the host and still subject to the deny list.
## Requirements
- macOS **15+** (gRPC-Swift v2 generated code requires macOS 15)
- Xcode / Swift toolchain 6.0+
- Go 1.23+
- Permissions, granted to the binary that hosts the agent (the MCP client process):
- **Accessibility** — System Settings → Privacy & Security → Accessibility
- **Screen Recording** — System Settings → Privacy & Security → Screen Recording
Run `task doctor` to check both at once.
## Download (pre-built binaries)
Each tagged release publishes a macOS arm64 (Apple Silicon) tarball on the [Releases page](../../releases). If you just want to run the MCP server, download instead of building:
```bash
# Replace VERSION with the tag you want (e.g. 1.0.0)
VERSION=1.0.0
NAME="tingly-cu-${VERSION}-macos-arm64"
curl -LO "https://github.com/<owner>/<repo>/releases/download/v${VERSION}/${NAME}.tar.gz"
curl -LO "https://github.com/<owner>/<repo>/releases/download/v${VERSION}/${NAME}.tar.gz.sha256"
shasum -a 256 -c "${NAME}.tar.gz.sha256" # verify integrity
tar -xzf "${NAME}.tar.gz"
# The two binaries sit side-by-side, so the Go wrapper auto-discovers the
# native bridge — no TINGLY_CU_NATIVE env required.
./${NAME}/tingly-cu doctor
```
Point your MCP client at `<extracted-dir>/tingly-cu` (see [MCP client configuration](#mcp-client-configuration) below).
> **First run on macOS — Gatekeeper notice**
>
> The release binaries are **not codesigned or notarized**. The first time you run them, macOS will refuse with "Apple cannot check it for malicious software". Bypass once:
>
> - **Finder**: right-click `tingly-cu` (and `tingly-cu-native`) → **Open** → confirm in the dialog. Subsequent runs are unrestricted.
> - **Or via terminal**: `xattr -d com.apple.quarantine tingly-cu tingly-cu-native`.
>
> If you prefer signed binaries, build from source — `task release` produces unquarantined binaries directly.
Architectures other than arm64 (Intel x86_64, etc.) are not currently published. Build from source via the [Build](#build) section if you need them.
## Build
```bash
# Build Swift native server (debug) + Go CLI in place (for `task mcp` dev loop)
task build
# Release build of both binaries, in place
task build:release
# Release build + copy both binaries side-by-side into ./dist
# (this is what MCP clients should point at — see "MCP client configuration")
task release
```
Build outputs:
- `task build` / `build:release` → `swift/.build/{debug,release}/tingly-cu-native`, `./tingly-cu`
- `task release` → `dist/tingly-cu`, `dist/tingly-cu-native`
The Go binary auto-locates `tingly-cu-native` via, in order:
1. `TINGLY_CU_NATIVE` env var
2. Same directory as the Go binary
3. `$PATH`
`dist/` puts the two binaries in the same directory, so step 2 fires and **no env var is needed** in MCP client configs.
## Run
```bash
# Start the MCP stdio server (auto-spawns the Swift native process)
task mcp # or: go/tingly-cu mcp
# Diagnostic / utility commands
task doctor # Check macOS permissions
task list-app # List running apps
task ax -- Safari # Dump accessibility tree of an app
task snap -- Safari # Save screenshot to ./snap.png
```
## MCP client configuration
`tingly-cu mcp` is a standard **MCP stdio server** (JSON-RPC 2.0 over stdin/stdout). Any MCP-capable host can use it by spawning the binary with `mcp` as its only argument.
**Set up once:**
```bash
task release # builds release binaries into ./dist
./dist/tingly-cu doctor # must report OK on Accessibility + Screen Recording
```
`task release` co-locates `tingly-cu` and `tingly-cu-native` under `dist/`, so the Go wrapper auto-discovers the native bridge — **MCP configs only need to point at `dist/tingly-cu`, no `TINGLY_CU_NATIVE` env required**.
Grant the two macOS permissions to **the host process that will spawn `tingly-cu`** (e.g. Claude Desktop.app, your terminal, VS Code), not to `tingly-cu` itself — macOS attributes the access to the parent.
In every example below, replace `/abs/path/to/tingly-computer-use/` with your actual absolute path to the repo.
### Claude Desktop
Edit `~/Library/Application Support/Claude/claude_desktop_config.json`:
```json
{
"mcpServers": {
"tingly-computer-use": {
"command": "/abs/path/to/tingly-computer-use/dist/tingly-cu",
"args": ["mcp"]
}
}
}
```
Restart Claude Desktop. The tools appear under the 🔨 menu; if not, check `~/Library/Logs/Claude/mcp*.log`.
### Claude Code (CLI)
```bash
claude mcp add tingly-computer-use \
-- /abs/path/to/tingly-computer-use/dist/tingly-cu mcp
claude mcp list # verify it shows up
```
Use `--scope user` to register globally instead of per-project.
### Codex CLI
Add to `~/.codex/config.toml`:
```toml
[mcp_servers.tingly-computer-use]
command = "/abs/path/to/tingly-computer-use/dist/tingly-cu"
args = ["mcp"]
```
### Cursor / Cline / Continue / opencode / Gemini CLI
These hosts all accept the same stdio shape — only the config file path differs:
- Cursor: `~/.cursor/mcp.json` (or project-level `.cursor/mcp.json`)
- Cline (VS Code): `cline_mcp_settings.json` via the Cline panel
- Continue: `~/.continue/config.json` under `experimental.modelContextProtocolServers`
- opencode: `~/.config/opencode/config.json`
- Gemini CLI: `~/.gemini/settings.json` under `mcpServers`
Use the same `command` + `args` pair shown for Claude Desktop above.
### Verify the connection
From any client, calling `list_apps` should return the running app list. If a call hangs or returns "native bridge not available":
1. Re-run `./dist/tingly-cu doctor` — permission revocations are silent.
2. Confirm `dist/tingly-cu-native` exists next to `dist/tingly-cu` (re-run `task release` if not).
3. Tail the host's MCP log, or run the server directly to see structured errors:
```bash
TINGLY_CU_LOG_FORMAT=text TINGLY_CU_LOG_LEVEL=debug ./dist/tingly-cu mcp </dev/null
```
## Configuration (env vars)
| Variable | Purpose | Default |
|-----------------------------------------|--------------------------------------------------------------------------------------------------|-----------------------|
| `TINGLY_CU_NATIVE` | Absolute path to `tingly-cu-native` | (resolves automatically) |
| `TINGLY_CU_LOG_LEVEL` | Log threshold: `debug` \| `info` \| `warn` \| `error` | `info` |
| `TINGLY_CU_LOG_FORMAT` | `json` (one JSON object per line) or `text` | `json` |
| `TINGLY_CU_DENYLIST` | Extra bundle IDs (comma-separated) to add to the deny list | (empty) |
| `TINGLY_CU_DENYLIST_FILE` | Path to a file containing one bundle ID per line (`#` starts a comment) | (unset) |
| `TINGLY_CU_ALLOWLIST` | Bundle IDs (comma-separated) that bypass all deny rules | (empty) |
| `TINGLY_CU_ALLOWLIST_ONLY` | If `1`/`true`, run in **whitelist mode** — only allowlisted apps are permitted | `false` |
| `TINGLY_CU_DISABLE_DEFAULT_DENYLIST` | If `1`/`true`, drop the built-in baseline (use only your custom list) | `false` |
**Built-in deny baseline** (terminals, password managers, system security agents, Chrome) lives in [`DenyList.defaultDeniedBundleIDs`](swift/Sources/TinglyComputerUseKit/DenyList.swift). Add your own via `TINGLY_CU_DENYLIST_FILE`:
```text
# ~/.config/tingly/denylist.txt
com.apple.Safari # don't let agents drive my browser
com.tinyspeck.slackmacgap
```
```bash
export TINGLY_CU_DENYLIST_FILE=~/.config/tingly/denylist.txt
```
## Logging
Both Go and Swift sides emit JSON-line logs to stderr:
```json
{"ts":"2026-04-26T10:11:22.345Z","level":"info","msg":"native server listening","socket":"/tmp/tingly-cu-501.sock"}
{"ts":"2026-04-26T10:11:23.012Z","level":"warn","msg":"no window for app, reopening","app":"Safari"}
```
For human-readable output during development:
```bash
TINGLY_CU_LOG_FORMAT=text TINGLY_CU_LOG_LEVEL=debug task mcp
```
## Layout
```
proto/computeruse/v1/computeruse.proto contract shared by Swift + Go
swift/Sources/TinglyComputerUse/ CLI entry (serve | doctor | version)
swift/Sources/TinglyComputerUseKit/ gRPC service, AX traversal, input simulation
go/cmd/tingly-cu/ CLI entry (mcp | doctor | ls-apps | ax | snap | version)
go/internal/bridge/ spawns native, gRPC client wrapper
go/internal/mcpserver/ MCP tool registration
go/internal/tools/ per-tool argument coercion + dispatch
build/ build/gen-proto scripts
Taskfile.yml task runner targets
```
## Regenerating proto code
```bash
# Prerequisites:
# brew install protobuf swift-protobuf
# go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
# go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
# (Swift gRPC v2 plugin from grpc/grpc-swift)
task gen
```
Generated artifacts (`go/pkg/proto/...`, `swift/Sources/TinglyComputerUseKit/Generated/...`) are checked in.
## Status & limitations
- **macOS-only.** Linux/Windows targets are not implemented.
- **No tests** in this branch yet — Swift `Tests/` is empty, no `*_test.go`. Integration tests against a fixture app are planned.
- The screenshot is base64-encoded into the MCP `CallToolResult`; very large Retina screens can stress the agent's token budget. Streaming-via-resource is on the roadmap.
- `tingly-cu-native` runs in the foreground with no auto-restart-on-crash; the Go side cleans up on `SIGINT/SIGTERM` but does not yet supervise the child.
## License
See [LICENSE](LICENSE).
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available