Content
# dsh-computer-use
**Computer Use for DeepSeek Harness (macOS)** — let DeepSeek Harness agents read and drive real desktop apps through the accessibility tree, screenshots, and synthesized input.
> An independent, clean-room reimplementation of the "Computer Use" pattern for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness). Not affiliated with or endorsed by OpenAI.
[中文文档](README.zh.md)
## Features
- **10 model-facing tools** registered as native dsh tools (`computer_use_*`):
`list_apps`, `get_app_state`, `click`, `drag`, `press_key`, `scroll`, `select_text`, `set_value`, `type_text`, `perform_secondary_action`
- **Native macOS daemon (Swift)**:
- Accessibility-tree capture with a token-efficient **diff mode**
- Window screenshots (`file://` PNG), gated by Screen Recording permission
- Input synthesis: AX actions first, CGEvent coordinates as fallback
- **dsh-native integration**:
- Cordis function plugin following dsh package conventions (`dsh.bundle` profile layer)
- Approval through `ctx.approval` with optional persistent store
- Per-turn session lifecycle (sessions cleared on `turn/end`)
- Prompt-safety section marking UI content as untrusted observation
- **Security**: peer-authenticated Unix socket (`getpeereid`), restrictive file permissions, TCC guidance
## Architecture
```mermaid
flowchart LR
H[DeepSeek Harness] -->|dsh.bundle layer| P[dsh-computer-use plugin<br/>Cordis function plugin]
P -->|ctx.tools / ctx.approval / ctx.systemPrompt| H
P -->|JSON-RPC 2.0 over Unix socket| D[CUADaemon<br/>Swift native daemon]
D -->|AXUIElement / CGEvent / ScreenCaptureKit| M[macOS]
```
The TypeScript plugin is pure glue: tool schemas, approval, lifecycle. All system access lives in the Swift daemon, which keeps TCC permissions and AX state in one place.
## Requirements
- macOS 14.4+
- [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh` CLI or repo checkout)
- Node.js 20+ and Swift 5.9+ toolchain
- Permissions: **Accessibility** (required) and **Screen Recording** (for screenshots)
## Installation
### 1. Build
```sh
# Native daemon
cd native && swift build -c release
# TypeScript plugin
cd packages/computer-use/dsh-computer-use
npm install && npm run build
```
### 2. Register with dsh
The package declares `dsh.bundle`, so installing it into a profile auto-activates it:
```sh
dsh plugin --profile web add "link:$(pwd)/packages/computer-use/dsh-computer-use"
```
### 3. Configure
Either set environment variables:
```sh
export DSH_COMPUTER_USE_SOCKET="$HOME/.dsh-computer-use/computeruse.sock"
export DSH_COMPUTER_USE_DAEMON="$PWD/dist/DSH Computer Use.app/Contents/MacOS/CUADaemon"
export DSH_COMPUTER_USE_APPROVALS="$HOME/.dsh-computer-use/approvals.json"
```
Or override the plugin row in your profile patch (`~/.dsh/profiles/<profile>/cordis.patch.yml`):
```yaml
- id: computer-use
config:
socketPath: /Users/<you>/.dsh-computer-use/computeruse.sock
daemonCommand: /path/to/dist/DSH Computer Use.app/Contents/MacOS/CUADaemon
daemonArgs:
- '--socket'
- /Users/<you>/.dsh-computer-use/computeruse.sock
toolCallTimeoutMs: 120000
requireAppApproval: true
approvalStorePath: /Users/<you>/.dsh-computer-use/approvals.json
```
### 4. Grant permissions
Add the daemon (or the `DSH Computer Use.app` bundle) to **System Settings → Privacy & Security → Accessibility** and **Screen Recording**. Full walkthrough: [docs/accessibility-setup.md](docs/accessibility-setup.md).
## Usage
Start dsh and ask the model to use the tools:
```sh
dsh web
```
Example prompts:
- "Use `computer_use_list_apps` to show which apps are running."
- "Use `computer_use_get_app_state` to read the VS Code window title."
- "Open the first QQ conversation and read the latest messages."
## Tools
| Tool | Description |
|---|---|
| `computer_use_list_apps` | Running + recently used apps |
| `computer_use_get_app_state` | AX tree (diff by default) + screenshot URL |
| `computer_use_click` / `computer_use_drag` | Click by element index or coordinates / drag |
| `computer_use_press_key` / `computer_use_type_text` | xdotool-style keys / type text |
| `computer_use_scroll` / `computer_use_select_text` / `computer_use_set_value` | Scroll / select / set values |
| `computer_use_perform_secondary_action` | AX secondary actions (e.g. Show Menu) |
## Verification
```sh
cd native && swift test # Swift unit tests
cd packages/computer-use/dsh-computer-use && npm test
./scripts/verify-install.sh # install + config-tree + permission checks
RUN_E2E=1 ./scripts/verify-install.sh # + headless agent smoke test
```
## Project Structure
```text
.
├── packages/computer-use/dsh-computer-use/ # Cordis plugin (TS)
├── native/ # Swift daemon + tests
├── docs/ # setup, virtual cursor, monorepo guides
├── scripts/ # build / install / verify / e2e
├── openspec/ # spec-driven planning artifacts
└── dist/ # DSH Computer Use.app bundle
```
## Known Limitations
- Independent reimplementation; wire protocol derived from black-box observation of the reference implementation (never copied code).
- QQ/Electron-style apps expose limited AX data for message text; OCR may be needed for full content.
- `select_text` depends on the target exposing AX text ranges (best effort).
- ScreenCaptureKit upgrade for screenshots is a planned improvement (currently `CGWindowListCreateImage`).
## License
[MIT](LICENSE)
MCP Store