Content
# WorkScreen (Gilbreth)
A desktop app that records what you do — clicks, keystrokes, focus
changes, clipboard — through OS accessibility APIs, into a local
SQLite database that you (and your LLM tools) can query later.
## Status
Early. macOS and Windows both have working capture backends (macOS via
CGEventTap + the Accessibility API; Windows via UI Automation + event
hooks). Signed installers (macOS `.dmg`, Windows NSIS) with auto-update
are published via GitHub Releases. The storage schema is not yet stable —
migrations are additive, but queries written against it may need updating.
What exists is capture, meeting recording, on-device transcription and
real-time suggestions. Pattern mining over the captured stream — the
therbligs the name refers to — is the next layer and is not implemented.
## What it captures
One row per atomic action in `actions`:
- `click` — mouse button down, enriched with the AX element (role,
name, value) under the cursor when available
- `text` — typed text after a 300 ms debounce (one row = one "burst")
- `key` — non-printable navigation/editing keys (Enter, Tab, arrows,
Backspace, …)
- `scroll` — wheel events
- `clipboard` — clipboard text
- `focus_change` — frontmost app changed
Every row carries the foreground app context (`app_name`,
`app_bundle_id`, `window_title`) and a session ID. Rows captured
while a password field had focus are masked at the SQL layer
(`text_content` and `element_value` become `'[masked]'`,
`password_flag = true`).
Meetings are captured in parallel: when a video-conference app
starts a call, WorkScreen records the meeting (start/end, app, audio/video
paths) into a `meetings` table and links subsequent actions to it via
`actions.meeting_id`. After the call ends, the audio is transcribed
fully on-device with whisper.cpp into `meeting_transcripts`.
## Real-time suggestions (optional)
While a call is running, WorkScreen can transcribe it live and show short
suggestions in a small always-on-top panel — powered by an agent on your
own machine, talked to over [ACP](https://agentclientprotocol.com).
Nothing leaves the machine except what that agent itself sends.
It works with the coding CLI you already have — Claude Code or Gemini
CLI. Nothing else to install: the interactive CLIs do not speak ACP
themselves, so WorkScreen runs the adapter for you, fetching it with `npx` the
first time if it is not already there. Point `GILB_ASSIST_AGENT` at
something else if you would rather use your own.
It is off by default at three levels: a Cargo feature, a switch in the
app, and a ~570 MB speech model downloaded only if you turn it on. The
prompt is a plain file you can edit —
`~/Documents/Gilb/prompts/realtime_assist.md` — not a string baked into
the binary. See [`docs/assist.md`](./docs/assist.md) for how the
pipeline fits together.
## Requirements
- macOS (Apple Silicon or Intel) or Windows. Linux is out of scope.
- Rust toolchain (stable, edition 2021).
- Node + npm (only for the Tauri shell).
- On macOS: Accessibility and Input Monitoring permissions, granted
in *System Settings → Privacy & Security*. The app exposes an
`open_privacy_pane` command that jumps you to the relevant pane.
## Build and run
```sh
# Tauri shell with the recorder UI.
cd apps/gilb-app-tauri
npm install
npm run tauri dev
# Read-only MCP server over ~/Documents/Gilb/db.sqlite (stdio transport).
# Point an MCP client (e.g. Claude Code) at this binary to query
# recorded activity. See apps/gilb-mcp/help.md for the tool catalog.
cargo run -p gilb-mcp
```
Build options live in `RecordingSettings::from_env`:
| Env var | Default | Effect |
|--------------------------|---------|-----------------------------------------|
| `CAPTURE_EVENTS` | `true` | Toggle the entire capture pipeline |
| `CAPTURE_MOUSE_MOVE` | `false` | Record raw mouse-move events (noisy) |
| `CAPTURE_CLIPBOARD` | `true` | Record clipboard text |
| `CAPTURE_TREE_SNAPSHOTS` | `true` | Periodic full AX tree dumps |
| `RUST_LOG` | varies | Standard `tracing` filter |
## Querying recorded activity from Claude Code
If you installed WorkScreen from a release (the macOS `.dmg`), the read-only
MCP server ships inside the app bundle — no build step needed. The
binary lives at:
```
/Applications/WorkScreen.app/Contents/MacOS/gilb-mcp
```
Register it with Claude Code:
```sh
claude mcp add gilb --scope user /Applications/WorkScreen.app/Contents/MacOS/gilb-mcp
```
Use `--scope user` so the server is available in every project, since
WorkScreen records activity regardless of which repo you're working in. Drop
the flag to register it for the current project only. Confirm it
registered and connected:
```sh
claude mcp list
```
Inside a Claude Code session the `gilb_*` tools are now available (see
[`apps/gilb-mcp/help.md`](./apps/gilb-mcp/help.md) for the full
catalog). The server reads `~/Documents/Gilb/db.sqlite` over stdio; `WorkScreen.app`
itself does not need to be running. If you built WorkScreen from source, point
the same command at the built binary instead (`cargo run -p gilb-mcp`,
or `target/release/gilb-mcp`). See [`INSTALL.md`](./INSTALL.md) for the
end-user install and permissions guide.
## Architecture
Cargo workspace with three runnable apps and fourteen library crates
under `crates/`:
- **`apps/gilb-app-tauri`** — the desktop app (tray + one window).
- **`apps/gilb-mcp`** — read-only MCP server over the recorded database.
- **`apps/gilb-analyzer`** — runs prompt-jobs against your own recorded
activity and pushes findings to a server. Requires credentials most
users will not have; nothing else depends on it.
The capture pipeline is platform-gated behind a `CapturePlatform` trait;
macOS uses CGEventTap + the Accessibility API, Windows uses UI Automation
+ event hooks. A no-op backend keeps the workspace compiling elsewhere,
which is what CI builds on Linux.
See [CLAUDE.md](./CLAUDE.md) for the full crate graph, capture →
DB data flow, and macOS-specific notes (entitlements, signing,
permission prompts).
## Privacy
Capture, storage and transcription are entirely local: the database is
a file in `~/Documents/Gilb/`, whisper.cpp runs on-device, and nothing is
uploaded. The capture pipeline drops events from a fixed block-list of
password managers — 1Password, Bitwarden, KeePassXC, and macOS Keychain
Access — at the source, so those apps never produce rows. For everything
else, rows captured while a password field had focus are masked in-place
as described above.
Two optional parts do leave the machine, and only if you enable them:
- **Real-time suggestions** send the transcribed conversation to the
agent you configured. Where that goes is that agent's business — a
cloud model if it is Claude or Gemini, nowhere if it is a local one.
- **`gilb-analyzer`** posts its findings to a server, and needs
credentials you must supply. Without them it does nothing.
Meeting recording captures screen and audio to disk while a call is
running. That is the point of the feature, but it is worth saying
plainly: those files are as sensitive as the calls themselves.
## License
MIT — see [LICENSE](./LICENSE).
Connection Info
You Might Also Like
No recommendations available