Content
# talkthrough-mcp
<!-- mcp-name: io.github.korovin-aa97/talkthrough-mcp -->
[](https://github.com/korovin-aa97/talkthrough-mcp/actions/workflows/ci.yml)
[](LICENSE)
[](pyproject.toml)
[](https://pypi.org/project/talkthrough-mcp/)
[Quickstart](#quickstart) · [Tools](#tools) · [FAQ](#faq) ·
[Troubleshooting](docs/TROUBLESHOOTING.md) · [Changelog](CHANGELOG.md) ·
[Contributing](CONTRIBUTING.md)
**Feedback ingestion for AI agents.** Record your screen and talk; your agent
does the rest — files the bugs, writes the spec, builds the backlog.
`talkthrough-mcp` is a local-first MCP server that turns a narrated screen
recording (or any video/audio file) into agent-ready structured data:
timestamped transcript segments, scene-change keyframes, OCR'd on-screen text,
and wall-clock anchoring. Everything is served through lazy retrieval tools, so
a 30-minute recording never floods the model context — the agent pulls exactly
the transcript slice, moment bundle, or frame it needs.
There is no LLM inside the server and no cloud anywhere in the path: ffmpeg,
faster-whisper, and RapidOCR run on your machine, and the calling agent brings
the intelligence. What makes it different from screen-recorder SaaS and
video-analyzer MCPs: it works on arbitrary local files, it ships the agent
workflows (server prompts + example agents), and it anchors every timestamp to
**wall-clock time** — so "the moment I said the checkout hung" maps straight to
the right window of your server logs.
## Quickstart
One command, no system dependencies: ffmpeg falls back to a bundled build,
OCR is pip-only, and whisper models download themselves on first use.
<!-- gen:install — generated by scripts/gen_integrations.py; do not hand-edit -->
[](https://cursor.com/en/install-mcp?name=talkthrough&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ0YWxrdGhyb3VnaC1tY3AiXX0%3D)
[](https://vscode.dev/redirect/mcp/install?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%2C%22type%22%3A%22stdio%22%7D)
[](https://insiders.vscode.dev/redirect/mcp/install?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%2C%22type%22%3A%22stdio%22%7D&quality=insiders)
[](https://lmstudio.ai/install-mcp?name=talkthrough&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ0YWxrdGhyb3VnaC1tY3AiXX0%3D)
[](https://kiro.dev/launch/mcp/add?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%7D)
[](https://block.github.io/goose/extension?cmd=uvx&arg=talkthrough-mcp&id=talkthrough&name=Talkthrough&description=Turn%20narrated%20screen%20recordings%20into%20agent-ready%20data)
### Claude Code
Two install paths — **pick one**, not both (the plugin already includes
the server; installing both would register it twice):
**Server only** — the 7 tools + 5 prompts, and nothing else on your
system. Choose this for a minimal setup, or when you manage MCP servers
yourself across several clients:
```bash
claude mcp add -s user talkthrough -- uvx talkthrough-mcp
```
**Full plugin** — the same server, plus native slash commands
(`/talkthrough:triage-recording`, …) that handle the ceremony for you,
a ready-made triage subagent, and an agent skill that teaches Claude the
workflow. Choose this for the best out-of-the-box experience:
```
/plugin marketplace add korovin-aa97/talkthrough-mcp
/plugin install talkthrough@talkthrough
```
### Every other MCP client
<details>
<summary><b>Claude Desktop</b></summary>
`claude_desktop_config.json`:
```json
{
"mcpServers": {
"talkthrough": {
"command": "uvx",
"args": [
"talkthrough-mcp"
]
}
}
}
```
More: [`integrations/claude-desktop/`](integrations/claude-desktop/)
</details>
<details>
<summary><b>Cursor</b></summary>
`~/.cursor/mcp.json (or project .cursor/mcp.json)`:
```json
{
"mcpServers": {
"talkthrough": {
"command": "uvx",
"args": [
"talkthrough-mcp"
]
}
}
}
```
More: [`integrations/cursor/`](integrations/cursor/)
</details>
<details>
<summary><b>OpenAI Codex CLI</b></summary>
`~/.codex/config.toml (or project-scoped .codex/config.toml in trusted projects)`:
```toml
[mcp_servers.talkthrough]
command = "uvx"
args = ["talkthrough-mcp"]
```
More: [`integrations/codex/`](integrations/codex/)
</details>
<details>
<summary><b>Gemini CLI</b></summary>
`~/.gemini/settings.json`:
```json
{
"mcpServers": {
"talkthrough": {
"command": "uvx",
"args": [
"talkthrough-mcp"
]
}
}
}
```
More: [`integrations/gemini-cli/`](integrations/gemini-cli/)
</details>
<details>
<summary><b>Cline / Roo Code</b></summary>
`cline_mcp_settings.json (via MCP Servers UI)`:
```json
{
"mcpServers": {
"talkthrough": {
"command": "uvx",
"args": [
"talkthrough-mcp"
]
}
}
}
```
More: [`integrations/cline/`](integrations/cline/)
</details>
<details>
<summary><b>OpenClaw</b></summary>
`~/.openclaw/openclaw.json`:
```json
{
"mcp": {
"servers": {
"talkthrough": {
"command": "uvx",
"args": [
"talkthrough-mcp"
]
}
}
}
}
```
More: [`integrations/openclaw/`](integrations/openclaw/)
</details>
<details>
<summary><b>OpenCode</b></summary>
`opencode.json (project) or ~/.config/opencode/opencode.json`:
```json
{
"mcp": {
"talkthrough": {
"type": "local",
"command": [
"uvx",
"talkthrough-mcp"
],
"enabled": true
}
}
}
```
More: [`integrations/opencode/`](integrations/opencode/)
</details>
<details>
<summary><b>Goose</b></summary>
`~/.config/goose/config.yaml`:
```yaml
extensions:
talkthrough:
enabled: true
type: stdio
cmd: uvx
args: ["talkthrough-mcp"]
```
More: [`integrations/goose/`](integrations/goose/)
</details>
<details>
<summary><b>GitHub Copilot CLI</b></summary>
`~/.copilot/mcp-config.json`:
```json
{
"mcpServers": {
"talkthrough": {
"command": "uvx",
"args": [
"talkthrough-mcp"
]
}
}
}
```
More: [`integrations/copilot-cli/`](integrations/copilot-cli/)
</details>
<details>
<summary><b>Windsurf</b></summary>
`~/.codeium/windsurf/mcp_config.json`:
```json
{
"mcpServers": {
"talkthrough": {
"command": "uvx",
"args": [
"talkthrough-mcp"
]
}
}
}
```
More: [`integrations/windsurf/`](integrations/windsurf/)
</details>
<details>
<summary><b>Zed</b></summary>
`settings.json (Zed)`:
```json
{
"context_servers": {
"talkthrough": {
"source": "custom",
"command": {
"path": "uvx",
"args": [
"talkthrough-mcp"
]
}
}
}
}
```
More: [`integrations/zed/`](integrations/zed/)
</details>
Any other MCP stdio client uses the same server command: `uvx talkthrough-mcp`.
Per-engine folders with exactly these snippets plus verification steps live
in [`integrations/`](integrations/); agents can self-install via
[`llms-install.md`](llms-install.md).
<!-- /gen:install -->
### Local checkout (development)
```bash
git clone https://github.com/korovin-aa97/talkthrough-mcp
claude mcp add talkthrough -- uv run --directory /path/to/talkthrough-mcp talkthrough-mcp
```
Then, in your agent:
> Process `~/Desktop/recording.mov` and triage it — or just invoke the
> `triage-recording` server prompt.
## Tools
| Tool | What it does |
|---|---|
| `process_media(path, recorded_at?, vocabulary?, language?, model?, force?)` | Ingest a video/audio file: local STT, keyframes, OCR, wall-clock. Returns a compact summary. Idempotent by content hash — re-calls are instant. |
| `get_transcript(job_id, start_ms?, end_ms?, format?)` | Paginated transcript as `segments`, `text`, or `srt`; truncation returns `next_start_ms`. |
| `get_frames(job_id, at_ms? \| start_ms?+end_ms?, max_frames?, include_duplicates?)` | Keyframe images nearest a timestamp or evenly thinned across a range (unique frames by default, max 6/call). |
| `get_moment(job_id, start_ms, end_ms)` | The "one remark" bundle: transcript slice + up to 3 frames + their OCR text + wall-clock range. |
| `search(job_id, query)` | Substring search over the transcript AND on-screen OCR text; hits carry `t_ms`/`t_wall` and frame refs. |
| `extract_frame(job_id, at_ms, crop?)` | Exact-timestamp full-resolution re-extract from the source video (optional crop) when keyframes miss the instant. |
| `list_jobs()` | Recent processed recordings with durations, wall-clock starts, and counts. |
Every tool description ships 10+ usage examples, so agents pick the right tool
without extra prompting.
## Server prompts (slash commands in MCP clients)
| Prompt | Workflow |
|---|---|
| `triage-recording` | Narrated screencast → precise findings JSON (bug/feature/question routing, frame evidence) |
| `spec-from-workshop` | Recorded workshop → structured spec with quoted decisions and open questions |
| `backlog-from-demo` | Product demo → prioritized backlog with timestamped evidence |
| `meeting-actions` | Meeting audio → action items, decisions, open questions |
| `correlate-with-logs` | Recording remarks ↔ system logs via wall-clock windows |
The same prompts live as plain files in [`examples/prompts/`](examples/prompts/)
if your client doesn't surface MCP prompts. The findings contract used by
`triage-recording` is [`examples/output-contract.schema.json`](examples/output-contract.schema.json).
## Works as a skill too (no MCP required)
The same workflow ships as a cross-engine [Agent Skill](https://agentskills.io)
at [`.agents/skills/talkthrough/`](.agents/skills/talkthrough/) — Claude Code,
Codex CLI (`$talkthrough`), Cursor, Copilot, Gemini CLI, Goose and other
SKILL.md-compatible tools read it. Agents without MCP wiring can drive the
[CLI](#cli) directly: `talkthrough-mcp process recording.mov --json` prints the
same summary the MCP tool returns, and the job store is shared either way.
## Wall-clock anchoring
Every timestamped result carries both `t_ms` (video-relative) and `t_wall`
(ISO 8601 real time) once the recording start is known. Resolution ladder:
1. `recorded_at` parameter (agent/user override) → confidence `exact`
2. QuickTime `com.apple.quicktime.creationdate` tag, carries the local
timezone (QuickTime Player recordings; ⌘⇧5 wrote it before macOS 26) → `high`
3. Container `creation_time` tag (UTC) → `medium` — macOS 26+ ⌘⇧5/ReplayKit
screen recordings land here (no `creationdate` tag anymore); pass
`recorded_at=` when local-tz `t_wall` matters
4. File mtime minus duration (recorders finalize files at recording END) → `low`
5. Nothing → tools still work with relative `t_ms` only
Why it matters: "the upload spinner froze *here*" becomes a ±30 s grep window
in your server logs.
## Privacy
Everything runs locally: your recordings never leave your machine, speech is
transcribed by a local whisper model, OCR is local ONNX inference, and there is
no telemetry. The only network access is one-time tool/model downloads (ffmpeg
build, whisper model, OCR models).
## Languages
Narration in any of Whisper's ~99 languages works: the language is
auto-detected per recording, and the summary reports both `language` and
`language_probability` so agents can tell a confident detection from a shaky
one (silence or music at the start can fool the detector — pin it with
`language="ru"` and `force=true` when that happens).
Pick the model for your languages — per call (`model=` parameter, agents do
this themselves when a transcript comes back garbled) or as the server
default (`TALKTHROUGH_WHISPER_MODEL`):
| Model | Size | Best for |
|---|---|---|
| `small` (default) | 464 MB | English and major-language narration on CPU |
| `large-v3-turbo` | ~1.5 GB | **recommended for non-English** — near-large quality at near-small speed |
| `medium` | ~1.5 GB | conservative alternative to turbo |
| `tiny` / `base` | 75–145 MB | quick drafts, CI |
| `*.en` variants | — | English-only, slightly faster/better for EN |
Tips that work in every language: pass product names via
`vocabulary="Term1, Term2"` (biases the decoder so jargon survives), and note
that the workflow prompts instruct agents to write digests in the
**narrator's language** while keeping quotes verbatim — the server never
translates (exact quotes are evidence; translation is the agent's job).
On-screen text (OCR) defaults to RapidOCR's Latin + Chinese models. For other
scripts set `TALKTHROUGH_OCR_LANG` to your language — `ru`/`uk` (→ the
`eslav` pack), `ja`, `ko`, `ar`, `hi`, `el`, `th`, or any RapidOCR pack name
like `cyrillic` — and reprocess with `force=true`; the matching recognition
model downloads once. Spoken-language support is unaffected either way.
## Configuration
| Env var | Default | Meaning |
|---|---|---|
| `TALKTHROUGH_WHISPER_MODEL` | `small` | default whisper model (`tiny`/`base`/`small`/`medium`/`large-v3`/`large-v3-turbo`); the `model` tool param overrides per call |
| `TALKTHROUGH_OCR` | `on` | set `off` to skip OCR |
| `TALKTHROUGH_OCR_LANG` | Latin+Chinese | recognition script for on-screen text: a language code (`ru`, `ja`, `ko`, `ar`, `hi`, …) or a RapidOCR pack name (`eslav`, `cyrillic`, `latin`, …); the model downloads once |
| `TALKTHROUGH_OCR_PARAMS` | — | advanced: JSON object of raw RapidOCR params merged over the derived ones, e.g. `{"Rec.lang_type": "cyrillic"}` |
| `TALKTHROUGH_MAX_SECONDS` | `7200` | max media duration |
| `TALKTHROUGH_MAX_FRAMES` | `600` | keyframe cap per job |
| `TALKTHROUGH_HOME` | `~/.talkthrough` | job store root |
## CLI
The pipeline is also a CLI — useful for pre-processing long recordings outside
an agent session (the store is content-addressed, so the agent then queries the
same job instantly):
```bash
talkthrough-mcp process ~/Videos/long-session.mov # prints the summary
talkthrough-mcp process demo.mov --json # machine-readable
talkthrough-mcp gc --keep-days 30 # clean the job store
talkthrough-mcp serve # stdio MCP server (default)
```
First run notes: missing system ffmpeg triggers a one-time `static-ffmpeg`
download; the first transcription downloads the whisper model (~460 MB for
`small`); both are cached. After that, expect roughly 3× faster than real time
on an Apple-Silicon CPU with the default model, OCR included (a 2-minute clip
processes in ~40 s) — and instant re-runs on the same file. Progress streams
as MCP progress notifications, and the CLI prints stage lines. More:
[docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
## Windows (best-effort)
CI runs lint, the unit suite, and a full CLI smoke on `windows-latest`
(static-ffmpeg Windows build, whisper `tiny` transcription, OCR, and the
instant idempotent re-run). Notes: the per-job lock is POSIX `fcntl` and
degrades to a no-op on Windows — fine for a single-user machine; quote paths
with spaces (`uv run talkthrough-mcp process "C:\Videos\Screen Recording.mp4"`).
Windows is not a release gate — if something breaks, please open an issue.
## Supported inputs
Video: `.mov` `.mp4` `.webm` `.mkv` — audio-only: `.m4a` `.mp3` `.wav` `.ogg`
`.flac` (transcript tools only; frame tools explain why they're unavailable).
Local files only.
## Limitations
Honest edges, so you can decide fast:
- **One speaker stream.** No diarization yet — "who said it" isn't tracked
([#4](https://github.com/korovin-aa97/talkthrough-mcp/issues/4)).
- **Local files only.** No URL/YouTube ingestion
([#5](https://github.com/korovin-aa97/talkthrough-mcp/issues/5)) — download
first.
- **Keyframes + transcript, not motion analysis.** A glitch *between* scene
changes can be invisible in the frame set; `extract_frame` re-checks any
instant, but frame-by-frame motion reasoning is your multimodal model's job.
- **STT quality tracks the model you pick.** The default `small` favors speed;
non-English narration wants `model="large-v3-turbo"` (see
[Languages](#languages)).
- **OCR reads crisp UI text well;** tiny or low-contrast print is best-effort.
- **Wall-clock confidence depends on recorder metadata** — worst case pass
`recorded_at=` (see the ladder above).
- **Windows is best-effort** (see above).
## How it compares
| | talkthrough | cloud recorder SaaS | meeting notetakers | typical video-analyzer MCPs |
|---|---|---|---|---|
| Runs fully locally | ✅ | ❌ | ❌ | varies |
| Any local video/audio file | ✅ | browser/app captures | meetings only | ✅ |
| Wall-clock anchoring (log correlation) | ✅ | ❌ | ❌ | ❌ |
| Ships agent workflows (prompts, skill, findings contract) | ✅ | ❌ | ❌ | ❌ |
| OCR of on-screen text, searchable | ✅ | some | ❌ | rare |
## FAQ
**Why not just upload the video to a multimodal model (e.g. Gemini)?**
For a short, non-sensitive clip — do that. The trade-offs appear with length
and sensitivity: an hour of screen recording costs on the order of a million
tokens *per question*, the file leaves your machine, and you still can't map a
remark to `14:32:07 UTC` to grep your server logs. talkthrough indexes once,
locally, then answers any number of follow-ups from the index.
**Why not screenpipe?**
Different job. screenpipe is an always-on recorder of *your* machine going
forward (commercial license). It can't open the `.mov` a teammate or customer
just sent you. talkthrough analyzes any file it's handed — the two compose
fine.
**There are agent skills that "watch" videos. Why a server with an index?**
Watch-style skills push a budgeted frame dump into the context window (and go
sparse on long videos), often call cloud STT for the audio, and keep nothing.
talkthrough builds a persistent local index — transcript + OCR, full-text
searchable — retrieves exact frames lazily, anchors everything to wall-clock
time, and answers the next question without reprocessing.
**I use Jam for bug reports — do I need this?**
Keep Jam for browser bugs: console+network captured at record time is great
evidence. talkthrough covers what a browser extension can't — desktop apps,
mobile screencasts, ops incidents, meetings, any file — with no account, and
correlates with *server-side* logs via wall-clock time.
**Can't I just script ffmpeg + whisper myself?**
Yes — that's exactly this pipeline. What you'd be rebuilding: scene-change
detection with perceptual dedup, OCR, transcript+OCR search, the wall-clock
ladder, MCP tools with embedded usage examples, five workflow prompts, and a
findings contract. One `uvx` command instead of an afternoon of glue.
**Is it really local? What leaves my machine?**
Nothing at runtime. The network is used only for one-time downloads (ffmpeg
build, whisper/OCR models). No telemetry. See [Privacy](#privacy) — and
[SECURITY.md](SECURITY.md) treats a violation of this promise as a
vulnerability.
## For agents & tooling
Machine-readable entry points, so AI agents can install and use this server
without a human reading docs:
- [`llms-install.md`](llms-install.md) — step-by-step install instructions for agents
- [`llms.txt`](llms.txt) — index of the documentation
- [`.agents/skills/talkthrough/SKILL.md`](.agents/skills/talkthrough/SKILL.md) — an [Agent Skill](https://agentskills.io) teaching the tool workflow; discovered automatically inside a checkout by Codex CLI (`$talkthrough`) and readable by Claude Code, Cursor, Copilot, Gemini CLI and other SKILL.md-compatible tools
- [`AGENTS.md`](AGENTS.md) — instructions for coding agents contributing to this repo
- [`server.json`](server.json) — MCP registry manifest
- [`integrations/`](integrations/) — per-engine adapters, all generated from one source of truth and drift-tested (incl. the Claude Code plugin under [`integrations/claude-code/`](integrations/claude-code/))
## Roadmap (not in v1)
URL/YouTube ingestion · speaker diarization · cloud STT · embeddings/semantic
search · hosted/remote mode · `.mcpb` bundle · whisper.cpp backend
## License
MIT
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available