Content
# clip-catalog
Turn your **video portfolio** into a **searchable clip catalog**, then cut **natural-language supercut reels** from it at any aspect ratio. clip-catalog ingests finished work from a **local folder** (or a Frame.io project), scene-splits it into per-shot clips, tags each clip with a vision-language model (description, objects, actions, mood, shot type, quality score, commercial category, on-screen-text flag), optionally **transcribes dialog/VO with Whisper**, and catalogs everything in SQLite + FTS5. Search it in plain English and assemble a reel with one command. The whole pipeline is agent-operable — drive it from the CLI, a LAN HTTP API, or an MCP server.
## Quickstart
```bash
git clone <repo-url> clip-catalog && cd clip-catalog
ffmpeg -version # ffmpeg must be on PATH (and Python >= 3.11)
pip install -e ".[mcp,transcribe]" # core + MCP server + Whisper; add ,frameio for Frame.io ingest
cp .env.example .env # add your OpenRouter API key (that's all you need)
clip-catalog ingest-local /path/to/your/portfolio # scene-detect + split into clips
clip-catalog analyze # tag every clip with the VLM
clip-catalog transcribe # make dialog/VO searchable (optional)
clip-catalog supercut "epic aerial drone" --aspect 9:16 --duration 15
```
Organize masters as `Brand/Project/file.mp4` — subfolder names become per-video
context hints that sharpen the VLM analysis. Re-running ingest is idempotent.
### Or with Docker
```bash
cp .env.example .env # add your OpenRouter API key
mkdir -p media && cp your-videos/* media/
docker compose up -d serve # GUI at http://localhost:8787/
docker compose run --rm serve ingest-local /media
docker compose run --rm serve analyze
docker compose run --rm serve transcribe
```
The container encodes with CPU x264. For NVIDIA GPU encoding, run natively (the
encoder auto-detects NVENC) or build on a CUDA ffmpeg base with the NVIDIA
container toolkit.
## Architecture
```
local folder ─┬─► catalog videos (Frame.io alternative: download deliverables)
Frame.io ─────┘
└► PySceneDetect ──► split into per-shot clips + thumbnails
└► VLM analysis ──► description, objects, actions, mood, shot_type,
quality_score (0-10), commercial_category, has_text + graphics_note
└► Whisper transcription (optional) ──► searchable dialog/VO per clip
└► SQLite + FTS5 catalog (full-text searchable)
└► natural-language query ──► select best clips ──► ffmpeg ──► reel
```
Every clip carries a **visually-verified `has_text` flag** (is there an added text/graphic overlay?) so reels can exclude on-screen text — overlays laid out for 16:9 look bad cropped to 4:5/9:16. Reel cuts use `clean_cuts` by default, scanning each clip and cutting from its longest internally-consistent run to dodge hidden cuts PySceneDetect missed and boundary strays.
## Commands
| Command | What it does |
| --- | --- |
| `clip-catalog ingest-local <folder>` | Catalog every video under a local folder, scene-detect, split into clips. No accounts needed. |
| `clip-catalog ingest-url <url...>` | Download videos from YouTube/Vimeo/any yt-dlp site, then scene-detect and split. Needs `pip install -e ".[url]"`. **Only ingest content you have the rights to** — see note below. |
| `clip-catalog ingest` | Same, but sourced from a Frame.io project (downloads finished deliverables). |
| `clip-catalog analyze-both` | Tag unanalyzed clips with two VLMs in parallel from one shared queue (`VISION_MODEL` + `VISION_MODEL_2`). |
| `clip-catalog transcribe` | Whisper speech-to-text per clip so dialog/VO is searchable (`pip install -e ".[transcribe]"`). |
| `clip-catalog search "query"` | Full-text search the catalog in natural language (LLM query expansion by default). |
| `clip-catalog supercut "query" --aspect 4:5` | Build a fast-cut supercut from the best clips. Aspects: `4:5`, `9:16`, `16:9`, `1:1`. |
| `clip-catalog scene "brief"` | **Scene designer**: parse a plain-English brief, apply film-grammar rulesets (shot-size arcs, jump-cut avoidance, pacing curves), cast each slot from the catalog, output a shot list + rendered cut. Templates: coverage, reveal, montage, action, dialogue. |
| `clip-catalog serve` | LAN HTTP API + web GUI for browsing, searching, and cutting reels (binds `0.0.0.0:8787`). |
| `clip-catalog serve-mcp` | MCP server so LAN agents can search + cut reels as tool calls (binds `0.0.0.0:8765`). |
| `clip-catalog status` | Show pipeline progress (videos / clips / analyzed counts). |
Other commands: `analyze` (single-model fallback), `compile` (duration-targeted reel), `context` (per-video brand/product prompts that sharpen the VLM). Run `clip-catalog --help` for all options.
> **URL ingest & rights.** `ingest-url` is a source-neutral downloader built on
> [yt-dlp](https://github.com/yt-dlp/yt-dlp). You are responsible for only ingesting
> material you have the rights to — your own uploads, Creative Commons / public-domain
> works, or licensed footage. Downloading third-party content may violate the host's
> Terms of Service and copyright; the tool does not police this. Set `YTDLP_FORMAT` to
> override the download format (default: best ≤1080p MP4).
## For agents
This package is built to be driven by agents, not just humans.
- **Skill**: [`skills/frameio-clip-reels/SKILL.md`](skills/frameio-clip-reels/SKILL.md) — the full operating guide (ingest → analyze → curate → cut), including the reel-curation patterns that actually produce good reels. Start here.
- **HTTP API** (`clip-catalog serve`, `:8787`): self-describing OpenAPI at `/docs`. Cut a reel with `POST /api/cut`, search with `GET /api/search?q=...`, plus `/api/outputs` and `/api/status`. The web GUI lives at `/`.
- **MCP** (`clip-catalog serve-mcp`, `:8765`): add `http://<host>:8765/mcp` as an HTTP MCP server. Tools: `search_clips`, `cut_reel`, `list_outputs`, `pipeline_status`. Reel URLs point at the HTTP server, so run `serve` alongside it.
## Configuration
All settings are read from a project-local `.env` — copy [`.env.example`](.env.example) and fill it in. For local-folder ingest, the only required value is the vision model key (`OPENROUTER_API_KEY`; `VISION_MODEL` / `VISION_MODEL_2` to choose models). Frame.io V4 OAuth settings (`FRAMEIO_*`) are only needed for `clip-catalog ingest`. Optional: `WHISPER_MODEL` / `WHISPER_LANGUAGE` (transcription), `CLIP_CATALOG_DATA_DIR`, `VIDEO_ENCODER`.
## Cross-platform
Works on Windows, macOS, and Linux. The ffmpeg encoder **auto-detects** the best available accelerator (`h264_nvenc` → `h264_videotoolbox` → `h264_qsv` → `libx264`); set `VIDEO_ENCODER` in `.env` to force one. Both `data/` (catalog + media) and `.env` (your credentials) are gitignored — nothing local or secret is committed.
