cutforge-studio

<div align="center"> <img src=".github/readme/hero-banner.png" alt="Cutforge Studio — agent-driven video editing" width="100%" /> # Cutforge Studio **Drive professional video composition, generation, and editing — from your coding agent.** [](./LICENSE) [](https://nodejs.org) [](https://www.typescriptlang.org/) [](https://pnpm.io) [](#effects-catalog-for-pro-agent-driven-editing) [](#drive-it-from-your-coding-agent) [](#viral-templates-capcuttiktok-style-fill-in-the-blanks) </div> Cutforge Studio is **not a black-box video agent**. A video is expressed as a readable, diffable, re-renderable plan (`plan.json`) that your agent — and you — can edit; change one line and only that piece re-renders. The agent is the brain; this project ships the **knowledge** (what makes a good video, and which production line to take), the **deterministic capabilities** (render / edit / transcribe / generate — thin wrappers over [`hyperframes`](https://github.com/heygen-com/hyperframes), `ffmpeg`, and `whisper.cpp`), and that **editable IR** — plus a **135+ op effects catalog** with agent-ergonomics primitives (keyframed camera moves, blend-mode and mask-selective compositing, kinetic text, one-call effect pipelines, beat-sync analysis) built for professional agent-driven editing. You talk to your agent in plain language — *"make a 60-second vertical explainer on vector databases with a Chinese voiceover and captions"* — and it reads the material, writes the timeline, and produces the file. --- ## Demo A single effects pipeline chaining four Cutforge-native primitives in one clip — keyframed zoom, blend-mode compositing, a luma-mask region, and keyframed kinetic text: <div align="center"> <img src=".github/readme/demo.gif" alt="Cutforge Studio effects pipeline demo: keyframe-zoom, blend, mask-composite, and kinetic text chained on one clip" width="360" /> </div> --- ## The four production lines Three orthogonal capability axes, plus an automatic pipeline that weaves them together: - **Compose** — script → designed HTML motion graphics → mp4. Explainers, kinetic typography, lower-thirds, data viz, title cards, transitions. **No paid keys.** - **Edit** — cut / join / trim-silence / de-filler / mix / burn-in subtitles / dub / localize real footage you supply, plus highlight selection over long recordings. **No paid keys.** - **Generate** — talking-head or cinematic AI footage and imagery via **your own** provider keys (BYO — OpenAI, Gemini, Doubao; no managed backend, no lock-in). - **Auto (end-to-end)** — when a deliverable needs more than one axis, the agent routes to a single cross-modal `plan.json`: `stage-plan` builds the EDL, `stage-assemble` walks it and delegates each segment back to compose / generate / edit, and a deterministic **delivery guard** (`cutforge plan promise-check`) verifies the finished cut keeps its promise (e.g. real motion, not a silent slideshow) before anything ships. The plan is the contract: *"the footage in the middle, generate an opener, compose the stats, one voiceover over all of it"* becomes one editable file where every segment, caption, and narration line is independently re-renderable. --- ## Viral templates — CapCut/TikTok-style fill-in-the-blanks New: a fifth, zero-thought entry point sitting on top of the four production lines. The library ships **67 preset templates** — a locked motion curve, transition style, and beat-synced click-track music, keyed only by how many media slots you fill: - **6 viral categories**: `velocity`, `shake_zoom`, `flash`, `glow_up`, `slideshow`, `ai_reveal` - **2 aspect ratios per category**: `9:16` (vertical) and `1:1` (square) - **Slot counts 3–7** per category/aspect combo ```bash cutforge template list # browse all 67 presets (id, category, slots, aspect, beats) TPL=templates/velocity-v-5/template.json cutforge template validate "$TPL" a.jpg b.jpg c.jpg d.jpg e.jpg # check media matches the 5 slots cutforge template compile "$TPL" a.jpg b.jpg c.jpg d.jpg e.jpg --out plan.json # fuse into a plan.json cutforge template render "$TPL" a.jpg b.jpg c.jpg d.jpg e.jpg --out reel.mp4 # validate → compile → assemble → promise-check → mp4 ``` **Build your own template and save it for later reuse.** `template create` generalizes the same authoring pipeline the preset library is built from — validated `template.json` + generated click-track music + `edit_beat_sync`-derived beat-map — so a one-off template you like becomes a reusable asset: ```bash cutforge template create --id my-promo --slots 4 --category glow_up --slot-dur 1.5 --aspect 9:16 # → writes templates/my-promo/template.json — reuse it any time with validate/compile/render cutforge template render templates/my-promo/template.json --media a.jpg b.jpg c.jpg d.jpg --out promo.mp4 ``` All five `template` subcommands (`list · validate · compile · render · create`, plus `author` for regenerating the whole preset library) are mirrored 1:1 as MCP tools (`template_list · template_validate · template_compile · template_render`). --- ## Use cases Each is a real prompt you'd give your agent; the router picks the line for you. | You say… | Line | What happens | |---|---|---| | *"Make a 60s vertical explainer on what a vector DB is, Chinese voiceover + captions."* | Compose | HTML motion-graphic scenes → narration (BYO TTS) → burned-in captions → mp4. | | *"Turn this 1-hour screen recording into three 30s highlight clips with captions."* | Edit | `scenes`/`quality` find the good parts, `rank-takes` picks, `trim-silence`/`remove-fillers` tighten, subtitles burned in. | | *"Add an English voiceover + subtitles to product-demo.mp4 and normalize the loudness."* | Edit | transcribe → localize → `speak` → `mix` → `burnsubs` → loudness pass. | | *"Clean the silences and 'um's out of my webcam take and tighten it."* | Edit | deterministic jump-cut with an auditable evidence trail. | | *"Generate a 5s cinematic shot of a city at dawn for the intro."* | Generate | BYO image/video provider → clip on your timeline. | | *"Make a 15s 9:16 promo from this script: generate the opener, compose the feature callouts, one VO."* | Auto | one `plan.json` woven across generate + compose + narration, guarded before delivery. | | *"Turn these 5 photos into a viral TikTok-style reel, no editing skill needed."* | Template | pick a preset, drop in N photos/clips, `cutforge template render` fuses preset motion + beat-synced music + transitions into an mp4. | --- ## Install **Prerequisites:** Node ≥ 20, and `ffmpeg` + `ffprobe` on your `PATH` (needed for edit / transcribe / local media QA). Compose drafts use a VideoStudio QA gate backed by [HyperFrames](https://github.com/heygen-com/hyperframes) at runtime via `npx` (first run needs network). Generation is opt-in and needs your own keys. > **Early development:** the npm packages are being published. Until then, install from source > — the `cutforge` CLI works exactly the same. ### From source (works today) ```bash git clone https://github.com/nguyenminhduc9988/cutforge-studio.git cd cutforge-studio pnpm install && pnpm build node packages/cli/dist/index.js doctor # verify ffmpeg/ffprobe/node # optionally alias it: alias cutforge="node $PWD/packages/cli/dist/index.js" ``` ### From npm (once published) ```bash npm i -g @cutforge/studio # provides the `cutforge` command cutforge doctor # checks ffmpeg/ffprobe/node; guides any install ``` --- ## Drive it from your coding agent The `cutforge` CLI is the canonical interface; the MCP tools mirror it 1:1. There are three ways an agent picks it up — use whichever your agent supports: **1) Native skills (Claude Code / Codex).** Materialize the `SKILL.md` knowledge pack so the agent discovers it by progressive disclosure: ```bash cutforge skills --install --target claude # → ~/.claude/skills (add --scope repo for ./.claude/skills) cutforge skills --install --target codex # → ~/.agents/skills ``` **2) MCP typed tools.** Register the server (mirrors the CLI): ```bash claude mcp add cutforge -- npx -y @cutforge/studio-mcp # Claude Code codex mcp add cutforge -- npx -y @cutforge/studio-mcp # Codex # from source (until published): point it at node <repo>/packages/mcp/dist/index.js ``` **3) Self-describing CLI (any agent that can run a shell).** No native loader needed: ```bash cutforge skills # list the skills cutforge skill video-router # print a skill's full instructions into context cutforge --help # the full command surface ``` **How a session flows.** The agent reads **`video-router`** first (it locks the line), then the relevant stage skills, authors the composition and/or `plan.json`, runs the deterministic ops, and self-verifies with the delivery guard: ```text You: Make a 60s vertical explainer on vector databases, with a Chinese voiceover. Agent: → reads video-router (locks: compose-primary) → reads stage-plan / stage-compose / video-craft → writes composition/index.html + plan.json → cutforge draft composition --out draft.mp4 --report draft-report.json → cutforge plan promise-check plan.json # guard passes → returns draft.mp4 ``` The full command surface: `doctor · draft · render · lint · inspect · snapshot · edit {probe,trim,concat, burnsubs,overlay,extract-frame,loudness,mix,trim-silence,remove-fillers} · transcribe · silence · scenes · quality · plan {validate,summarize,promise-check,rank-takes} · template {list,validate,compile,render,create,author} · speak · image · video · skills`. --- ## BYO generation providers The compose / edit / transcribe trunk is **zero-key**. Generation (image / video / TTS) is opt-in and uses **your** keys — no managed backend, no account binding. Configure via `~/.config/cutforge-studio/config.json` (or `CUTFORGE_CONFIG_DIR`) or environment variables: | Capability | Providers | Env | |---|---|---| | Image (`cutforge image`) | OpenAI-compatible · Gemini · Doubao Seedream | `CUTFORGE_IMAGE_PROVIDER` · `CUTFORGE_IMAGE_BASE_URL` · `CUTFORGE_IMAGE_API_KEY` · `CUTFORGE_IMAGE_MODEL` | | Video (`cutforge video`) | Doubao Seedance (image-to-video) | `CUTFORGE_VIDEO_PROVIDER` · `CUTFORGE_VIDEO_BASE_URL` · `CUTFORGE_VIDEO_API_KEY` · `CUTFORGE_VIDEO_MODEL` | | TTS (`cutforge speak`) | OpenAI-compatible (incl. ElevenLabs-style) | `CUTFORGE_TTS_BASE_URL` · `CUTFORGE_TTS_API_KEY` · `CUTFORGE_TTS_MODEL` · `CUTFORGE_TTS_VOICE` · `CUTFORGE_TTS_FORMAT` | --- ## How it compares HyperFrames, ffmpeg, and whisper.cpp are **dependencies** here, not competitors — Cutforge Studio is the agent-facing knowledge, draft QA gate, and IR layer on top of them. Against other ways to make video: | | Cutforge Studio | Programmatic frameworks (Remotion, Revideo) | AI SaaS editors (Descript, Opus Clip, Runway) | Thin MCP / ffmpeg wrappers | |---|---|---|---|---| | **Driver** | any coding agent, in natural language (CLI + MCP) | you hand-write React/TS | GUI / hosted | an agent, but tools only | | **Artifact** | editable, diffable `plan.json` IR — per-segment re-render | code (re-run to render) | black-box timeline / hosted project | none | | **Scope** | compose + edit + generate + **AUTO** end-to-end | composition (programmatic) | mostly edit *or* generate, per product | whatever the tool exposes | | **Guidance** | ships "what makes a good video" as skills + a deterministic delivery guard | none — you decide | product-opinionated | none | | **Keys / hosting** | zero-key trunk; **BYO** keys for generation; local-first, self-host | your own infra | vendor keys + hosted, lock-in | varies | | **License** | MIT | may require a company license for teams | proprietary | varies | Where it fits: reach for a **framework** when you want to hand-code every frame; a **SaaS editor** when a GUI and hosting are the point; **Cutforge Studio** when you want your *agent* to make the video, keep the result as an auditable file you can edit and re-render, and stay local + open with your own keys. --- ## Effects catalog for pro agent-driven editing Pro adds seven agent-ergonomics primitives on top of upstream's effects catalog (135+ ffmpeg ops — color, stylize, camera, layout, transition, captions, audio), aimed at the specific friction points of driving edits from an agent loop rather than a human timeline: | MCP tool | What it solves | |---|---| | `effects_apply` | Every catalog op reachable by name in one generic call — no more hand-writing a bespoke tool per effect. | | `effects_pipeline` | Chain N ops (e.g. zoom → grade → grain → vignette) through one call instead of N sequential agent round-trips. | | `effects_keyframe` | Multi-point zoom ramps (`points: [{at_sec, zoom}, ...]`) — a real Ken-Burns curve, not just a single at/ramp/peak punch-in. | | `effects_blend` | Full-frame blend-mode compositing (normal/multiply/screen/overlay/darken/lighten/addition/difference/exclusion/softlight/hardlight). | | `effects_mask` | Composite an overlay onto a base only where a third luma-mask clip exceeds a threshold — selective, rotoscope-style effect regions. | | `effects_kinetic_text` | Keyframed `drawtext` (x/y/opacity/scale over time) for agent-authored moving/fading titles, beyond the 4 fixed caption styles. | | `edit_beat_sync` | Onset/transient-detection heuristic — candidate cut/SFX timestamps instead of an agent hand-computing them from probed clip durations. | The `plan.json` IR also gained an `overlays` track (`tracks.overlays: [{path, start_sec, end_sec, z, blend_mode, opacity}]`) for arbitrary N-layer compositing on top of the primary edit, beyond the fixed narration/music/captions tracks. --- ## Packages | Package | What | |---|---| | `@cutforge/studio-core` | the `plan.json` IR (schema + validator + delivery guard), decision layer, runtime/config | | `@cutforge/studio-tools` | capability backends (render / edit / analyze / speech / image / video) | | `@cutforge/studio` | the `cutforge` CLI | | `@cutforge/studio-mcp` | MCP server (mirrors the CLI 1:1) | | `@cutforge/studio-skills` | the host-neutral `SKILL.md` knowledge pack | The technical plan and roadmap live in [`PLAN.md`](./PLAN.md). ## Development ```bash pnpm install pnpm build # tsc per package (core → tools → cli/mcp) pnpm test # vitest pnpm typecheck ``` ## License MIT — see [`LICENSE`](./LICENSE). Rendering uses [HyperFrames](https://github.com/heygen-com/hyperframes) at runtime via `npx`; editing/transcription use system `ffmpeg` and `whisper.cpp`. See [`PLAN.md`](./PLAN.md) for how third-party runtimes are located and the licensing notes.