palmier-pro-win

# palmier-pro-win A **headless, agent-driven video editor for Windows** — a Rust port of [Palmier Pro](https://github.com/palmier-io/palmier-pro)'s MCP agent layer. It exposes the same 45-tool MCP contract as the macOS original, but runs natively on Windows backed by **FFmpeg** (media decode/encode/filter), **ComfyUI** (generative models), and **whisper.cpp** (transcription) instead of AVFoundation / Metal / Core Image. Point Claude Code (or any MCP client) at it and edit video by talking: "create a project, import this clip, cut the dead air, add captions, grade it, export an MP4," or "generate a 5-second drone shot of the coast and place it on the timeline." > **This README is written for Claude Code sessions.** Read it before driving the server. Sections [How to drive it](#how-to-drive-it-a-claude-code-session-playbook) and [Example workflows](#example-workflows) are the operating manual. --- ## What this is — and is not **Is:** a Windows binary `palmier-pro-win.exe` that runs a Model Context Protocol server on `127.0.0.1:19789/mcp`. It operates on real `.palmier` project packages on disk (the same format the macOS app reads/writes). It is **API-compatible** with Palmier Pro's MCP contract — the 45 tool names and their input schemas match, so any agent prompt that works on macOS works here. **Is not:** a GUI timeline. There is no preview window, no real-time Metal render, no click-to-drag. Everything happens through tool calls. Color/effects are **FFmpeg-filter approximations** of Palmier's 11 Metal shaders — faithful in intent, not pixel-identical. The closed-source generative backend is replaced by **your own local ComfyUI**. **Scope-honest stubs:** a few tools depend on capabilities not yet ported and return **actionable errors** rather than fake results: - `sync_clips` — audio cross-correlation alignment (structure in place, correlation computation TODO). - `remove_silence` — on-device speech/VAD detection (workaround: `get_transcript` + `remove_words`, or `ffmpeg silencedetect`). - `detect_beats` — beat/downbeat tracking (workaround: external `aubio`/`librosa`, or `ffmpeg ebur128`). When a tool returns one of these errors, follow the workaround in the message — do not retry in a loop, do not fabricate the result. --- ## What I use it for This is a production tool for **video editing, AI video generation, and professional SaaS demo / launch videos**. Typical work: - **SaaS demo videos** — screen recordings + narration, cut to length, captions burned in, brand colors graded, exported as H.264 for web. - **Launch / announcement videos** — b-roll + generated clips (via ComfyUI) composited with text overlays, beat-synced cuts, picture-in-picture product shots. - **AI video generation** — prompt-to-clip via ComfyUI (`generate_video`/`generate_image`/`generate_audio`), then assemble, trim, grade, and export without leaving the agent conversation. - **Transcript-based editing** — transcribe, remove filler words / dead air, tighten pacing, re-export. - **Multicam / multi-source** — sync multiple cameras + external audio, switch angles across the timeline. Treat every edit as **non-destructive and undoable** — the agent records an undo stack per session, so a bad cut is one `undo` away. --- ## Quick start ### 1. Build **Always build through the wrapper script** (it loads MSVC vcvars64 and disables a broken schannel revocation check — plain `cargo` will fail): ```powershell C:\Users\Victor\cargo-msvc.cmd build --release ``` Output: `target\release\palmier-pro-win.exe` (~13 MiB). > **Stop the server before rebuilding** — the exe is file-locked while running, and `cargo build` will fail with `Access is denied (os error 5)`. Kill it first: `taskkill /F /IM palmier-pro-win.exe`. ### 2. Run ```powershell .\target\release\palmier-pro-win.exe ``` Log line on startup: ``` palmier-pro-win MCP server listening on http://127.0.0.1:19789/mcp ``` It runs in the foreground (Ctrl-C to stop). To run detached in the background from a Unix-style shell: ```bash (./target/release/palmier-pro-win.exe > server.log 2>&1 &) ``` ### 3. Connect Claude Code ```powershell claude mcp add --transport http palmier-pro http://127.0.0.1:19789/mcp ``` Verify: `claude mcp list` should show `palmier-pro` with 45 tools. In a Claude Code session, the tools appear as `mcp__palmier-pro__<tool_name>` and the server's `instructions` text is loaded as context. --- ## Configuration & backends All configuration is via environment variables (set with `setx VAR "value"` for persistence, or inline in the shell that launches the server). | Variable | Purpose | Default | |---|---|---| | `PALMIER_FFMPEG_BIN` | Path to `ffmpeg.exe`. | `ffmpeg` (must be on PATH — FFmpeg 8.x is already installed) | | `PALMIER_FFPROBE_BIN` | Path to `ffprobe.exe`. | `ffprobe` | | `PALMIER_WHISPER_BIN` | Path to `whisper-cli.exe` (whisper.cpp). **Unset → transcription tools return an actionable error.** | unset | | `PALMIER_WHISPER_MODEL` | Path to a ggml `.bin` model for whisper.cpp. | unset | **ComfyUI** is not configured by env var — run it separately on `http://127.0.0.1:8188` (the default the server expects). The server is an HTTP client of ComfyUI's `/prompt` + `/history` endpoints. Without ComfyUI running, `generate_*` tools fail; `list_models` still returns the catalog. **Projects** are stored as `.palmier` package directories under `C:\Users\Victor\Documents\Palmier Pro\`. A package contains: - `project.json` — `ProjectFile` (timelines, tracks, clips, settings) - `media.json` — `MediaManifest` (asset index) - `generation-log.json` — generation history - `media/` — imported/generated asset files --- ## The desktop app (GUI launcher + timeline viewer) For a double-clickable app "like the real Palmier", build the **Tauri 2** desktop app in `gui/`. It runs the **same** MCP server in-process, plus a control surface and a read-only timeline viewer. Editing itself still happens through Claude Code driving the MCP tools — the GUI is a launcher + viewer, not an editor. **Build:** ```powershell C:\Users\Victor\cargo-msvc.cmd build --release -p palmier-gui ``` Output: `target\release\palmier-gui.exe`. Double-click to open (needs the WebView2 Runtime — already present on Windows 11). **What the window gives you:** - **Server** — Start / Stop the in-process MCP server, a live status pill, and a one-click "Copy connect command" button that copies `claude mcp add --transport http palmier-pro http://127.0.0.1:19789/mcp` to the clipboard for pasting into Claude Code. - **Projects** — browse / create / open `.palmier` projects. "Create" picks a name, aspect ratio (16:9 / 9:16 / 1:1 / 4:3 / 2.4:1 / 9:14) and quality (720p / 1080p / 2K / 4K), and writes a real package identical to what `manage_project action='create'` produces. - **Backend** — set ffmpeg / ffprobe / whisper.cpp / whisper model / ComfyUI URL paths. Saved to `settings.json` next to the project registry and pushed into the server's env on start (restart the server to apply). This is the GUI alternative to `setx PALMIER_*`. - **Timeline viewer (read-only)** — renders the active timeline as SVG: a ruler with timecode, one lane per track, clip blocks colored by type (video / audio / image / text / lottie / sequence), caption markers, and click-any-clip → inspector (frames, duration, media ref, opacity, speed, volume, link/caption group, text content). **Layout:** ``` gui/ src-tauri/ # Rust backend (Tauri 2): src/commands.rs (IPC), src/state.rs (server token) tauri.conf.json # window, withGlobalTauri, frontendDist capabilities/ # core:default IPC permission icons/ # generated from app-icon.png via `cargo tauri icon` frontend/ # vanilla HTML/CSS/JS (no bundler); uses window.__TAURI__.core.invoke index.html style.css main.js ``` The root `Cargo.toml` is a workspace: `members = ["gui/src-tauri"]` plus the `palmier-pro-win` lib/bin. The GUI crate depends on the lib via a path dep and reuses `palmier_pro_win::ops` (project / timeline / settings) and `palmier_pro_win::mcp::serve` (the in-process server bootstrap — the same `serve()` the headless binary calls). Regenerate the icon set after editing `gui/src-tauri/app-icon.png`: ```powershell C:\Users\Victor\.cargo\bin\cargo-tauri.exe icon gui\src-tauri\app-icon.png -o gui\src-tauri\icons ``` For a proper Windows **installer** (NSIS) in addition to the raw exe, run `cargo tauri build` instead — slower, but it produces an installer under `gui\src-tauri\target\release\bundle\nsis\`: ```powershell cd gui\src-tauri ; C:\Users\Victor\cargo-msvc.cmd tauri build ``` --- ## The 45-tool surface Grouped by file in `src/tools/`. Every tool is async, takes a JSON `arguments` object, and returns `ToolResult` content (`text`, sometimes `image`). All mutations are one undoable action unless noted. ### Projects & timeline | Tool | What it does | |---|---| | `manage_project` | `list` / `open` / `create` / `close` projects for this session. **Always start a session with `action:"list"` or `create`.** | | `get_timeline` | Read project settings + tracks + clips. **Call at the start of every session** (the server instructions mandate this). Supports frame windowing + `captionDetail`. | | `inspect_timeline` | Render composited frames (what the user actually sees) — verify edits landed. Returns images with frame numbers burned in. | | `create_timeline` | Create (or duplicate with `from`) a timeline and switch to it. Versioning primitive. | | `set_active_timeline` | Switch the active timeline. Re-read `get_timeline` after — ids from the old timeline are invalid. | | `set_project_settings` | Change fps / resolution / aspect ratio. Existing clips auto-refit and rescale. | ### Media library | Tool | What it does | |---|---| | `get_media` | Library inventory — assets, folders, timelines. **Every `mediaRef` in other tools comes from here.** | | `inspect_media` | Look at one asset: sampled frames, transcription, EXIF, dimensions. | | `search_media` | Semantic visual + spoken search across the library. | | `import_media` | Import from `url` / `path` / `bytes` / `matte`. URL + path import run in the background — poll `get_media` until `generationStatus` clears. | | `organize_media` | Create folders, move, rename, delete — one undoable action. | ### Clips & tracks (the editing core) | Tool | What it does | |---|---| | `add_clips` | Place assets on tracks. Overwrite on overlap. Auto-creates linked audio for video-with-audio. `mediaRef` may be a `timelineId` (nested sequence). | | `insert_clips` | Splice in without overwriting — ripples everything at/after `atFrame` right. | | `move_clips` | Move clips to a new track and/or frame. Linked partners follow. | | `remove_clips` | Remove by id; linked group follows (linked-delete). | | `manage_tracks` | Reorder / set (mute/hide/syncLock) / remove tracks. Index 0 = top. | | `split_clips` | Cut at frame(s) — boundary only, no trim/shift. Linked A/V split together. | | `ripple_delete_ranges` | Cut ranges + close gaps in one action. The fast path for filler/dead-air removal. | | `set_clip_properties` | duration / trim / speed / volume / opacity / transform / blendMode. Setting volume/opacity clears keyframes on that property. | | `set_keyframes` | Animate one property (volume/opacity/rotation/position/scale/crop). Frames are clip-relative. | | `apply_layout` | Multi-clip layouts (split screen, PIP, grid) — computes transforms + crop so each clip fills its slot. | | `sync_clips` | Align clips to a reference. **Honest stub** — returns an actionable error (audio cross-correlation not ported). | ### Text & captions | Tool | What it does | |---|---| | `add_texts` | Add text clips as timeline layers. | | `update_text` | Edit text content / typography / color / animation. Accepts a `captionGroupId`. | | `add_captions` | Transcribe the timeline's audio and create styled caption clips automatically. | ### Transcription & audio | Tool | What it does | |---|---| | `get_transcript` | Spoken transcript of the **current timeline** in project frames — walks every clip, maps words through trim/speed/position. Use before `remove_words`. | | `remove_words` | Cut by word index (Descript-style). Resolve `get_transcript` indices or `matches` (filler tokens), remove surrounding pause, close gaps. | | `remove_silence` | **Honest stub** — on-device VAD not ported. Workaround in the error message. | | `detect_beats` | **Honest stub** — beat tracking not ported. Workaround in the error message. | | `denoise_audio` | Real ffmpeg `afftdn` bake — denoises clip audio, writes a new asset, swaps the clip's `mediaRef`. | ### Color & effects | Tool | What it does | |---|---| | `apply_color` | Color grade with named knobs (exposure/contrast/saturation/wheels/curves/hue curves/LUT). Merges onto current grade; `reset:true` for neutral. | | `apply_effect` | Apply looks/FX (the 20 effect descriptors in `media_ops/filters.rs`). | | `inspect_color` | Read a clip's grade + reference histogram for iterative grading. | ### Export | Tool | What it does | |---|---| | `export_project` | Queue an export. `mode`: `video` (H.264/H.265/ProRes), `xml` (Premiere), `fcpxml` (Resolve/FCP), `palmier` (package). Returns a `jobId`. | | `manage_exports` | `list` (progress/warnings/results) or `cancel` by `jobId`. | ### Generation (ComfyUI) | Tool | What it does | |---|---| | `list_models` | Available video/image/audio generation models (ComfyUI workflow catalog). | | `generate_video` | txt2vid via ComfyUI → asset lands in the library. | | `generate_image` | txt2img via ComfyUI. | | `generate_audio` | text2audio via ComfyUI (e.g. StableAudio custom node). | | `upscale_media` | Upscale an asset via ComfyUI. | ### Multicam | Tool | What it does | |---|---| | `manage_multicam` | `create` (sync sources into a group) / `ungroup`. | | `change_cam` | Switch a group's camera angle over frame ranges. | | `get_multicam` | Read a group: members, program cut as run-length rows, track indexes. | ### Meta | Tool | What it does | |---|---| | `undo` | Revert the assistant's most recent timeline edit (this session only). | | `send_feedback` | Send feedback (throttled). | ### MCP resources - `palmier://models/video` — JSON catalog of video generation models. - `palmier://models/image` — JSON catalog of image generation models. --- ## How to drive it: a Claude Code session playbook The server's `instructions` field (ported verbatim from Palmier's `AgentInstructions.swift`) is loaded into your context when the MCP server connects. **Follow it.** The condensed operating rules: 1. **Open a project first.** `manage_project` with `action:"list"` (to resume) or `action:"create"` (new). Nothing else works until a project is open. 2. **`get_timeline` at the start of every session**, and again after any structural change (`set_active_timeline`, `create_timeline`, `undo`, big edits). Clip/track ids from a prior read may be stale after a change. 3. **`get_media` before referencing any asset.** Every `mediaRef` comes from the ids it returns. 4. **Read before you edit.** Use `inspect_timeline` (composited frames) or `inspect_media` (raw asset) to verify edits landed — don't assume. 5. **One undoable action per logical step.** Mutations (`add_clips`, `split_clips`, `remove_words`, `apply_color`, …) each push one undo entry. If an edit overshoots, `undo` and retry with corrected params. 6. **Transcript-driven cutting:** `get_transcript` → read as prose → `remove_words` with the indices → re-read `get_transcript` (indices shift after every cut). Drop to `ripple_delete_ranges` only for non-word-aligned spans. 7. **Verify cuts against the word list, not a summary.** After `remove_words`, the transcript reflects what's actually audible. 8. **Export is async.** `export_project` returns a `jobId`; poll `manage_exports` `action:"list"`. Don't infer a job is stuck from elapsed time alone. 9. **Honest stubs:** when `sync_clips`/`remove_silence`/`detect_beats` return an actionable error, follow the workaround in the message. Don't loop, don't fabricate. 10. **Generation needs ComfyUI running** on `127.0.0.1:8188` with the model workflows you want. Generated assets land in the library with a `generationStatus` you poll via `get_media` `ids:[mediaRef]`. ### Session state is per-session Each MCP session has its own `AppState` (open project) and `SessionState` (undo stack, transcript context). `undo` only reverts edits **this session's agent** made — it never touches the user's manual edits. Don't assume state carries between sessions. --- ## Example workflows ### A. SaaS demo video (screen recording + narration) ``` 1. manage_project {action:"create", name:"Acme Demo", fps:30, aspectRatio:"16:9", quality:"1080p"} 2. import_media {source:{path:"C:\\recordings\\demo.mp4"}, name:"Screen recording"} 3. import_media {source:{path:"C:\\recordings\\voiceover.wav"}, name:"VO"} → poll get_media {ids:[...]} until both generationStatus clear 4. get_timeline → note trackIds, fps 5. add_clips {entries:[{mediaRef:"<video>", startFrame:0}]} → auto-creates video+audio tracks 6. add_clips {entries:[{mediaRef:"<vo>", trackIndex:<audio track>, startFrame:0}]} 7. get_transcript {clipId:"<vo clip>"} → read words 8. remove_words {matches:["um","uh","hmm"], cutAggressiveness:"balanced"} 9. add_captions {language:"en", color:"#FFFFFF", backgroundColor:"#000000", animation:"highlight"} 10. apply_color {clipIds:[<video>], exposure:0.1, contrast:1.05, saturation:1.1} 11. export_project {mode:"video", codec:"H.264", outputPath:"C:\\out\\acme-demo.mp4"} 12. manage_exports {action:"list"} → poll until status:"completed" ``` ### B. Launch video with generated b-roll ``` 1. manage_project {action:"create", name:"Launch v1", fps:30, aspectRatio:"16:9", quality:"4K"} 2. list_models {type:"video"} → pick a model id 3. generate_video {model:"comfyui-txt2vid", prompt:"slow drone shot over a coastal highway at sunrise, cinematic", duration:5, aspectRatio:"16:9"} → poll get_media until ready 4. import_media {source:{path:"C:\\assets\\logo-sting.mp4"}} 5. add_clips {entries:[{mediaRef:"<generated>", startFrame:0}, {mediaRef:"<logo>", startFrame:150}]} 6. apply_layout {layout:"pip_bottom_right", slots:[{slot:"main", mediaRef:"<generated>"},{slot:"inset", mediaRef:"<logo>"}], startFrame:150, endFrame:300} 7. add_texts {entries:[{startFrame:0, endFrame:90, content:"Introducing\nAcme", fontSize:120, color:"#FFFFFF", alignment:"center"}]} 8. export_project {mode:"video", codec:"H.265", outputPath:"C:\\out\\launch.mp4"} ``` ### C. Transcript-based tighten + re-export ``` 1. manage_project {action:"open", name:"Podcast Ep 4"} 2. get_timeline 3. get_transcript {granularity:"segments"} → skim for the section to cut 4. get_transcript {clipId:"<clip>", startFrame:<...>, endFrame:<...>} → word indices 5. remove_words {words:[[12,18],[44,44]], cutAggressiveness:"tight"} 6. get_transcript → verify the cut reads cleanly 7. export_project {mode:"palmier"} → writes a self-contained .palmier package ``` --- ## Project layout ``` palmier-pro-win/ Cargo.toml src/ main.rs # thin binary: bind 127.0.0.1:19789, serve lib.rs # pub mod declarations (lib+bin so tests can name the crate) state.rs # AppState, SessionState, Config, ExportQueue editor.rs # in-memory Editor: timelines, media manifest, snapshot/apply/save mcp/ mod.rs # INSTRUCTIONS (ported AgentInstructions text) server.rs # PalmierServer (ServerHandler): tools, resources, per-session state tools.rs # 45 Tool definitions + input schemas (ported from ToolDefinitions.swift) dispatch.rs # execute(name, args) → 45-arm match args.rs # lenient int/double/bool/string coercion + unknown-key validation result.rs # ToolResult { content, is_error } model/ # serde structs mirroring Swift models verbatim (camelCase, defaults) project_file.rs timeline.rs clip_type.rs keyframe.rs effect.rs text.rs media.rs multicam.rs speakers.rs project/ package.rs # read/write .palmier dir registry.rs # ~/Documents/Palmier Pro/project-registry.json resolver.rs # mediaRef → file path media_ops/ probe.rs # ffprobe → MediaManifestEntry transcode.rs # ffmpeg encode (H.264/H.265/ProRes) + filtergraph filters.rs # 20 effect descriptors + color filter builders generative/comfyui.rs # ComfyUI HTTP client (txt2vid/txt2img/...) audio/whisper.rs # whisper.cpp shell + JSON parse → WordSegment tools/ # one file per tool group (mirrors ToolExecutor+*.swift) projects.rs timeline.rs media.rs clips.rs texts.rs transcription.rs color.rs export.rs generate.rs multicam.rs meta.rs tests/roundtrip.rs # 6 serde round-trip tests against .palmier packages ``` The original macOS Swift source (read-only reference) is at `C:\Users\Victor\palmier-pro`. Each Rust tool file mirrors the corresponding `Sources/PalmierPro/Agent/Tools/ToolExecutor+*.swift`. --- ## Build gotchas - **Always use `C:\Users\Victor\cargo-msvc.cmd`.** It loads MSVC `vcvars64` (link.exe isn't on PATH) and sets `CARGO_HTTP_CHECK_REVOKE=false` (schannel TLS revocation check fails on this network without it). Plain `cargo` fails with `CRYPT_E_NO_REVOCATION_CHECK` + link.exe-not-found. - **Stop the server before rebuilding.** `taskkill /F /IM palmier-pro-win.exe` — the running exe is file-locked. - **`#![recursion_limit = "512"]`** is set in `src/lib.rs` — the large `json!` tool schemas in `mcp/tools.rs` exceed the default 128 macro-recursion limit. Don't remove it. - **`gen` is a reserved keyword in Rust 2024** — the generation module is `src/generative/`, declared `pub mod generative;`. - **`rmcp::Tool::new`** requires `Into<Cow<'static, str>>` for name/desc — pass owned `String`, not borrowed `&str`. - Rust edition 2024, target `x86_64-pc-windows-msvc`. MSVC Build Tools 2026 (VS 18). --- ## Verification - `cargo test` → 6/6 round-trip tests pass (data model + package I/O byte-fidelity). - Live MCP handshake → 45 tools + 2 resources; `manage_project create` writes a real `.palmier` package; `get_timeline` reads it back. - To run the test suite: `C:\Users\Victor\cargo-msvc.cmd test`. --- ## Porting notes for future work - **Parity check:** run the same agent sequence against real Palmier Pro on a Mac and diff `claude mcp list` JSON — the 45 tool names + param shapes must match. - **Color shaders:** the 11 Metal shaders are FFmpeg-filter approximations. `HueCurves` / `LUTTetra` are the least faithful; revisit if a grade needs pixel-match. - **Short-id resolution** (`ToolExecutor+ShortId.swift`) is not yet ported — tools work with full ids; prefix resolution is a Phase 2 polish item. - **`sync_clips` / `remove_silence` / `detect_beats`** need an audio-analysis dependency (cross-correlation, VAD, beat tracking). The tool scaffolding and arg validation are in place; only the core computation is stubbed. - **ComfyUI workflows** are currently a default txt2vid/txt2img. The real catalog should come from the user's installed ComfyUI workflows (`list_models` TODO). See the plan file at `C:\Users\Victor\.claude\plans\starry-wobbling-sunrise.md` for the full 6-phase design.