Content
# Frameloop
MCP server that closes the feedback loop between Claude and [Remotion](https://remotion.dev). Claude can see reference videos, write Remotion code, render preview frames, and self-correct — all without leaving the conversation.
Also includes a CLI and web UI for standalone video frame extraction.
## The Problem
Claude can generate Remotion code, but it can't see its own output. It writes animations blind. When something doesn't look right, you have to manually screenshot, paste, describe the issue, and iterate.
## The Solution
Frameloop gives Claude eyes. The full workflow:
1. **Claude sees a reference video** — extracts key frames with scene detection
2. **Claude writes Remotion code** — based on what it saw
3. **Claude renders its own frames** — using `render_still` / `render_preview`
4. **Claude views the result** — reads the rendered images
5. **Claude self-corrects** — adjusts code if the output doesn't match
6. **Repeat** until it looks right
## Killer Feature: Replicate Any Video
Frameloop's flagship workflow: **point Claude at any video and get a working Remotion recreation.**
```
Reference video (file or YouTube URL)
│
▼
┌─────────────────────┐
│ Frameloop extracts │ Scene detection + interval sampling.
│ dense frame batches│ 180 frames per batch in 3x3 grids.
└─────────┬───────────┘
▼
┌─────────────────────┐
│ Claude analyzes each│ Describes elements, colors, timing,
│ batch frame-by-frame│ easing curves, transitions per scene.
└─────────┬───────────┘
▼
┌─────────────────────┐
│ Claude writes │ Uses Frameloop's recipe library,
│ Remotion code │ motion presets, and IR codegen.
└─────────┬───────────┘
▼
┌─────────────────────┐
│ render_preview │ Generates contact sheet of own output.
│ vs reference │ Compares with original via compare_frames.
└─────────┬───────────┘
▼
┌─────────────────────┐
│ Self-correct loop │ Adjusts timing, easing, layout until
│ until match │ visual match achieved.
└─────────────────────┘
```
**Two input modes:**
- **Local file** — drop any `MP4 / MOV / WebM / AVI` into the Studio Extract panel
- **YouTube URL** — paste a `youtube.com`, `youtu.be`, or `youtube.com/shorts` link. Frameloop streams the video directly via `yt-dlp` (no full download), extracts frames, and feeds them to Claude. Works with regular videos, shorts, and motion graphics.
**Two extraction strategies:**
| Mode | Output | Use case |
|------|--------|----------|
| **Extract Frames** | 15-60 key frames (scene detection + interval) | Visual inspection, single-shot analysis, ask Claude about content |
| **Replicate Motion** | Dense batches of 180 frames in 3x3 grids @ 6fps | Reproducing animations — Claude reads frame-by-frame timing |
The Replicate mode is what makes Frameloop unique. Standard frame extraction tools give you 10-20 frames; Frameloop gives Claude enough density to actually reconstruct the motion curves.
**Example prompt to Claude after Replicate:**
> "Here are the batches from a Stripe ad. Build a Remotion composition that recreates the title reveal and the dashboard slide-in. Use @recipe:fade-up-word-by-word for the headline and apply @preset:ease-out-expo for all transitions. Render a preview when done so I can compare."
Claude will:
1. Use `analyze_video` (already done — frames in chat)
2. Generate Remotion code with the recipe + preset
3. Call `render_preview` to see its own output
4. Use `compare_frames` to put original vs render side-by-side
5. Iterate until match
## MCP Tools
| Tool | Description |
|------|-------------|
| `render_still` | Render a single frame from a Remotion composition |
| `render_frames` | Render multiple specific frames to check animation progression |
| `render_preview` | Generate a contact sheet grid of the entire composition |
| `get_compositions` | List all compositions in a Remotion project |
| `analyze_video` | Extract key frames from a reference video (scene detection, interval, or combined) |
| `extract_frames_at_timestamps` | Extract frames at specific timestamps from a video |
## Requirements
- Python 3.10+
- [FFmpeg](https://ffmpeg.org/download.html) in your PATH
- [Node.js](https://nodejs.org/) 18+ (for Remotion rendering tools)
## Installation
```bash
pip install frameloop
```
Or from source:
```bash
git clone https://github.com/Alan-Lucena/frameloop.git
cd frameloop
pip install -e ".[web,dev]"
```
## Setup with Claude Code
Add Frameloop as an MCP server in your Claude Code settings (`~/.claude/settings.json`):
```json
{
"mcpServers": {
"frameloop": {
"command": "frameloop",
"args": ["mcp"]
}
}
}
```
Or in a project-level `.claude/settings.json`:
```json
{
"mcpServers": {
"frameloop": {
"command": "frameloop",
"args": ["mcp"],
"env": {
"FRAMELOOP_OUTPUT_DIR": "./frameloop_output"
}
}
}
}
```
After configuring, restart Claude Code. The tools will be available automatically.
## Example Workflow
```
You: Here's a reference video at ./reference.mp4. Analyze it and recreate it in Remotion.
Claude: [uses analyze_video to extract key frames from reference.mp4]
[views the extracted frames to understand the design]
[writes Remotion code for the composition]
[uses render_preview to see what its code produces]
[views the contact sheet and compares with reference]
[adjusts timing, colors, and layout]
[uses render_frames to check specific animation moments]
[makes final tweaks]
Done! The composition matches the reference.
```
## CLI Usage
The CLI preserves all the original frame extraction features:
### Extract key frames
```bash
# Default: combined method, 15 frames, saved as JPEGs
frameloop extract video.mp4
# Scene-detection only
frameloop extract video.mp4 --method scene --max-frames 8
# Generate a grid image
frameloop extract video.mp4 --format grid
# Detailed grid with rich metadata
frameloop extract video.mp4 --format grid --detailed --max-frames 40
# From YouTube
frameloop extract "https://www.youtube.com/watch?v=..." --format grid
```
### Ask Claude about a video
```bash
export ANTHROPIC_API_KEY=sk-ant-...
frameloop ask video.mp4 "Describe what happens in this video step by step"
frameloop ask "https://youtu.be/..." "Summarize this video"
```
### Replicate mode (high-density extraction for Remotion)
```bash
frameloop replicate video.mp4
frameloop replicate "https://youtu.be/..." --fps 5 --output-dir ./my_output
```
### Web UI
```bash
# Terminal 1: API server
frameloop serve
# Terminal 2: Frontend
cd web && npm install && npm run dev
```
Open [http://localhost:3000](http://localhost:3000) — drag-and-drop videos, tweak extraction settings, preview frames.
### Start the MCP server manually
```bash
frameloop mcp
```
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `FRAMELOOP_OUTPUT_DIR` | `./frameloop_output` | Where rendered frames and grids are saved |
| `ANTHROPIC_API_KEY` | — | Required for the `ask` CLI command |
## Use as a Library
```python
from frameloop.extractor import extract_frames, ExtractionMethod, compose_grids
# Extract frames
result = extract_frames("video.mp4", method=ExtractionMethod.COMBINED, max_frames=10)
for frame in result.frames:
print(f"Frame {frame.index}: {frame.timestamp:.1f}s — {frame.reason}")
# Generate a grid
grids = compose_grids(result, grid_layout="full", detailed=True)
grids[0].save("grid.jpg", "JPEG", quality=92)
```
## Project Structure
```
frameloop/
├── frameloop/
│ ├── __init__.py # Package version
│ ├── mcp_server.py # MCP server with 6 tools
│ ├── extractor.py # Core frame extraction engine
│ ├── downloader.py # YouTube stream resolution (yt-dlp)
│ ├── claude_client.py # Anthropic API integration
│ ├── cli.py # CLI (extract, ask, replicate, serve, mcp)
│ └── server.py # FastAPI backend for web UI
├── web/ # Next.js frontend
├── pyproject.toml
├── LICENSE
└── README.md
```
## Development
```bash
git clone https://github.com/Alan-Lucena/frameloop.git
cd frameloop
python -m venv .venv && source .venv/bin/activate
pip install -e ".[web,dev]"
ruff check frameloop/
pytest
```
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md).
## License
[MIT](LICENSE)
