Content
# flowcard
Motion understanding for LLM agents. Compresses a video into **one annotated,
readable image** (a "flowcard") plus exact machine-readable stats, so an
image-strong / video-blind model like Claude can debug motion: game glitches,
render stalls, teleports, z-fighting shimmer, animation timing.
The approach is pure Eadweard Muybridge (1878): collapse motion into still
images you can reason about.
## Why
Claude has no native video input, and even video-native LLMs are provably weak
at motion — most score near chance on "did it move left or right"
("directional motion blindness", [arXiv:2605.22823](https://arxiv.org/abs/2605.22823);
see also [TempCompass](https://arxiv.org/abs/2403.00476)). flowcard sidesteps
the problem: deterministic CV renders motion *spatially*, and the agent reads
the result with its strong image skills. No model inside the tool — free,
local, reproducible.
## What it produces
One card per clip. An edited TV clip — the findings flag the shot cuts and the
timeline shows the editing rhythm; the filmstrip anchors what each shot contains:
A single-take night aerial — no cut events, and the timeline instead tells the
camera story: fast travel for ~5s (magenta hump), then a slow glide. Note the
parallax caveat: under 3D camera travel, PATHS/FLICKER fill with scenery
fragments, while the camera stats and timeline stay trustworthy:
## Tools (MCP)
Register once (user scope, available in every project):
```
claude mcp add --scope user flowcard -- D:\flowcard\venv\Scripts\python.exe D:\flowcard\server.py
```
- **`record(target, seconds, fps)`** — capture the desktop (ddagrab/gdigrab)
or one window (`window:<exact title>`) and auto-analyze. Returns `card_path`.
- **`analyze(video_path, t0, t1, roi, camera_comp)`** — any video → flowcard +
stats. Drill down with `t0/t1` (seconds) and `roi=[x,y,w,h]`. Clips > 18 s
are windowed automatically.
- **`compare(video_a, video_b)`** — visual regression diff of two same-start
runs: divergence onset time, worst-region bbox, A/B filmstrips.
CLI equivalents:
```
python -m flowcard clip.mp4 outdir [--t0 2] [--t1 8] [--roi x,y,w,h] [--no-camera-comp]
python tests\validate.py # regenerate ground-truth clips and check everything
```
## The card
Six panels + timeline + filmstrip, each labeled and self-describing:
| Panel | Question it answers |
|---|---|
| REFERENCE | what does the scene look like (median / stabilized median) |
| WHERE | which pixels changed, total motion energy |
| WHEN | time of last motion per pixel (blue=early → red=late) |
| DIRECTION | mean optical-flow direction (hue) + speed (brightness), camera-removed |
| FLICKER | % of frames each pixel changed — separates shimmer/z-fighting from movement |
| PATHS | tracked object trajectories, colored by time, with APPEAR markers |
Header findings use a fixed vocabulary an agent can act on: `CAMERA`, `STALL`,
`MICROSTUTTER`, `DISCONTINUITY`, `APPEARANCE` (teleport/spawn),
`FLICKER HOTSPOT`, `DOMINANT OBJECT MOTION`. `stats.json` carries the same
facts as numbers (all coordinates in **frame-0 space**; camera pan reported
separately, per-pair series included in the file but stripped from MCP
responses to keep context bounded).
## How v2 works
1. **Camera estimation** per frame pair: LK feature tracks + RANSAC
partial-affine. Moving objects are RANSAC outliers; the background wins.
2. **Stabilization**: frames are warped into frame-0 coordinates on a padded
canvas, so every panel shows *object* motion in world space; camera speed
becomes its own timeline curve + stats.
3. **Detectors** on the stabilized stack: motion energy, last-motion time,
DIS optical flow (Farneback fallback), duty-cycle flicker map + hotspot
boxes, blob tracking with edge-fragment agglomeration, spatial-novelty
appearance detection (activity where nothing moved for ≥0.2 s = teleport /
spawn), stall + microstutter pacing from raw duplicate frames.
## Validation
`tests\validate.py` generates synthetic clips with planted ground truth
(render freeze, teleport, shimmer patch, bouncing vs linear movers, sinusoidal
camera pan, a +25 px regression bug) and asserts every event is recovered —
21 checks, including camera pan trace RMSE < 0.5 px/frame and compare()
divergence onset within 0.2 s. All pass as of v2.
## Does it actually help a VLM? (bench)
`bench/run_bench.py` runs qwen3-vl-32b-instruct (OpenRouter, pydantic-ai) on
two unknown ~11s 1080p clips — an edited TV clip and a single-take night city
aerial — under three conditions. Raw outputs in `bench/results/`. Qualitative,
n=1 per cell:
| Condition | Result |
|---|---|
| native video upload | **fails** — video input unsupported on the OpenAI-compatible path |
| 8 uniform frames | good content ID; found 1 of ~8 cuts; missed the two-phase camera profile; **invented technicals** ("tripod", "no zoom", pacing inferred from its own sampling interval, a prop "placed during the performance") |
| flowcard + stats + `grab_frames` tool | every flagged cut with exact timestamps; single-take correctly confirmed *from instruments*; technicals quoted from measurements (effective fps, dup frames, camera %, pan/zoom); correctly discounted parallax tracks per the caveats |
Takeaways:
1. The card moves a VLM from *describing sampled pictures* to *citing
measurements* — the biggest gains are in event structure (cut lists with
times) and technical claims, where the frames-only baseline hallucinates.
2. The baseline's failure mode is exactly the one this tool targets: it reasons
from its own sampling grid ("slow pacing", "camera static") because it never
sees between the frames.
3. Honest miss: uniform sampling caught an on-screen name card that event-driven
frame-grabbing skipped — so the recommended recipe (now in
`bench/card_system_prompt.md`) is card + event frames + a uniform spread.
The instruments and blanket coverage are complements, not substitutes.
## Known limitations
- Stabilization resampling leaves a faint background echo in WHERE under
camera motion (objects still dominate; stats unaffected).
- Detectable motion is bounded by capture fps; record at ≥ the app's rate.
- Full-screen scene cuts read as global discontinuities, not object events.
- Camera model is partial-affine (pan/rotate/zoom) — strong parallax or FPS
camera rotation in a 3D world violates it; panels degrade gracefully but
world coordinates become approximate.
## Roadmap (v3)
- RAFT optical flow + CoTracker point trails (torch, GTX 1080) for
small/fast-object fidelity
- SAM2 object masks → per-object motion cards
- HUD OCR for correlating motion events with on-screen state
- Temporal alignment (DTW) in compare() for runs that don't start together
