mixamo-llm-mocap

# Mixamo LLM Mocap **Turn any locked-camera video — filmed or AI-generated — into a clean FK animation on any Mixamo character. One performer, or two fighting each other. No mocap suit, no manual keyframing, and every stage scriptable enough that an AI agent can run the whole loop.**      *Left: AI-generated source video. Right: the automatic retarget on a Mixamo character in Blender — 10 seconds, nine punches, a slip under and a high side kick, straight through the pipeline.*  *Two performers, one plate, one pass. The left fighter throws four punches and a roundhouse; the right one blocks, folds over the body shot and ducks under the kick. Both tracks are split out of the same video by screen side, retargeted onto two Mixamo characters with different proportions — Y Bot and Ninja — and placed at the distance the performers actually stood, measured from the footage.* ## How it works The default path transfers the estimator's SMPL-X **joint rotations** and runs without the Blender UI: ``` video plate (locked camera) │ ├─ 1. estimate_pose_gvhmr.py --smplx-only GVHMR → SMPL-X parameters, refined on the plate's 2D keypoints (+ heels/toes, + zoomed hands) ├─ 2. retarget_smplx.py rotations → your Mixamo rig; floor, foot locks, camera-true trajectory ├─ 3. bl_motion.py (blender -b) key the action in bulk and save it, or render — no MCP session ├─ 4. render_review.py 2×2 review video: plate | overlay on the plate / before | after, + hands & feet zoomed └─ 5. eval_fidelity.py limb and head angles vs the video's keypoints, foot slide, jitter ``` - **Twist is real.** Forearm roll, spine curve and head orientation come from SMPL-X instead of being re-derived from positions. - **Extremes survive.** GVHMR flattens fast limbs; a SMPLify-style fit on the plate's own ViTPose keypoints puts a head-high kick back at head height. - **Feet point where the performer's point.** Heel and toe keypoints (MediaPipe) join the fit, and the rig's foot is aligned on the foot's real axis — toe-out, heel raises and pivots come from the video. - **Feet stay down.** Contacts from GVHMR's detector, the floor, the image itself (a foot still on screen is planted) and physics (a foot carrying the body alone is planted, whatever the estimate says); planted feet lock flat or pivot on the point that carries the weight, and no foot goes under the floor. - **Fingers from the video.** GVHMR has no fingers: each hand is cropped around its wrist, zoomed, and read by a hand detector; fists, open hands and the thumb follow the performer. - **The trajectory is the camera's.** The plate's static camera is recovered; height drift is removed continuously and horizontal drift one footfall at a time, so the character ends on its mark. The original landmark path below is still available for existing specs and the two-character tooling: ``` video plate (locked camera, T-pose bookends) │ ├─ 1. estimate_pose_gvhmr.py GVHMR (SMPL-X mesh recovery) → 33 landmarks + pelvis height ├─ 2. analyze_landmarks.py numeric beat detection → you write a beat sheet from NUMBERS ├─ 3. action_specs/<name>.json the motion as data: support schedule, rest blends, fists ├─ 4. lift_to_mixamo.py direction-preserving retarget onto YOUR rig's proportions ├─ 5. apply_mixamo_fk.py FK aim + foot planting, inside live Blender (via Blender MCP) ├─ 6. qa_clip.py automated gate: no explosions, no pops, no foot skate ├─ 7. compare_reference.py frame-by-frame vs the video → which windows still differ ├─ 8. compare_pair.py two-character plates: separation, reach, intrusion ├─ 9. run_in_blender.py contact real mesh-vs-mesh collision between two characters └─ 10. render_preview.py preview + side-by-side showcase video ``` With two performers in the plate, stages 1–7 run once per fighter (`--person left|right` splits the tracks), `setup_duo.py` builds one scene holding both characters, and `compare_pair.py` checks what only exists when there are two of them: whether they stand, reach and miss each other the way the performers did. The estimator provides mesh-quality joints; the lift keeps its segment *directions* but rebuilds every position from your character's measured bone lengths; the apply plants feet by solving hip height (never IK — Mixamo rigs are FK-only); the spec contributes only what a video cannot know: which foot is the support in each phase (including `"none"` for airborne beats), when fists close, where the clip locks back to rest. ## Why it's different - **Any Mixamo character.** `setup_rig.py` builds a clean scene from your own Mixamo download and measures it into `rig_profile.json` (rest pose, bone lengths, hip and ground heights). Every stage reads that profile. - **Motions are data, not code.** A new motion is a small JSON spec — the `action_specs/` here (a kung-fu form, a combo with a jump, a fight combination, a 360° jumping spin kick and a two-fighter duel) are worked examples of the whole schema. - **Honest Mixamo FK.** Hips are the only translating bone, everything else is quaternions at 30 fps — clips drop into any Mixamo-style workflow without cleanup. - **Real ground contact.** Planted feet solve to ground height with zero skate (the support ankle is pinned through each stance); jumps integrate the estimator's real pelvis arc. - **A QA gate, not vibes.** Exploded bones, hip pops, foot skate, drifting roots and broken rest poses are caught numerically before a human ever looks. - **A closed refinement loop.** `compare_reference.py` measures the retarget against the source video frame by frame on what an eye actually reads — hand height relative to the face, distance between the hands, limbs inside the torso, gaze direction — and reports the exact frame windows that diverge. Notes like *"his hands are too high and his arm clips his back"* become numbers, and an over-correction gets caught before it ships instead of after. - **Two characters, one scene.** A two-performer plate is split into tracks by which side of frame each occupies — robust where tracker ids swap on contact — retargeted onto two different Mixamo characters with their own measured proportions, and placed at the distance the performers actually stood, recovered from the footage rather than eyeballed. `compare_pair.py` then verifies separation, strike reach and limb intrusion against the video, frame by frame, and a Blender BVH pass checks the actual skinned meshes for collision — because two Mixamo characters are thicker than two humans, and a choreography built out of 2 cm near-misses collides when you retarget it faithfully. Clearance is bought from the stage with a declared, measured offset, which the comparator keeps reporting so the cost stays visible. - **A review pass that is part of the loop.** Render the showcase, put source and retarget side by side at the same beat, name what looks wrong in one sentence, then measure it. When eye and numbers disagree it is usually the numbers — every false reading in this project came from a mismatched proxy (a nose against a skull-base joint, a capsule against a mesh). [docs/PIPELINE.md](docs/PIPELINE.md) section 10. - **Written for agents.** Beat decisions come from `analyze_landmarks.py` numbers (never from eyeballing frames), every stage is a CLI or a socket call, and `docs/PITFALLS.md` encodes every mistake so the next operator — human or AI — doesn't repeat them. ## Quickstart 1. **Install** — [docs/INSTALL.md](docs/INSTALL.md) walks through every dependency (list below). 2. **Build your rig scene**: ``` blender --background --python pipeline\setup_rig.py -- --fbx ybot.fbx --out ybot_rest.blend ``` 3. **Run a plate — SMPL-X path** (headless, no Blender window needed): ``` tools\GVHMR\.venv\Scripts\python.exe pipeline\estimate_pose_gvhmr.py --video plates\<name>\<name>.mp4 --out plates\<name>\landmarks.json --smplx-only python pipeline\retarget_smplx.py --spec action_specs\<name>.json blender -b ybot_rest.blend -P pipeline\bl_motion.py -- apply --motion clips\<clip>\motion.npz --save clips\<clip>\<clip>.blend python pipeline\render_review.py --spec action_specs\<name>.json --clip clips\<clip> python pipeline\eval_fidelity.py --spec action_specs\<name>.json --clip clips\<clip> ``` A spec needs only `name`, `action_name`, `clip_dir`, `landmarks` and the fps; `fists`, rest blends and `arm_overrides` are optional. Details: [docs/PIPELINE.md](docs/PIPELINE.md) section S. **Or the landmark path** (Blender open on the scene; plate rules in [docs/PROMPTING.md](docs/PROMPTING.md)): ``` tools\GVHMR\.venv\Scripts\python.exe pipeline\estimate_pose_gvhmr.py --video plates\<name>\<name>.mp4 --out plates\<name>\landmarks.json tools\GVHMR\.venv\Scripts\python.exe pipeline\analyze_landmarks.py --landmarks plates\<name>\landmarks.json # beat sheet → action_specs\<name>.json (schema: docs/PIPELINE.md) tools\GVHMR\.venv\Scripts\python.exe pipeline\lift_to_mixamo.py --spec action_specs\<name>.json python pipeline\run_in_blender.py all action_specs\<name>.json tools\GVHMR\.venv\Scripts\python.exe pipeline\qa_clip.py --spec action_specs\<name>.json tools\GVHMR\.venv\Scripts\python.exe pipeline\compare_reference.py --spec action_specs\<name>.json tools\GVHMR\.venv\Scripts\python.exe pipeline\render_preview.py action_specs\<name>.json --showcase ``` `compare_reference.py` tells you which frame windows still differ from the video; the last command produces `preview.mp4` and the side-by-side `showcase.mp4` — the same format as the demo GIF above. Two-performer plates add `--person left|right` to the estimate, one spec per fighter, and a `compare_pair.py` run — see [docs/PIPELINE.md](docs/PIPELINE.md) section 9. 4. **Iterate** with [docs/PIPELINE.md](docs/PIPELINE.md) and [docs/PITFALLS.md](docs/PITFALLS.md). ## What you need to bring (and where to get it) | What | Where | Notes | |---|---|---| | **A Mixamo character — any model** | [mixamo.com](https://www.mixamo.com) → Characters → download FBX Binary, T-pose | Adobe's terms don't allow redistributing them; `setup_rig.py` builds and validates the scene from your download | | **Blender 5.1+** | [blender.org](https://www.blender.org/download/) | | | **Blender MCP add-on** (official, Blender Lab) — landmark path only | [blender.org/lab/mcp-server](https://www.blender.org/lab/mcp-server/) | enable *Allow Online Access*; the legacy apply talks to its socket. The SMPL-X path runs `blender -b` and needs no add-on | | **GVHMR** (the pose estimator — **not in this repo**) | [github.com/zju3dv/GVHMR](https://github.com/zju3dv/GVHMR) | clone into `tools/GVHMR`; install per [docs/INSTALL.md](docs/INSTALL.md) — including a working Windows recipe (`docs/requirements_gvhmr_windows.txt` + prebuilt pytorch3d wheel) | | **GVHMR checkpoints** (~5 GB) | HuggingFace mirror | exact `curl` commands in [docs/INSTALL.md](docs/INSTALL.md) | | **SMPL-X body model** | [smpl-x.is.tue.mpg.de](https://smpl-x.is.tue.mpg.de/) | free research registration → download *SMPL-X v1.1*, place `SMPLX_NEUTRAL.npz` as shown in [docs/INSTALL.md](docs/INSTALL.md) | | **GPU** | ~8 GB VRAM | developed on an RTX 4080 | ## Docs | Doc | What it covers | |---|---| | [docs/INSTALL.md](docs/INSTALL.md) | Every dependency, step by step, Windows-proven | | [docs/PIPELINE.md](docs/PIPELINE.md) | The operational loop + the action_spec schema, field by field | | [docs/RIG.md](docs/RIG.md) | Mixamo rig conventions: spaces, units, the rules that must never break | | [docs/PITFALLS.md](docs/PITFALLS.md) | Every mistake this pipeline's development paid for, so you don't pay twice | | [docs/PROMPTING.md](docs/PROMPTING.md) | Writing gen-video plate prompts that survive retargeting | ## License MIT — see [LICENSE](LICENSE), including third-party notes (Mixamo, GVHMR, SMPL-X, Blender MCP).