Type a motion. A video model acts it out, and any rigged character performs it, fingers included.
A prompt goes to a fast video model on fal, your GPU recovers clean 3D body and hand motion from the clip,
and it plays live on a Mixamo character in the browser, ready to export to Blender or Unity.
## How it works
```mermaid
flowchart LR
P[prompt] --> V["FLUX 3 Max / H3 Max Turbo on fal"]
I["performer image or the last clip's final frame"] --> V
V --> L["MediaPipe · browser instant preview"]
V --> H["GVHMR · your GPU 3D body motion"]
V --> F["WiLoR · your GPU wrists + fingers"]
L --> R[Mixamo rig]
H -- cross-fade --> R
F --> R
R --> G[".glb · Blender render"]
```
| Stage | Where | Time (5s clip, RTX 4090) |
|---|---|---|
| Video | fal · FLUX 3 Max (720p) | ~8–9s (4.5s GPU) |
| | fal · H3 Max Turbo (480P) | ~2–3s (0.5s GPU) |
| **Live** preview: MediaPipe Pose, retargeted per frame | browser | immediate |
| **HQ** motion: GVHMR → SMPL rotations → Mixamo | local GPU worker | ~3.5–4s |
| **Fingers**: WiLoR (MediaPipe fallback), in parallel with GVHMR | local GPU worker | +~0.3s |
The live pass covers the wait; the HQ pass is the result. GVHMR sees the whole clip at once, outputs joint rotations
on a fixed-proportion body and recovers a gravity-aware root path, so there's no jitter, limb stretching or twisted
forearms. WiLoR adds the palm and all 15 finger joints per hand.
## Quick start
**Needs:** Node 18+ and a [fal](https://fal.ai) API key. For the HQ pass: Windows, an NVIDIA GPU,
[`uv`](https://docs.astral.sh/uv/) and `ffmpeg` on PATH.
```bash
git clone https://github.com/blendi-remade/prompt2rig && cd prompt2rig
cp .env.example .env # add your FAL_KEY
node server.mjs # → http://localhost:5177 (the live preview works now)
```
**HQ pass (recommended):**
```powershell
hq\setup.ps1 # venv, CUDA PyTorch, GVHMR + ~5 GB of checkpoints
```
Download **SMPL-X v1.1 (NPZ)** from [smpl-x.is.tue.mpg.de](https://smpl-x.is.tue.mpg.de/) (free registration) and put
`SMPLX_NEUTRAL.npz` in `hq/body_models/smplx/`.
**Finger tracking (optional):** WiLoR needs the MANO hand model. Register at [mano.is.tue.mpg.de](https://mano.is.tue.mpg.de),
download *Models & Code* (`mano_v1_2.zip`), then:
```powershell
hq\.venv\Scripts\python hq\setup_hands.py $HOME\Downloads\mano_v1_2.zip # WiLoR weights + MANO
```
Without MANO, fingers fall back to MediaPipe Hands (rougher depth). Then start the worker next to the web server:
```powershell
hq\.venv\Scripts\python hq\worker.py # → 127.0.0.1:5178, loads once and stays warm
```
## Using it
- **Generate** from a prompt or a preset. Every clip starts from the same flux-made *performer* (one person, full body,
plain backdrop, static camera), which is the input pose estimation likes best.
- **Video model.** *FLUX 3 Max* for the most realistic motion; *H3 Max Turbo* for the fastest loop and stream mode.
- **Character.** X Bot, Michelle or Soldier (textured Mixamo rigs from the three.js examples), or **load mixamo file…**
for any Mixamo `.fbx` / `.glb` from [mixamo.com](https://www.mixamo.com) (T-pose).
- **Track fingers.** WiLoR runs on the full-resolution clip; hands are matched to the body's wrists, rotated into the
motion's world frame and smoothed. Untracked hands rest in a natural curl.
- **Stream mode.** One continuous performance: each clip picks up from the last one's final frame, plays once its HQ
motion is in while the next one generates, and short idle clips fill the gaps between prompts.
- **Swap character every loop**, **cinematic camera**, portrait or landscape clips.
- **↓ glb** on any take exports the skinned character with its animation, and keeps a copy plus the source clip's URL in `takes/`.
**Prompts that retarget well** face the camera with the limbs clear of the body: dances, kicks, kata, sport mimes,
celebrations, clear hand shapes. Flips, spins, floor work and crossed or hidden arms confuse any single-camera pose
estimator; the presets and the prompt template steer away from them.
## Renders and recordings
**Blender.** Turn a saved take into a studio clip (cyclorama, three-point light, slow camera arc), headless:
```powershell
& "C:\Program Files\Blender Foundation\Blender 5.2\blender.exe" -b -P tools/render_blender.py -- takes/.glb --side-by-side
```
It writes `renders/.mp4` (1080×1920, or `--res 1920x1080`) and, with `--side-by-side`, the source clip next to the render.
**App recordings.** `tools/capture.mjs` drives an off-screen Chrome and records takes frame by frame at 2560×1440, so
nothing on your screen is touched:
```bash
node tools/capture.mjs --prompts tools/prompts_dynamic.txt --character xbot --out captures
node tools/capture.mjs --clip
