Inside our video stack: Wan 2.2 and MiniMax H3 on ComfyUI workers

Our video API serves six models — minimax-h3, minimax-h3-ref, wan2.2-s2v, wan2.2-i2v, wan2.2-animate and latentsync-1.6 — and every one of them runs as a ComfyUI graph on a single 32 GB consumer GPU (an RTX 5090) behind our own scheduler. This post is the engineering view: how a request becomes a graph, what each model needs, and three things we got wrong on the way.

From a request to a graph

client ── POST /v1/videos ──▶ avionics (API gateway: auth, validation, credit hold)
                                   │  JSON + base64 inputs
                                   ▼
                             engine-server (Go, scheduler)
                                   │  NATS: job → one worker
                                   ▼
                             engine-agent (Go, on the GPU host)
                                   │  1. fetch inputs over HTTPS (signed URLs)
                                   │  2. POST /upload/image  → ComfyUI input/
                                   │  3. POST /prompt        → API-format graph
                                   │  4. watch the progress websocket
                                   ▼
                             ComfyUI (native nodes only)
                                   │  SaveVideo → mp4
                                   ▼
                    agent uploads the output → avionics settles the hold

The agent never hands ComfyUI a hand-built workflow file. Each model has a Go builder that emits the API-format graph from native nodes only, checks that every node it patches is the class it expects (a template that drifts fails loudly instead of rendering the wrong thing), and sets the canvas directly rather than through a resolution selector. A model is only advertised to the scheduler while a probe confirms its node classes and weight files are present; ComfyUI's VRAM is freed after every job, so a card can alternate between H3 (fp8, ~30 GB working set) and the Wan family without restarts.

Inputs travel as signed HTTPS URLs straight to the worker, not through the message bus: a keyframe plus a driving video regularly exceeds what a lossy, high-latency link to the GPU host can move through the bus's object store before its deadline.

MiniMax H3: two checkpoints, one frame grid

H3 ships as two DiTs. fl2va (first/last-frame to video+audio) serves t2v and i2v; ref2va serves ref, where up to nine reference images are addressed in the prompt as <Picture 1>…<Picture 9>. We run both pruned and fp8-scaled, with the Qwen3-VL 32B text encoder quantised to NVFP4 (AWQ) and the official turbo LoRAs (8 steps for fl2v, 4 for ref). Picture and stereo audio come out of the same sampling pass — the audio VAE decodes alongside the video VAE.

// MiniMax H3 is trained on clip lengths of 17k + 5 frames at 24 fps
// (124 ≈ 5 s ... 362 ≈ 15 s). The template turns seconds into frames:
//
//   frames = max(5, round(seconds * 24))
//   frames = frames + (5 - frames) mod 17      // snap up to 17k + 5
//
// 5 s → 124 frames, 10 s → 243 frames. A "still" keyframe renders the
// minimum 5-frame clip and keeps frame 2 as a PNG.

Output is 768P only; the 2K regenerator is not open, so a 2K request is refused rather than silently downgraded.

H3 does what the prompt says — all of it

With the turbo LoRA at CFG 1 there is no negative prompt to lean on, and the text encoder is strong enough that every noun in the prompt tends to appear on screen. Three failures taught us to write prompts as a description of the shot and nothing else:

Negatives draw. “No on-screen text or subtitles” produced subtitles; “a jacket stays on” put a jacket on a character who never had one. We now say only what is in the frame.

Lighting lines move the camera. A setting line naming the light source (“lit by the desk lamp”) made H3 pull back to show the lamp, ruining a close-up. The keyframe already carries the light, so the video prompt no longer describes it.

A reference is not a first frame. In ref mode the keyframe is only <Picture 1>: H3 redraws the scene, repositions the subject and, prompted with a pet's generic “may wear a hat, a bandana, a backpack”, dressed it in all three. In i2v the keyframe is latent frame 0 and stays. The remaining failure mode is a hard cut about a second in when the prompted action contradicts the keyframe; opening the prompt with “one continuous take from the first frame” and describing an action that starts where the frame is removed it in every run we tried.

Giving H3 an audio track — and what it does with it

Audio enters H3 two ways. In ref mode a clip becomes <Audio 1> through the node's ref_audios input: a voice the character speaks in, while H3 still writes its own soundtrack. In t2v / i2v we chain MiniMaxH3AddGuide after the image-to-video node, which can anchor an image, a clip or an audio track at any latent frame:

// i2v clip with a soundtrack anchored at frame 0
"oa_first_frame": { "class_type": "LoadImage",  "inputs": { "image": "kf.png" } },
"oa_guide_audio": { "class_type": "LoadAudio",  "inputs": { "audio": "line.wav" } },
"140:131": { "class_type": "MiniMaxH3ImageToVideo",
             "inputs": { "first_frame": ["oa_first_frame", 0], "prompt": "...",
                         "width": 768, "height": 1344, ... } },
"oa_guide": { "class_type": "MiniMaxH3AddGuide",
              "inputs": { "positive": ["140:131", 0], "latent": ["140:131", 1],
                          "frame_idx": 0,
                          "audio_vae": ["140:120", 0], "audio": ["oa_guide_audio", 0] } },
"140:126": { "class_type": "BasicGuider",
             "inputs": { "conditioning": ["oa_guide", 0], ... } }

We expected the guide to pin the soundtrack. Comparing loudness envelopes of the output against the input line says otherwise:

RMS per 0.1 s (8 kHz mono), speech marked ▇

our TTS line       ▁▁▁▁▁▁▁▁▁▁▇▇▇▁▁▁▁▁▁▁▁▇▇▇▇▇▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁   1.0–1.3 s, 2.1–2.6 s
run 1 (guided)     ▁▁▁▁▁▁▁▁▇▇▇▇▇▁▁▁▁▁▁▁▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▁▁▇▇   follows it
run 2 (guided)     ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▇▇▇▁▁▁▁▁▁▁▇▇▇▇▇▇▇▁▁▁▁   ~2.9 s, ~4.0 s
run 3 (no audio)   ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▇▇▇▇▁▁▁▁▁▁▇▇▇▇▇▇▇▇▁   ~3.0 s, ~4.0 s

One run tracks the guide; the next is indistinguishable from a run with no audio at all. The guide conditions, it does not constrain — so we expose it as a soundtrack hint, keep H3's own audio in the cut, and route shots that must lip-sync to a specific recording to Wan 2.2 S2V, where the input audio is the soundtrack.

Wan 2.2 S2V: motion that comes from the audio

S2V is a single 14B model with an audio encoder: keyframe + waveform in, 16 fps video of exactly the audio's length out, rendered in chunks. We run it with the lightx2v 4-step LoRA, ModelSamplingSD3 shift 8 and CFG 1 (a no-turbo path at 20 steps / CFG 6 exists for quality runs). Distillation costs it nothing visible because the motion is driven by the audio — head, lips and shoulders follow the beat and the syllables — not invented by the sampler. That is why our singing music videos look alive at 4 steps.

Wan 2.2 I2V: why the MoE rendered slow motion

I2V A14B is a mixture of two experts split by noise level: the high-noise expert lays out composition and motion in the first steps, the low-noise expert resolves detail. Our first graph followed the stock 4-step recipe — lightx2v LoRAs on both experts, CFG 1, two steps each. Animals told to run off toward the corner stepped in place.

Timing was the first suspect and was clean: 16 fps, 96 frames for 6 s, a 6.0 s container. The motion itself was compressed. With the distilled high-noise expert at CFG 1 there are two steps and no guidance to commit to large displacement, so the trajectory collapses toward a gentle drift. S2V never showed it because its motion has another source.

The fix keeps the speed of the LoRA where it is harmless and removes it where motion is decided:

// Wan 2.2 I2V A14B: two experts, 6 steps, three KSamplerAdvanced stages
shift_raw   = ModelSamplingSD3(unet_high,               shift=5)   // no LoRA
shift_high  = ModelSamplingSD3(LoRA(unet_high, lightx2v_high), shift=5)
shift_low   = ModelSamplingSD3(LoRA(unet_low,  lightx2v_low),  shift=5)

sample_raw  = KSamplerAdvanced(shift_raw,  steps=6, cfg=3.5, start=0, end=2,
                               add_noise=enable,  leftover_noise=enable)
sample_high = KSamplerAdvanced(shift_high, steps=6, cfg=1.0, start=2, end=3,
                               add_noise=disable, leftover_noise=enable)
sample_low  = KSamplerAdvanced(shift_low,  steps=6, cfg=1.0, start=3, end=6,
                               add_noise=disable, leftover_noise=disable)

// length: WanImageToVideo renders L = 4k + 1 latent frames; we ask for
// F = seconds * 16 and drop the extra frame (ImageFromBatch) — 6 s → L 97, F 96.

Same seed, same keyframe, same prompt: the dog in the old graph trots in place; in the three-stage graph it runs out of frame down the street with the camera following. The un-distilled steps double the cost of only two of six steps, and a 6 s 480×832 shot still renders in about a minute.

Keeping it up

A GPU shared by six video models and a speech engine fails in mundane ways, so the platform now handles each one explicitly: every queued job submits under its own deadline (one slow pre-check used to starve the rest of a batch); a 5xx or busy answer from a dependency puts the job back in the queue instead of failing it; the speech pool retires an instance that holds a request 20 s past its caller's deadline or makes no progress for three minutes; and the monitor counts only jobs inside the live window, so rows nobody closed out are reported as stale instead of as a ten-day-old queue.

All of this ships in Engine Desktop 0.6.25 for GPU providers — see the Download page — and every model is callable through /v1/videos today.