源码: comfy_extras/nodes_minimax_h3.py(ComfyUI ≥ v0.30.0 内置,无需第三方插件)
功能: 创建空的视频+音频联合 latent
| 参数 | 类型 | 默认 | 说明 |
|---|---|---|---|
| width | INT | 1344 | 必须 32 的倍数 |
| height | INT | 768 | 必须 32 的倍数 |
| length | INT | 124 | 帧数@24fps。snap 到 17k+5 网格 (124≈5s, 362≈15s)。训练范围 124-362,超出未测试 |
输出: LATENT (NestedTensor: video[B,24,T,H/16,W/16] + audio[B,32,2,T40])
功能: T2VA + FL2VA 的条件构建 + latent 创建一步完成
| 参数 | 类型 | 默认 | 说明 |
|---|---|---|---|
| clip | CLIP | — | CLIPLoader(type="minimax") |
| vae | VAE | — | Video VAE (fp16) |
| prompt | STRING | — | 多行文本,支持 dynamic prompts |
| width | INT | 1344 | 32 的倍数 |
| height | INT | 768 | 32 的倍数 |
| length | INT | 124 | 17k+5 网格 |
| first_frame | IMAGE | optional | 首帧:lanczos 拉伸到画布("disabled" crop) |
| last_frame | IMAGE | optional | 尾帧:aspect-preserving cover-crop("center" crop) |
输出: CONDITIONING(positive) + LATENT
首帧和尾帧的 resize 策略不同:
_resize(first_frame, width, height, "disabled") — 纯拉伸,不做裁剪_resize(last_frame, width, height, "center") — 保持比例 center-crop关键帧 latent 通过 VAE encode 后注入 conditioning 的 minimax_keyframes 字段,每步采样时重新注入(不参与去噪)。
功能: 参考图/视频/音频驱动的生成
| 参数 | 类型 | 说明 |
|---|---|---|
| clip | CLIP | CLIPLoader(type="minimax") |
| vae | VAE | Video VAE |
| audio_vae | VAE | Audio VAE (fp32) — R2V 必须额外加载 audio VAE |
| prompt | STRING | 多行文本 |
| width / height | INT | 1344×768 默认 |
| length | INT | 帧数(17k+5 网格) |
| ref_image_size | COMBO | "match"(缩到生成分辨率,快)或 "max"(保持 2048px 短边,身份保真但慢数倍) |
| ref_images | AUTOGROW[IMAGE] | ≤9 张参考图。自动按 index 编号 → <Picture 1>~<Picture 9> |
| ref_videos | AUTOGROW[IMAGE] | ≤3 个参考视频帧(24fps,2-15s) |
| ref_video_audios | AUTOGROW[AUDIO] | 对应编号视频的音轨 |
| ref_audios | AUTOGROW[AUDIO] | ≤3 个独立参考音频 |
参考内容按固定顺序进入 tokenizer:
<Picture 1>~<Picture N>)<Audio j> 标签在对应 <Video k> 之前)视频参考经过 adapt_canvas() 缩放,帧数 snap 到 17k+5。Qwen 编码器以 2fps 采样视频帧并附带时间戳。
功能: 设置视频/音频的 flow shift 值。作用于 Model 对象。
| 参数 | 默认 | 说明 |
|---|---|---|
| model | — | UNETLoader 输出的 model |
| shift_video | 12.0 | 视频噪声调度偏移。⚠️ 不要降低! |
| shift_audio | 3.0 | 音频噪声调度偏移 |
机制: 创建 ModelSamplingAdvanced(继承 DiscreteFlow + CONST),patch 到 model 的 model_sampling。
同时通过 transformer_options 把 shift 值传给 DiT,DiT 内部用 video schedule 反推 audio schedule。
官方模板用打包节点 4c314f31-...(一键节点),内部封装了 CLIPLoader + UNETLoader + VAELoader×2 + MiniMaxH3ImageToVideo + KSampler + VAEDecode + CreateVideo 的完整链路。
打包节点的 widget 参数:
| # | 参数 | 示例值 |
|---|---|---|
| 0 | prompt | "Cinematic, action movie trailer..." |
| 1 | width | 1344 |
| 2 | height | 768 |
| 3 | duration_seconds | 5 |
| 4 | seed | 556589502035082 |
| 5 | dit_model | minimax_h3_fl2va_pruned_int8_convrot.safetensors |
| 6 | text_enc | qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors |
| 7 | video_vae | minimax_h3_video_vae_fp16.safetensors |
| 8 | audio_vae | minimax_h3_audio_vae_fp32.safetensors |
R2V 模板不用打包节点,而是完整的底级节点拓扑,给用户最大控制力:
帧数计算公式(ComfyMathExpression 节点):
max(5, round(a * 24)) + (5 - (max(5, round(a * 24)) % 17)) % 17
# a = duration_seconds (PrimitiveFloat)
# 效果:将秒数转为帧数后 snap 到 17k+5 网格
# 例: 5s → round(5*24)=120 → (5-(120%17))%17 = (5-1)%17=4 → 120+4=124
R2V 采样器选择:res_multistep(不是 euler!)+ simple scheduler + 20步
从工作流模板逆向得到的实际 API 参数映射:
{
"task": "t2va",
"prompt": "integrated_multimodal_description: [Shot 1]...\noverall_soundscape:...\nnon_diegetic_music:...",
"conditions": [],
"target": {
"short_edge": 768,
"aspect_ratio": "16:9",
"duration_seconds": 10
},
"seed": 0
}
{
"task": "fl2va",
"prompt": "For the target video, at 0.00 seconds...\n\nintegrated_multimodal_description:...",
"conditions": [
{
"type": "image",
"uri": "file:///data/image.png",
"role": "keyframe",
"frame_index": 0 // 0=首帧; duration末=尾帧
}
],
"target": {
"short_edge": 768,
"aspect_ratio": "auto", // auto = 跟随输入图比例
"duration_seconds": 8
},
"seed": 0
}
{
"task": "ref2va",
"prompt": "subject_definitions:\n<Subject 1> is...\n\nsummary:\n[reference generation]...\n\nretention_analysis:\n...\n\ndetailed_description:\n...",
"conditions": [
{"type": "image", "uri": "file:///ref1.png", "role": "reference"},
{"type": "video", "uri": "file:///ref.mp4", "role": "reference"},
{"type": "audio", "uri": "file:///voice.mp3", "role": "reference"}
],
"target": {
"short_edge": 768,
"aspect_ratio": "auto",
"duration_seconds": 5
},
"seed": 0
}
SGLang 支持更丰富的采样控制:
| 参数 | 说明 |
|---|---|
num_outputs_per_prompt | 1-10,同 prompt 多输出。seed 自动 +index |
quality | "lossless"(默认) 或 "high"(Cache-DiT 加速) |
conditions[].start_time_seconds | 视频参考的非负偏移量 |
flow_shift / audio_flow_shift | 可覆盖默认 12.0/3.0 |
num_inference_steps | 默认 50 |
| 参数 | T2V/FL2VA | Ref2VA | 说明 |
|---|---|---|---|
| steps | 50 | 20 | T2V 用 50 步;R2V 模板用 20 步 |
| flow_shift (video) | 12.0 | 12.0 | ⚠️ 不要手动降低! |
| audio_flow_shift | 3.0 | 3.0 | |
| CFG | 1.0 | 1.0 | CFG-distilled 模型 |
| sampler | euler | res_multistep | R2V 用 res_multistep + beta/simple |
| scheduler | simple | simple/beta | ref 多时 beta 更优 |
adapt_canvas)BASE_SHORT_EDGE)MAX_PIXELS)CANVAS_MULTIPLE)常用分辨率:
| 比例 | 输出尺寸 | 说明 |
|---|---|---|
| 16:9 | 1344×768 | 横屏标准 |
| 9:16 | 768×1344 | 竖屏标准(上限) |
| 1:1 | 768×768 | 方形 |
| 21:9 | 1344×576 | 超宽 |
| 4:3 | 1024×768 | 传统 |
temporal_shape)帧数必须满足 n % 17 == 5(17k+5 网格):
| 秒数 | 帧数 | video latent T | audio latent T |
|---|---|---|---|
| ~5s | 124 | 37 | 200 |
| ~6s | 141 | 42 | 240 |
| ~8s | 175 | 52 | 320 |
| ~10s | 229 | 67 | 400 |
| ~15s | 362 | 107 | 600 |
video_latent_t = 2 if frames ≤ 5 else ((frames-5)//17)*5 + 2
audio_latent_t = round(duration * 40)(40Hz 音频)
| Quality | 等效步数 | 延迟 | SSIM vs lossless |
|---|---|---|---|
| lossless | 50 | 75s | 1.000 |
| high (Cache-DiT) | ~36 | 54s (1.4×) | 0.931 / PSNR 28.16dB |
| medium | ~20 | 30s | 0.818 |
| low | ~17 | 26s | 0.794 |
| 模式 | 指令行 |
|---|---|
| T2VA | 无指令行,直接开始 |
| I2VA | For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced. |
| FL2VA | How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot N) aligns with the S.SS-second mark of the target video. |
| L2VA | How the reference pictures align with the target video — <Picture 1> (from [Shot N]) aligns with the S.SS-second mark of the target video. |
| 字段 | 描述 | 字数建议 |
|---|---|---|
integrated_multimodal_description | 主描述体:视觉+动作+镜头+对话+场景音 | 无限制 |
overall_soundscape | 环境音摘要(风/雨/脚步/撞击/呼吸等非语言声) | 1-4 句 |
non_diegetic_music | 非叙事 BGM(角色听不到,观众听到) | 1-3 句。无则 N/A |
[Shot 1] 不加时间戳[Shot 2] At 00:03.500, the camera cuts to...the camera cuts to / the shot transitions to| 维度 | 可用值 |
|---|---|
| 运动类型 | Zoom In/Out · Push In/Pull Out · Pan L/R · Truck L/R · Tilt Up/Down · Pedestal Up/Down · Arc Shot · Tracking Shot · Static Shot · Shake Slightly/Strongly · POV · Roll CW/CCW |
| 幅度 | with small amplitude · with large amplitude(中等省略) |
| 速度 | at slow speed · at fast speed(正常省略) |
写法示例: The camera pushes in with small amplitude at slow speed toward the folded letter.
The young woman with a quiet, breathy voice (S1) says: <d>[English] I get off at the next station.</d>
The two children (S1,S2) shout together, <d>[English] Wait for us!</d>
says in an off-screen voiceover + 必须加 his/her lips remain completely closed<scenetrans> 标记连续性<cutoff>画面可见文字(招牌/字幕/霓虹灯)用英文双引号:A red neon sign reading "营业中" glows above the doorway.
integrated_multimodal_description: [Shot 1] Live-action, cinematic, a medium-wide
shot frames a baker opening the shutters of a small street bakery before sunrise.
The camera pushes in with small amplitude at slow speed as the middle-aged baker
with a calm, slightly raspy voice (S1) places a fresh loaf on the wooden counter
and says: <d>[English] First batch of the morning.</d>
[Shot 2] At 00:05.000, the camera cuts to a close-up of steam rising from the
sliced bread while the baker's final words carry over from the previous shot.
overall_soundscape: Wooden shutters scrape open over a quiet street as trays clink
softly inside the bakery. The doorbell rings once, followed by light footsteps and
the crisp sound of bread being sliced.
non_diegetic_music: A soft acoustic-guitar pattern at a moderate tempo, joined by
sparse upright-bass notes and a gentle fade at the end.
| 段落 | 用途 |
|---|---|
subject_definitions | 定义 <Subject N> / <Picture N> / <Video N> / <Audio N> |
summary | 带任务类型前缀概述,如 [video editing + audio reuse] |
retention_analysis | 保留程度:fully_preserved / partially_preserved / attribute_transfer / weak_reference;音频:fully_copy / partially_copy / reference |
detailed_description | 主描述体(350-500词),比 T2V 的 integrated_multimodal_description 更详细 |
overall_soundscape | 环境音 |
non_diegetic_music | 非叙事 BGM |
| 类型 | 适用 |
|---|---|
keyframe completion | 图片=首帧/关键帧/尾帧 |
reference generation | 图/视频/音频提供生成指导 |
video editing | 直接修改已有视频 |
video continuation | 从已有视频续拍 |
audio reuse | 音频完整/部分复用 |
audio reference | 仅参考音色/节奏/风格 |
| 组件 | 参数 | 详情 |
|---|---|---|
| H3-Omni-Transformer | 33B (推理 ~20B) | 50 层 × 56 头 × 128 dim = 5376 hidden。~13B AdaLN 推理可跳过 |
| H3-Encoder | Qwen3-VL-32B | 64 层 text + 27 层 vision。5120 hidden。RoPE theta=5000000 |
| VisualVAE | f16t4d24 | 空间 16× / 时间 4× / 24 latent ch。ViT 解码器 36 层 |
| AudioVAE | 32kHz → 40Hz latent | 32 latent ch。encoder_rates=[2,4,4,5,5] → 800× 压缩 |
| patch_size | [1,2,2] | (time,height,width) → 空间有效下采样 32× |
| 文件 | 量化 | 大小 | 3090 推荐 |
|---|---|---|---|
| Diffusion Model (fl2va) | |||
| minimax_h3_fl2va_bf16 | bf16 | 61.7 GB | ❌ |
| minimax_h3_fl2va_int8_convrot | int8 | 31.7 GB | ❌ VRAM 不够 |
| minimax_h3_fl2va_pruned_int8_convrot | int8 剪枝 | 19.5 GB | ★ 推荐 |
| minimax_h3_fl2va_pruned_fp8_scaled | fp8 剪枝 | — | ? 待测 |
| Text Encoder | |||
| qwen3vl_32b_minimax_h3_bf16 | bf16 | 48.0 GB | ❌ |
| qwen3vl_32b_minimax_h3_int8_convrot | int8 | 25.3 GB | ❌ |
| qwen3vl_32b_minimax_h3_nvfp4_awq | NVFP4+AWQ | 14.6 GB | ★ 推荐(3090 软件模拟) |
| 阶段 | VRAM 需求 | 3090 | 判定 |
|---|---|---|---|
| Text Encoding (Qwen3-VL-32B) | ~15.6 GB | 24 GB | ✓ |
| Diffusion (fl2va int8 pruned) | ~22.5 GB | 24 GB | ⚠️ 勉强 (1.5GB headroom) |
| VAE Decode (video+audio) | ~7.5 GB | 24 GB | ✓ |
工作负载: 1344×768, 124帧, 50步, 5s T2VA
| 权重 | 精度 | 编码器 | 延迟 | 峰值/GPU |
|---|---|---|---|---|
| FL2VA | BF16 | auto | 19.0s | 83.6 GB |
| FL2VA | FP8 | auto | 18.0s | 51.9 GB |
| Ref2VA | BF16 | auto | 29.1s | 84.0 GB |
| Ref2VA | FP8 | auto | 27.1s | 52.8 GB |
| GPU | 拓扑 | 延迟 | 峰值/GPU |
|---|---|---|---|
| 4×H200 | Ulysses4 | 74.4s | 94.3 GB |
| 4×H200 | TP2+Ulysses2 | 78.3s | 63.5 GB |
| 4×H100 | TP2+Ulysses2 | 13.3s | 66.0 GB |
| 4×H100 | FSDP+Ulysses4 | 13.4s | 57.0 GB |
| 2×RTX5090 | TP2+layerwise | 559.7s (50步) | 26.3 GB |
| 8×MI355X | Ulysses8 | ~65s | 97.4 GB |
| quality | 延迟 | 加速 | SSIM | PSNR |
|---|---|---|---|---|
| lossless | 75.1s | 1.0× | 1.000 | — |
| high | 53.7s | 1.4× | 0.931 | 28.16 dB |
根因: H3 是 Flow Matching 模型,shift 控制噪声调度曲线偏移。值太低 → 高噪声阶段被压缩 → 帧间像素微漂移 → 水流波动。
解决: 永远用 shift_video=12.0(默认值),不要手动降低。
根因: H3 是 CFG-distilled 单分支模型。
解决: 不要启用 --enable-cfg-parallel 或 --cfg-parallel-size > 1,会被拒绝。CFG=1.0 即可。
根因: SGLang 验证发现 spatial 和 spatial_shard 模式有输出不匹配。
解决: 用默认的 tiled decode recipe(overlapping tiles)。
根因: 当前 torch.compile 路径会改变模型数值输出。
解决: 不要用于生成 CI 一致性 ground truth。推荐用 eager BF16/FP32。
docker compose up 重建容器根因: compose 检测配置变化会 Recreate → 文件系统从镜像恢复。
解决: 修改容器后用 docker restart,永远不用 docker compose up。
根因: 参考 latent(keyframes + refs)在每个采样步骤都重新注入 DiT(never denoised),所以 ref_image_size="max" 会比 "match" 慢数倍。
解决: 速度优先用 "match",身份保真优先用 "max"(2048px 短边)。
源码差异: 首帧 "disabled" crop(纯拉伸),尾帧 "center" crop(保持比例居中裁剪)。
影响: 首尾帧的宽高比不同时,尾帧会被裁剪而不是拉伸。输入图需提前调整到目标比例。
integrated_multimodal_description + overall_soundscape + non_diegetic_music),不是自由文本。管线 P09 shot-list 的 prompt 应按此格式输出。<d>: 直接控制视频中的角色对话时间点和内容。可用 [Chinese] 语言标签。max(5, round(sec*24)) + (5-(max(5,round(sec*24))%17))%17。管线需要按此 snap。