所属模块：
M6 · 多模态与生成模型 (Multimodal & Generative Models)| 专题分类：视频 / 3D / 音频 (Video, 3D & Audio Generative Models)| 难度等级：Easy
一、核心一句话结论 (One-Sentence Summary)
时间一致性（帧间不闪烁）、运动建模、长时序依赖、以及 token 数与成本（帧数 × 每帧 token）。
Video generation extends 2D image synthesis into the temporal dimension, confronting severe challenges in inter-frame consistency, continuous physical law adherence, and explosive spatio-temporal computational costs.
二、核心考点要义 (Key Insights)
- 📌 时间一致性：相邻帧不能闪烁/跳变（最核心）
- 📌 运动建模：需理解物理运动与因果关系
- 📌 成本：帧数 × 每帧 token → 注意力成本爆炸
- 📌 长时序：长视频的全局一致性与剧情连贯
English Insights:
– Temporal consistency hurdles: independent or weakly coupled 2D frame generation causes severe flickering, identity morphing, and boundary jitter across adjacent frames
– Physical common-sense modeling: video models must learn intuitive physics (gravity, collision dynamics, fluid flow, object permanence) without explicit 3D engines
– Spatio-temporal computational scaling: expanding generation to $T$ frames multiplies token count and self-attention memory by $,T,$ to $,T^2,$, requiring 3D causal VAEs and spatio-temporal factorization
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{video}: Ttimes Htimes W;qquad text{cost}propto(Tcdot Hcdot W)^2;qquad text{challenge}: text{temporal consistency}$$
数学机理：四个主要挑战。(1) 时间一致性（temporal consistency）——相邻帧必须连贯（不闪烁、不跳变、物体不’瞬移’）；这是视频生成的核心难题——因为 (a) 图像模型逐帧独立生成会导致闪烁（每帧的纹理/光照略有不同）；(b) 需模型显式建模’帧间的对应关系’。(2) 运动建模——需理解 (a) 物理运动（重力、碰撞、流体）；(b) 因果关系（’球被踢→飞出去’）；(c) 复杂动作（人物的连续动作）；这比’静态图像’难得多（静态图只需’合理’，视频需’物理与因果合理’）。(3) 成本爆炸——视频的 token 数 = T（帧数）× 每帧 token；故 (a) 注意力成本 ∝ (T·H·W)²（若用全局时空注意力）；(b) VAE 潜空间——视频常用时空 VAE（时间维也压缩，如 4~8 倍）；(c) 生成 5 秒 24fps 的 720p 视频 = 120 帧 × (720/8 × 1280/8) 潜变量 → 巨大的序列长度（数万 token）；故需时空注意力 + 分块。(4) 长时序依赖——长视频（分钟级）的 (a) 全局一致性（人物/场景不变）；(b) 剧情连贯（事件顺序）；(c) 长程依赖（开头的元素在结尾出现）；这需要’记忆’机制（如关键帧参考、长上下文）。其他挑战——(a) 数据稀缺（高质量视频远少于图像）；(b) 评估困难（时序质量难自动评估，见多图/视频评估题）；(c) 可控性（视频编辑/控制比图像难）；(d) 计算资源（训练与推理都极贵）。技术路线——(1) 时空注意力（见下一题）；(2) 时空 VAE（时间维压缩）；(3) 分块生成 + 拼接（避免全局注意力的成本）；(4) 关键帧 + 插帧（先生成关键帧、再插值中间帧）；(5) 图像模型 + 运动模块（如 AnimateDiff 用图像模型 + 时间注意力层）；(6) 原生视频 DiT（如 Sora 用时空 patch 的 DiT）。评估——(a) 时间一致性（光流平滑度、帧间特征相似度）；(b) 运动合理性（人工或物理约束）；(c) 文本对齐（视频-文本相似度）；(d) 视觉质量（FVD 等）；(e) 人工评估（仍是主要方法）。实践建议——(a) 短片段优先（长视频难）；(b) 时空 VAE 降成本；(c) 关键帧 + 插帧（比直接生成长视频更稳）；(d) 明确评估维度（一致性/运动/对齐）。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Mathematical Mechanism: 1. The Spatio-Temporal Token Explosion: Let an image latent be $z in mathbb{R}^{C times H times W}$. A video consists of $T$ consecutive latent frames: $$Z_{text{video}} in mathbb{R}^{C times T times H times W}$$ Token count under patch size $(p_t, p_s, p_s)$: $$N = left( frac{T}{p_t} right) left( frac{H}{p_s} right) left( frac{W}{p_s} right)$$ For a 5-second video at 24 FPS ($T = 120$ frames) at $1024 times 1024$ resolution with VAE compression $(f_t=4, f_s=8)$: Latent dimensions are $30 times 128 times 128$. For patch size $(1, 2, 2)$: $$N = 30 times 64 times 64 = 122,880 text{ tokens}$$ Full pairwise self-attention requires: $$mathcal{O}(N^2 D) = mathcal{O}big( (1.23 times 10^5)^2 D big) approx 1.5 times 10^{10} D text{ FLOPs per layer}$$ Requiring spatial-temporal factorization to be computationally tractable. 2. Temporal Consistency Loss Metric: Measured via optical flow warping error. Given estimated forward optical flow $v_t to t+1$: $$mathcal{L}_{text{temp}} = frac{1}{T-1} sum_{t=1}^{T-1} M_t odot big| I_{t+1} – text{Warp}(I_t, v_{t to t+1}) big|_1$$ where $M_t$ is an occlusion mask. Large values indicate visual flickering and temporal discontinuities.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① ‘时间一致性是最核心挑战’——它区分了’视频生成’与’逐帧图像生成’；面试中能指出这一点是深度理解的标志。② ‘成本 ∝ (T·H·W)²’是根本约束——故视频生成需要时空压缩（时空 VAE）与分块；这解释了’为什么视频生成如此昂贵’。③ ‘运动/因果建模’比静态图像难——因为需要’物理与世界模型’的知识（而非仅’合理的外观’）。④ ‘长时序依赖需记忆机制’——长视频的全局一致性需要’参考帧/长上下文’；这是当前研究的重点。⑤ ‘数据稀缺’是训练瓶颈——高质量视频数据远少于图像；故常 (a) 用图像数据预训练、(b) 用视频数据微调。⑥ 面试要点——被问’视频生成难在哪’，应给出’时间一致性 + 运动/因果建模 + 成本爆炸（(THW)²）+ 长时序依赖 + 数据稀缺‘与’技术路线（时空注意力/时空 VAE/分块/关键帧+插帧）‘；能指出’时间一致性是核心’是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
Deep Dive & Engineering Trade-offs: ① 3D Causal VAE Compression: Standard 2D VAEs encode each frame independently, achieving only spatial compression ($8times$). Modern video foundation models (Sora, CogVideoX) deploy 3D Causal Temporal VAEs: using 3D causal convolutions that compress both spatial dimensions ($8times$) and temporal dimensions ($4times$). Compressing 16 video frames into 4 temporal latent frames slashes diffusion token count by $75%$, while causal padding ensures the VAE can decode frames in a streaming fashion. ② The ‘Hallucinated Physics’ Problem: Because video diffusion models are trained on internet video clips, they learn statistical pixel transitions rather than true Newtonian mechanics. Common failures include objects vanishing when occluded (breaking object permanence), water flowing upwards, or hands growing extra fingers during motion. Adding synthetic 3D physics engine trajectories (Unreal Engine simulations) to training data grounds physical realism. ③ Autoregressive Extension vs Full-Sequence Generation: Generating full 10-second videos in one diffusion pass is memory-prohibitive. Production architectures generate an initial 2-second anchor clip, then autoregressively condition on trailing frames to generate subsequent extensions, using sliding-window KV caching. ⑤ Interview Strategy: Formulate the spatio-temporal token calculation showing quadratic attention explosion ($N > 120K$), define temporal consistency via optical flow warping error, explain 3D causal VAE compression ($f_t times f_s times f_s$), and contrast statistical pixel correlation with true physical modeling.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 忽略时间一致性（逐帧生成导致闪烁）
- ⚠️ 不考虑成本（直接全时空注意力）
English Pitfalls:
– Attempting full 3D self-attention over raw video patch tokens, which causes catastrophic out-of-memory crashes due to $100K+$ sequence lengths
– Encoding video frames independently with 2D VAEs, missing temporal compression and creating severe inter-frame flickering
– Evaluating video models exclusively on static image metrics (FID) without measuring temporal consistency and motion coherence
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么’时间一致性’最难？
- Why is 3D causal temporal compression in VAEs essential for scaling video diffusion models to long-duration sequences?
- 视频的 token 数怎么算？
- How does optical flow warping error quantify inter-frame temporal flickering independently of static visual quality?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
时空视频扩散架构、3D 高斯泼溅 (3DGS) 与语音音频生成模型(Spatiotemporal Video Diffusion, 3DGS & Audio Generation) - 🗺️ 知识图谱模块：
多模态与扩散模型导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。