所属模块：
M6 · 多模态与生成模型 (Multimodal & Generative Models)| 专题分类：视频 / 3D / 音频 (Video, 3D & Audio Generative Models)| 难度等级：Easy
一、核心一句话结论 (One-Sentence Summary)
三种设计：全时空注意力（最贵）、时空分离（先空间后时间）、以及分块/窗口；需在质量与成本间权衡。
Video diffusion models manage the quadratic computational complexity of multi-frame sequences by factorizing attention into decoupled spatial self-attention, cross-attention, and temporal self-attention blocks.
二、核心考点要义 (Key Insights)
- 📌 全时空：所有帧所有位置互相注意（最贵、最强）
- 📌 时空分离：先做空间注意力（每帧内）、再做时间注意力（跨帧同位置）
- 📌 分块/窗口：局部时空窗口（成本最低）
English Insights:
– Factorized spatio-temporal attention: decomposes full 3D attention into spatial-only attention (operating across pixels within each frame) followed by temporal-only attention (operating across frames for each pixel)
– Complexity reduction factor: reduces attention complexity from quadratic in space-time $,mathcal{O}((T cdot H cdot W)^2),$$ down to $,mathcal{O}(T(HW)^2 + HW(T^2)),$
– Modern unified 3D DiT evolution: recent frontier models (Sora, CogVideoX) deploy full 3D attention across 3D patchified tokens using 3D RoPE and sequence parallelism
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{full}: O((THW)^2);qquad text{sep}: O((HW)^2T)+O(T^2HW);qquad text{window}: text{local}$$
数学机理：三种设计。(1) 全时空注意力（full spatio-temporal）——把所有帧的所有 patch 展平为一个长序列，做全局自注意力；复杂度 O((T·H·W)²)；优点——最强（任意帧任意位置可交互）；缺点——成本极高（T 稍大就不可行）。(2) 时空分离注意力（factorized / separated）——把注意力分解为两步：(a) 空间注意力——在每一帧内部做自注意力（所有帧并行）：复杂度 O((HW)²·T)；(b) 时间注意力——在同一空间位置跨帧做注意力：复杂度 O(T²·HW)。总复杂度 O((HW)²T+T²HW) ≪ O((THW)²)（因为 T 与 HW 是’相加’而非’相乘’）。为什么时间注意力要’同位置’——因为’同一空间位置在不同帧’对应’同一个物体/区域’（如’第 3 帧的左上角’与’第 4 帧的左上角’）；这样时间注意力建模’该位置如何随时间变化’（运动），而空间注意力建模’该帧内的空间关系’。优点——成本大幅降低、且符合’空间与时间可分离’的先验；缺点——无法建模’跨帧的跨位置’关系（如’第 3 帧左上角的物体移动到第 5 帧右下角’——这在分离注意力中需两步传递）。(3) 分块/窗口注意力——(a) 时空窗口（如 3D 窗口，只在局部时空块内注意）；(b) 分块生成（把视频切成片段，各自生成 + 拼接）；(c) 稀疏注意力（只对部分帧/位置）。优点——成本最低；缺点——长程依赖受限。其他设计——(a) 混合（部分层用全时空、部分用分离）；(b) 层次化（先在低分辨率做全时空、再高分辨率做分离）；(c) 隐式运动建模（用光流或运动 token 辅助）；(d) 参考帧机制（长视频用关键帧作为参考，避免全局注意力）。代表实现——(a) AnimateDiff——在图像扩散模型的注意力层间插入时间注意力层（只训时间层）；(b) Sora——用时空 patch 的 DiT（架构细节未完全公开）；(c) CogVideoX / Open-Sora——用时空分离或 3D 注意力；(d) Wan / HunyuanVideo——3D VAE + DiT。选择依据——(a) 短视频、质量优先 → 全时空或混合；(b) 中等长度 → 时空分离（主流）；(c) 长视频 → 分块 + 参考帧 + 稀疏注意力。评估——(a) 时间一致性（光流/帧间特征）；(b) 成本（延迟/显存）；(c) 长程一致性（长视频的全局一致）。实践建议——(a) 时空分离是主流折中；(b) 3D VAE 压缩时间维（降低 T 的有效值）；(c) 长视频用分块 + 参考帧。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Mathematical Mechanism: 1. Full 3D Spatio-Temporal Attention Complexity: For video latent tensor of shape $T times H times W$ with $N = T cdot H cdot W$ tokens: $$Omega(text{Full 3D}) = 2 (T cdot H cdot W)^2 D = 2 T^2 (HW)^2 D$$ 2. Factorized Spatial-Temporal Attention (Ho et al., Blattmann et al.): Decomposes processing into two decoupled sequential operations inside each Transformer block: (a) Spatial Attention: Reshapes tensor to $(B cdot T) times (HW) times D$. Attention operates purely across spatial tokens within the same frame: $$Omega(text{Spatial}) = T times 2 (HW)^2 D = 2 T (HW)^2 D$$ (b) Temporal Attention: Transposes and reshapes tensor to $(B cdot HW) times T times D$. Attention operates across time frames at the exact same spatial coordinate: $$Omega(text{Temporal}) = (HW) times 2 T^2 D = 2 (HW) T^2 D$$ (c) Total Factorized Complexity: $$Omega(text{Factorized}) = 2 T(HW)^2 D + 2(HW)T^2 D ll 2 T^2 (HW)^2 D$$ Slashing compute and memory by factor of $approx min(T, HW)$. 3. Modern 3D Patchify in DiT (Sora Paradigm): Extracts 3D spacetime cuboids of size $p_t times p_h times p_w$ (e.g., $2 times 2 times 2$). Positions are encoded using 3D RoPE decomposing frequencies across $(t, y, x)$ axes.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① ‘时空分离把乘法变加法’是关键——复杂度从 (THW)² 降到 (HW)²T+T²HW；面试中能给出这一复杂度对比是深度理解的标志。② ‘时间注意力在同位置’符合’运动’的直觉——同位置跨帧对应’同一物体的时间演化’。③ ‘分离的局限是跨帧跨位置’——需要多步传递（削弱长程时空依赖）；故有混合/层次化设计。④ ‘插入时间注意力层’是低成本的微调方案（AnimateDiff）——复用图像模型 + 只训时间层；这大幅降低训练成本。⑤ ‘3D VAE 压缩时间维’是必要的——否则 T 太大（成本爆炸）；故视频生成普遍用时空 VAE。⑥ 面试要点——被问’视频的注意力怎么设计’，应给出’全时空（最贵最强）/ 时空分离（主流折中，乘法变加法）/ 分块窗口（最省）‘与’时间注意力在同位置、跨帧跨位置需多步传递‘；能给出复杂度对比是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
Deep Dive & Engineering Trade-offs: ① The Expressivity vs Cost Trade-off: Factorized spatial-temporal attention enables training on consumer GPUs and scaling up to 64-128 frames. However, factorization imposes an architectural bottleneck: a pixel at coordinate $(x_1, y_1)$ at time $t_1$ cannot directly attend to a moving object at coordinate $(x_2, y_2)$ at time $t_2$ in a single layer. Diagonal motion must be approximated across multiple alternating spatial and temporal layers. Full 3D attention models complex motion trajectories natively, but requires distributed sequence parallelism (DeepSpeed Ulysses / Ring Attention) across 8+ GPUs. ② Temporal Positional Encodings: Temporal attention blocks require dedicated temporal positional encodings (sinusoidal or rotary). Without temporal position information, the model cannot distinguish forward motion from reverse motion. ③ Zero-Convolution Temporal Initialization: When fine-tuning a pre-trained 2D image diffusion model into a video generator (e.g., AnimateDiff), the newly added temporal attention layers are initialized with zero-projections: $h = h_{text{spatial}} + alpha cdot text{TemporalAttn}(h)$, with $alpha = 0$ initially, ensuring the model begins with the exact generative capabilities of the 2D foundation model. ⑤ Interview Strategy: Formulate the complexity comparison between full 3D attention and factorized spatial-temporal attention, detail tensor reshaping for spatial vs temporal blocks, explain why factorized attention struggles with fast diagonal motion, and describe zero-initialization temporal fine-tuning.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 用全时空注意力处理长视频（成本爆炸）
- ⚠️ 忽略时间注意力’同位置’的设计动机
English Pitfalls:
– Attempting full 3D spacetime attention without sequence parallelism on long video sequences, causing instant GPU memory exhaustion
– Omitting temporal positional encodings in temporal attention blocks, causing temporal disorder and inability to learn motion direction
– Initializing newly added temporal attention blocks with random weights, destroying pre-trained 2D spatial image generation quality
六、高频深度面试追问与预测 (Follow-Up Questions)
- 时空分离为什么能省这么多？
- Why does factorizing spatio-temporal attention into separate spatial and temporal blocks reduce computational complexity from $mathcal{O}((THW)^2)$ to $mathcal{O}(T(HW)^2 + (HW)T^2)$?
- 时间注意力为什么要’同位置’？
- How does AnimateDiff initialize temporal attention layers to zero to preserve pre-trained 2D diffusion priors during video fine-tuning?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
时空视频扩散架构、3D 高斯泼溅 (3DGS) 与语音音频生成模型(Spatiotemporal Video Diffusion, 3DGS & Audio Generation) - 🗺️ 知识图谱模块：
多模态与扩散模型导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。