所属模块：
M6 · 多模态与生成模型 (Multimodal & Generative Models)| 专题分类：扩散模型基础 (Diffusion Models Foundations (DDPM))| 难度等级：Hard
一、核心一句话结论 (One-Sentence Summary)
文本条件常用 cross-attention（U-Net）或 adaLN 调制（DiT）；类别/低层条件常用 concat；可组合使用。
Diffusion models inject conditional guidance via three primary architectural mechanisms: spatial channel concatenation for pixel-aligned masks, cross-attention for rich text semantics, and adaptive layer normalization (adaLN) for global embeddings.
二、核心考点要义 (Key Insights)
- 📌 cross-attention：隐状态查询条件（文本条件的主流）
- 📌 adaLN（自适应层归一化）：用条件调制归一化的 scale/shift
- 📌 concat：把条件在通道维拼接（类别/掩码/低层条件）
English Insights:
– Channel concatenation: concatenates condition tensors directly along the input spatial channel dimension; optimal for spatially aligned conditions (ControlNet, depth, inpainting masks)
– Cross-attention: projects visual intermediate features as queries and text embeddings as keys and values; optimal for unaligned token sequences like text prompts
– Adaptive Layer Normalization (adaLN): modulates normalized feature activations using scale and shift parameters dynamically predicted from timestep and class embeddings (DiT standard)
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{cross-attn}: mathrm{Attn}(Q=h, K,V=c);qquad text{adaLN}: hleftarrowgamma(c)odot h+beta(c);qquad text{concat}: [h;c]$$
数学机理：三种条件注入方式。(1) Cross-Attention——把条件（如文本嵌入）作为 key/value，隐状态作为 query：Attn(Q=h, K=V=c)；每个空间位置可’查询’条件的不同部分。特点——(a) 灵活（条件长度可变，如不同长度的文本）；(b) 语义对齐（注意力权重可解释为’哪些词影响哪些区域’）；(c) 主流（Stable Diffusion 的 U-Net 用 cross-attention 注入文本）。代价——额外的注意力计算（O(N·M)，M 为条件长度）。(2) adaLN（Adaptive Layer Norm）——用条件调制归一化层的参数：h←γ(c)⊙LN(h)+β(c)，其中 γ(c)、β(c) 由条件经 MLP 生成。特点——(a) 计算便宜（只是逐元素缩放与平移，无注意力）；(b) 全局调制（所有位置用同一组 γ/β，适合’全局风格/类别’类条件）；(c) DiT 的选择（DiT 用 adaLN-Zero 注入 t 与类别/文本条件）。局限——若条件很长（如长文本），用一个向量表示会丢信息；故 DiT 的文本条件常用’池化的文本嵌入 + adaLN’（简单但对长文本表达能力有限）。(3) Concat（拼接）——把条件在通道维（或序列维）与隐状态拼接：[h; c]。特点——(a) 简单（无需额外模块）；(b) 适合与隐状态同构的条件（如类别 embedding 广播、掩码、低层特征、参考图的潜变量）；(c) 不适合变长条件（文本长度可变，拼接需固定长度）。组合使用——实践中常组合：(a) Stable Diffusion——时间步用 adaLN（或加性嵌入）+ 文本用 cross-attention + 参考图用 concat（inpainting 的掩码与掩码图）；(b) DiT——时间步与类别/池化文本用 adaLN-Zero；(c) ControlNet——在残差层面注入控制条件（见条件控制题）；(d) IP-Adapter——用额外的 cross-attention 注入图像条件（与文本 cross-attention 并列）。选择依据——(a) 条件长度可变/需细粒度对齐（文本）→ cross-attention；(b) 全局/标量条件（时间步、类别、风格）→ adaLN；(c) 与隐状态同构/固定长度（掩码、参考图潜变量）→ concat；(d) 可组合。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Mathematical Mechanism: 1. Channel Concatenation Formulation (Inpainting / ControlNet): For spatial latent $x_t in mathbb{R}^{C times H times W}$ and spatially aligned condition $c_{text{spatial}} in mathbb{R}^{C’ times H times W}$ (e.g., inpainting mask + masked image): $$x_{text{input}} = [x_t ; c_{text{spatial}}] in mathbb{R}^{(C + C’) times H times W}$$ The first convolutional layer is expanded from $C$ to $C + C’$ input channels. 2. Cross-Attention Conditioning (Latent Diffusion / SD): Intermediate U-Net/DiT spatial tokens $h in mathbb{R}^{N times d}$ attend to contextual text sequence $c_{text{text}} in mathbb{R}^{L times d_{text{text}}}$: $$Q = h W_Q, quad K = c_{text{text}} W_K, quad V = c_{text{text}} W_V$$ $$text{CrossAttn}(h, c_{text{text}}) = text{Softmax}left( frac{Q K^T}{sqrt{d_k}} right) V$$ 3. Adaptive Layer Normalization (adaLN-Zero in DiT, Peebles & Xie): For global vector $y$ (combining timestep embedding $t$ and class/pooled text embedding $c$): Compute scale $gamma$, shift $beta$, and gate $alpha$ via linear projection: $$(gamma_1, beta_1, alpha_1, gamma_2, beta_2, alpha_2) = text{MLP}(t_{text{embed}} + c_{text{embed}})$$ Inside the DiT block: $$h’ = h + alpha_1 cdot text{SelfAttn}big( (1 + gamma_1) odot text{LayerNorm}(h) + beta_1 big)$$ $$h_{text{out}} = h’ + alpha_2 cdot text{MLP}big( (1 + gamma_2) odot text{LayerNorm}(h’) + beta_2 big)$$ Initializing $alpha_1, alpha_2$ to zero acts as an identity function at step 0, stabilizing early training.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① ‘条件的信息量与结构决定注入方式’——文本（长、变长、需对齐）用 cross-attention；标量/全局（时间步、类别）用 adaLN；同构（掩码）用 concat。② ‘DiT 用 adaLN 的取舍’——计算便宜、实现简单，但对’长文本’的表达力有限（故后续 DiT 变体也加了 cross-attention，如 PixArt-α 用 cross-attention 处理 T5 文本）。③ ‘adaLN-Zero 的巧妙’——把 γ/β 初始化为 0，使训练开始时’条件分支为恒等’（h←h），模型从’无条件’开始学；这提升了训练稳定性（见 adaLN-Zero 题）。④ ‘CFG 与条件注入的关系’——CFG 需要’有条件’与’无条件’两次前向；故条件注入方式需支持’关闭条件’（如把条件设为空/零）。⑤ ‘多条件的组合’——当有多个条件（文本 + 参考图 + 控制信号）时，需分别注入（cross-attention 各一路、或 concat + adaLN）；这是 ControlNet/IP-Adapter 的设计空间。⑥ 面试要点——被问’条件怎么注入’，应给出’cross-attention（文本，灵活但贵）+ adaLN（全局条件，便宜）+ concat（同构条件）‘与’选择依据（条件的信息量与结构）+ 可组合‘；能指出’DiT 用 adaLN 但对长文本表达力有限’是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
Deep Dive & Engineering Trade-offs: ① The Conditioning Mechanism Selection Guide: (a) Cross-Attention: Universally chosen for variable-length, spatially unaligned condition sequences (complex text prompts, clip captions). Consumes significant FLOPs in deep U-Net blocks. (b) AdaLN-Zero: Extremely parameter-efficient and computationally lightweight. Replaces cross-attention for global attributes (timesteps, camera poses, class labels, aesthetic scores). DiT relies on adaLN for timesteps and cross-attention for text. (c) Concatenation: The only mechanism capable of preserving exact 1-to-1 pixel coordinate alignment for inpainting, scribble-to-image, and depth-to-image workflows. ② ControlNet Dual-Branch Strategy: ControlNet locks the original diffusion weights and creates a trainable copy of the encoder blocks conditioned via zero-convolutions. Spatial conditions are injected via channel addition across intermediate skip connections, preserving pre-trained generative priors. ③ Cross-Attention Key/Value Caching: During generation, text prompt embedding $c_{text{text}}$ is fixed across all 50 diffusion sampling steps. Pre-computing keys $K = c W_K$ and values $V = c W_V$ once and reusing them across all steps saves up to 15% of total inference latency. ④ Interview Strategy: Contrast the three mechanisms across spatial alignment and parameter overhead, formulate adaLN-Zero modulation with zero-initialization gating $alpha$, explain cross-attention key-value caching across timesteps, and justify ControlNet’s zero-convolution design.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 用 concat 注入变长文本条件（长度不匹配）
- ⚠️ 认为一种注入方式适用于所有条件
English Pitfalls:
– Attempting to inject spatially unaligned text prompts via channel concatenation rather than cross-attention
– Re-computing text encoder projections and cross-attention Key/Value matrices at every reverse sampling step
– Omitting zero-initialization in adaLN gating parameters, destabilizing deep DiT Transformer training at initialization
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么 DiT 用 adaLN 而不是 cross-attention？
- Why is zero-initialization of the gating parameter $alpha$ in adaLN-Zero essential for training deep Diffusion Transformers?
- 不同条件的’信息量’如何影响选择？
- How does ControlNet inject pixel-aligned conditions into pre-trained diffusion models without disrupting existing generative weights?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
去噪扩散概率模型 (DDPM)：前向加噪马尔可夫链与变分下界 (ELBO) 推导(DDPM: Forward Markov Noise & ELBO Denoising Derivation) - 🗺️ 知识图谱模块：
多模态与扩散模型导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。