所属模块：
M6 · 多模态与生成模型 (Multimodal & Generative Models)| 专题分类：条件控制与编辑 (Controllable Generation & Image Editing)| 难度等级：Medium
一、核心一句话结论 (One-Sentence Summary)
用掩码标记重绘区域，在扩散中让掩码外保持原图（每步替换已知区域），掩码内按 prompt 生成。
Inpainting and outpainting restore or extend missing image regions by conditioning diffusion trajectories on masked image pixels, inpainting masks, and Fourier-based frequency convolutions.
二、核心考点要义 (Key Insights)
- 📌 掩码 m 标记重绘区域（1=重绘、0=保留）
- 📌 每步用’原图的加噪版本’替换掩码外区域
- 📌 掩码内由 prompt 生成；9 通道输入（图 + 掩码 + 掩码图）
English Insights:
– Channel concatenation conditioning (9-channel inpainting): concatenates noisy latent $z_t$, masked clean latent $z_0 odot (1-M)$, and binary mask $M$ along the channel dimension
– Training-free sampling heuristic (RePaint): overwrites unmasked regions with forward-diffused ground truth noise at every reverse sampling step
– Outpainting geometry: expands canvas boundaries with padding masks, applying directional extrapolation and boundary blending to achieve seamless peripheral continuity
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$x_tleftarrow modot x_t^{text{noisy}}+(1-m)odot x_t^{text{known}};qquad m=text{mask}$$
数学机理：inpainting（局部重绘） 的实现——(1) 掩码——用二值掩码 m 标记’要重绘的区域’（m=1）与’要保留的区域’（m=0）。(2) 条件输入——把 (a) 原图（或其潜变量）、(b) 掩码、(c) 掩码后的图（即原图 ×(1−m)）拼成9 通道输入（Stable Diffusion Inpainting 的做法：4 潜通道 + 1 掩码 + 4 掩码图潜通道）。(3) 每步替换——在采样的每一步，用’原图的加噪版本’替换掩码外区域：x_t←m⊙x_t^{noisy}+(1−m)⊙x_t^{known}，其中 x_t^{known} 是’原图在时间步 t 的加噪版本’（用闭式解从原图加噪得到）。为什么每步替换——(a) 扩散的采样是’逐步去噪’；若只在最后替换，则中间步的’保留区域’仍是噪声，会通过注意力影响重绘区域（导致不一致）；(b) 每步替换使’保留区域’始终是’原图在该噪声水平的版本’，从而 (i) 重绘区域能’看到’正确的上下文、(ii) 边界自然衔接。(4) 掩码的’软边’——实践中常对掩码做羽化（blur）（使边界平滑过渡）；否则会有明显的接缝。(5) 掩码内生成——由文本 prompt 引导（可加 CFG）；掩码内的内容与’保留区域’通过注意力交互，从而实现风格/光照一致。outpainting（画布扩展） 的实现——(a) 扩展画布——把图像放在更大的画布中（新区域待生成）；(b) 掩码——新区域设为’重绘’（m=1）、原图区域设为’保留’（m=0）；(c) 生成——与原图区域在边界处衔接；(d) 迭代扩展——若要扩展到很大，可分块迭代（每次扩展一圈，用上一次的结果作为新的’保留区域’）；(e) 提示——常用’与原图一致的描述’（或让模型从原图推断）。其他方法——(a) 专门的 inpainting 模型（SD Inpainting 版，9 通道）；(b) 用普通模型 + 掩码条件（不需专门模型，但效果稍差）；(c) Blended Diffusion（在潜空间混合）；(d) RePaint（用’重采样’策略）。与’图像编辑’的关系——inpainting 是’局部编辑’的基础（改局部内容）；与 SDEdit/inversion（全局编辑）互补。评估——(a) 边界一致性（接缝是否自然）；(b) 内容质量（掩码内）；(c) 与上下文的一致性（风格/光照/透视）；(d) 人工评估。实践建议——(a) 掩码羽化（避免硬边）；(b) 每步替换（关键）；(c) 合适的重绘区域大小（太大则失去上下文、太小则无意义）；(d) prompt 描述一致（与原图风格匹配）。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Mathematical Mechanism: 1. Dedicated Inpainting Concatenation (9-Channel Model): Standard latent diffusion operates on 4-channel latents $z_t in mathbb{R}^{4 times h times w}$. Dedicated inpainting backbones expand the input convolutional layer to 9 channels: $$z_{text{input}} = big[ z_t ; ; ; z_{text{masked}} ; ; ; M_{text{down}} big] in mathbb{R}^{(4 + 4 + 1) times h times w}$$ where: (a) $z_t$: Noisy latent at timestep $t$ ($4$ channels). (b) $z_{text{masked}} = mathcal{E}(x_0 odot (1 – M))$: Latent of the unmasked context image ($4$ channels). (c) $M_{text{down}}$: Binary mask downscaled to latent resolution ($1$ channel, $M=1$ for inpaint target, $M=0$ for context). The model natively learns to condition generation on unmasked boundary pixels via supervised training on random synthetic masks. 2. Training-Free RePaint Formulation (Lugmayr et al., 2022): Uses an off-the-shelf unconditioned model $epsilon_theta$. At each reverse step $t$: (a) Sample unmasked known region from ground truth: $$x_{t-1}^{text{known}} sim mathcal{N}big( sqrt{bar{alpha}_{t-1}} x_0, ; (1 – bar{alpha}_{t-1}) I big)$$ (b) Predict masked unknown region from model: $$x_{t-1}^{text{unknown}} = frac{1}{sqrt{alpha_t}} left( x_t – frac{beta_t}{sqrt{1-bar{alpha}_t}} epsilon_theta(x_t, t) right) + sigma_t z$$ (c) Composite the intermediate sample: $$x_{t-1} = (1 – M) odot x_{t-1}^{text{known}} + M odot x_{t-1}^{text{unknown}}$$ (d) Resampling / Harmonization Loop: Adds noise back to $x_{t-1}$ to diffuse back to $x_t$, repeating the step $U$ times to harmonize boundary seams.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① ‘每步替换’是 inpainting 正确性的关键——只在最后替换会导致不一致；面试中能解释’为什么每步’是深度理解的标志。② ‘掩码羽化’是消除接缝的实用技巧——硬边会产生明显接缝；羽化使过渡自然。③ ‘9 通道输入’是 SD Inpainting 的设计——把掩码与掩码图作为额外条件；这使模型’知道’哪些要保留。④ ‘outpainting 需迭代’——一次扩展太多会导致结构崩坏；故分块迭代（每次扩展一圈）。⑤ ‘与上下文一致性’是评估重点——掩码内的内容必须与保留区域在风格/光照/透视上一致；否则’看起来是贴上去的’。⑥ 面试要点——被问’inpainting 怎么做’，应给出’掩码 + 9 通道条件 + 每步替换保留区域 + 羽化‘与’outpainting 的迭代扩展‘；能指出’每步替换的必要性’是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
Deep Dive & Engineering Trade-offs: ① Dedicated 9-Channel vs Training-Free RePaint: (a) Training-Free RePaint: Requires zero retraining; works on any pre-trained diffusion model. However, because $x_{t-1}^{text{known}}$ and $x_{t-1}^{text{unknown}}$ originate from independent distributions, the composited state suffers from severe boundary seams and semantic incoherence unless expensive resampling loops ($U=5text{–}10$) are run, increasing inference latency by $5text{–}10times$. (b) Dedicated 9-Channel Inpainting: Generates photorealistic, seamless boundary transitions in a single standard sampling run (20-30 steps). It is the undisputed industry standard for production photo editing tools. ② Latent vs Pixel Masking Artifacts: In Latent Inpainting, because the VAE downsamples images by $8times$, downsampling a sharp binary mask to latent space blurs mask edges across an 8-pixel radius. When decoded, mask boundaries can show subtle feathering or edge discoloration. Blending the final decoded pixel image with original ground truth using a softened pixel mask resolves boundary artifacts. ③ Outpainting as Inpainting Extension: Outpainting places the original image at the center of an expanded canvas, marking the peripheral borders as mask $M=1$. Applying directional prompt conditioning guides seamless background landscape extension. ⑤ Interview Strategy: Formulate the 9-channel concatenation input $[z_t ; z_{text{masked}} ; M]$, contrast dedicated inpainting with training-free RePaint, explain RePaint’s resampling loop for boundary harmonization, and describe post-decode pixel boundary blending.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 只在最后替换保留区域（导致不一致）
- ⚠️ 掩码不做羽化（出现明显接缝）
English Pitfalls:
– Downsampling inpainting masks to latent space without post-decode pixel-space boundary blending, leaving visible 8-pixel seam artifacts
– Using training-free RePaint without resampling loops ($U > 1$), causing severe semantic and lighting mismatches at mask borders
– Attempting outpainting with square aspect-ratio prompts that do not describe the extended peripheral composition
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么每步都要替换（而不是只在最后）？
- Why does the 9-channel dedicated inpainting architecture achieve significantly cleaner boundary transitions than training-free RePaint?
- outpainting 如何实现？
- How does the $8times$ spatial downsampling of VAE encoders introduce boundary feathering challenges in latent inpainting?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
精细条件控制生成：ControlNet 零卷积微调、IP-Adapter 与重绘修复(Controllable Generation: ControlNet Zero-Conv & IP-Adapter) - 🗺️ 知识图谱模块：
多模态与扩散模型导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。