所属模块：
M2 · 经典机器学习 (Classical Machine Learning)| 专题分类：正则化 (Regularization (L1 / L2))| 难度等级：Medium
一、核心一句话结论 (One-Sentence Summary)
① 集成视角：等价于指数级子网络的 bagging；② 破坏共适应，迫使特征独立有用。
Dropout acts as an implicit ensemble of $2^B$ sub-networks sharing weights, while simultaneously functioning as data-dependent adaptive feature noise that prevents co-adaptation among neurons.
二、核心考点要义 (Key Insights)
- 📌 推理期需缩放（inverted dropout）
- 📌 与 BN 同用需注意方差偏移
English Insights:
– 1. Ensemble Interpretation: During training, randomly dropping units with probability $p$ samples one of $2^B$ thinned subnetworks; at test time, multiplying weights by $(1-p)$ approximates the geometric ensemble average.
– 2. Anti-Co-adaptation Interpretation: Neurons cannot rely on specific companion neurons to correct errors, forcing every unit to learn independently useful, robust internal representations.
– 3. Connection to L2: In linear models, Dropout is mathematically equivalent to adaptive L2 weight decay.
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{train}: h’=frac{modot h}{1-p},qquad text{eval}: h’=h$$
两种解释：① 集成视角（Srivastava et al. 2014）——每次前向随机丢弃一部分神经元，相当于训练了 2ⁿ 个共享权重的子网络；推理时用全部神经元并缩放，相当于对这些子网络的几何平均做集成。集成降方差，故有正则效果。② 共适应破坏视角——神经元不能依赖特定同伴的存在（因为同伴可能被丢弃），被迫学习独立有用的特征。这抑制了’特征间互相补偿’的脆弱依赖，提升鲁棒性。Inverted dropout 的实现细节：训练时对保留的激活除以 (1−p)（放大），推理时恒等——这样训练与推理的期望一致，避免推理期额外缩放。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Proof of equivalence between Dropout and L2 weight decay (Wager et al., 2013): Consider linear regression $y = w^T x$. Apply dropout by multiplying feature $x_i$ by Bernoulli mask $m_i sim text{Bernoulli}(1-p)$. The objective is $min_w E_mleft[left(y – sum_{i=1}^d frac{m_i}{1-p} w_i x_iright)^2right]$. Decomposing via the Law of Total Variance: $E_m[|y – tilde{X}w|^2] = |y – Xw|^2 + sum_{i=1}^d text{Var}(m_i) frac{w_i^2 x_i^2}{(1-p)^2}$. Since $text{Var}(m_i) = p(1-p)$, the second term simplifies to: $frac{p}{1-p}sum_{i=1}^d x_i^2 w_i^2 = w^T text{diag}left(frac{p}{1-p} X^T Xright) w$. This is an adaptive L2 regularization where features with higher variance are regularized more aggressively!
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
关键权衡与坑：① p 的选择——输入层通常 p=0.1–0.2，隐藏层 0.3–0.5；p 过大导致欠拟合与训练不稳，p 过小无效果。② 与 BatchNorm 的冲突——BN 在训练时用 batch 统计、推理时用 running 统计，而 dropout 改变激活的方差，导致训练/推理的统计不一致（方差偏移，variance shift）；常见解法是把 dropout 放在残差分支上（而非主路径）、或用 LN/GN 替代 BN、或降低 p。③ 现代趋势——大模型预训练常用 dropout=0（数据量大时正则需求低），而微调时用 0.1；Transformer 中 dropout 常用于注意力权重与残差连接处。④ 与 weight decay 的关系——两者机制不同（dropout 作用于激活、weight decay 作用于参数），可叠加；但 dropout 的隐式正则已较强时，需相应减小 weight decay。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
Industrial practices: (1) Inverted Dropout: Multiplies activations by $frac{1}{1-p}$ during training: $tilde{h} = frac{m odot h}{1-p}$, allowing test-time inference to run unmodified without weight scaling. (2) In modern LLM Transformers, Dropout is often disabled ($p=0$) during large-scale pretraining because massive datasets already provide natural regularization and dropout slows compute throughput.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 推理时忘记关闭 dropout（输出随机、期望被放大）
- ⚠️ dropout 与 BN 直接叠加而不处理方差偏移
English Pitfalls:
– Forgetting to call model.eval() at inference time (leaving dropout enabled, injecting stochastic noise into predictions).
– Using standard dropout alongside Batch Normalization without careful positioning (variance shift can destabilize training).
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么推理时不能开 dropout？
- Why does Inverted Dropout eliminate the need for scaling adjustments during model evaluation?
- Dropout 与 weight decay 的关系？
- Why has modern LLM pretraining largely abandoned Dropout in favor of weight decay and data scale?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
L1 Lasso 与 L2 Ridge 正则化几何与拉普拉斯/高斯先验(L1 Lasso & L2 Ridge Regularization Geometry & Priors) - 🗺️ 知识图谱模块：
经典机器学习思维导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。