所属模块：
M1 · 数学与统计基础 (Mathematics & Statistics Fundamentals)| 专题分类：数值稳定性 (Numerical Stability)| 难度等级：Hard
一、核心一句话结论 (One-Sentence Summary)
来源：指数溢出、除零、log(0)、梯度爆炸/下溢、FP16 范围、方差过小归一化。
Numerical instability stems from vanishing/exploding gradients, floating-point overflow/underflow, matrix near-singularity, and precision loss; mitigated by normalization, gradient clipping, log-domain computation, and mixed precision design.
二、核心考点要义 (Key Insights)
- 📌 归一化层分母加 ε
- 📌 概率裁剪到 [1e-12, 1-1e-12]
- 📌 监控 grad norm 与 loss scale
- 📌 数值问题常表现为 loss NaN，需逐层定位
English Insights:
– Gradient explosion/vanishing: Mitigated via Residual connections, LayerNorm/RMSNorm, proper initialization (Xavier/Kaiming), and Gradient Clipping.
– Exponent overflow/underflow: Mitigated via Log-Sum-Exp tricks, subtracting max in Softmax, and log1p/expm1 formulations.
– FP16 dynamic range overflow: Mitigated via dynamic loss scaling, FP32 master weights, or upgrading to BF16/FP8 with scaling.
– Matrix division by zero: Mitigated via numerical guards (epsilon clamping: x / (norm + eps)).
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{缓解}: logtext{-sum-exp}, epsilon text{平滑}, text{clipping}, text{BF16}, text{LN/RMSNorm}$$
按算子分类更系统：① 指数类（softmax、sigmoid、exp、log-sum-exp）——溢出/下溢，缓解是减 max、分段计算、log1p；② 除法与归一化（LayerNorm、BatchNorm、attention 的 1/√d）——除零或方差过小，缓解是加 ε（且 FP16 下 ε 需放大到 1e-3 量级）；③ 对数类（交叉熵、KL、log 概率）——log(0)=−inf，缓解是 clip 概率到 [1e-12, 1−1e-12] 或用 log 域直接计算；④ 梯度类——爆炸（连乘 >1）或下溢（连乘 <1 或 FP16 范围），缓解是梯度裁剪、BF16、loss scaling、残差连接；⑤ 累加类（大数求和、注意力 logits）——浮点累加误差，缓解是 Kahan 求和、FP32 累加。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Survey of instability equations: (1) Gradient norm explosion in deep RNNs/Transformers: $g leftarrow g cdot minleft(1, frac{text{clip_norm}}{|g|_2}right)$, projecting gradients back into a bounded hypersphere without changing direction. (2) Epsilon placement in normalization: In RMSNorm, $text{RMS}(x) = sqrt{frac{1}{d}sum_{i=1}^d x_i^2 + epsilon}$, where $epsilon = 10^{-5}$ (FP32) or $10^{-6}$ prevents division by zero when activations collapse. (3) Activation saturation: Sigmoid derivatives $sigma'(z) = sigma(z)(1-sigma(z)) le 0.25$, compounding to $0.25^L to 0$; replaced by non-saturating activations like GELU and SwiGLU.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
系统化的排查与预防：① 定位手段——用 torch.autograd.set_detect_anomaly(True) 捕获产生 NaN 的具体算子；逐层 hook 监控激活与梯度的 min/max/norm；先看是哪一步（step）开始 NaN，再二分定位层。② 结构性的稳定化——残差连接保证梯度至少有恒等通路（+1），归一化层把激活分布拉回稳定尺度，二者共同改善数值条件，这也是深层网络能训练的前提。③ 训练层面的防线——梯度裁剪（clip by norm，LLM 常用 1.0）、BF16 替代 FP16、降低学习率、loss scaling 的动态调整。④ 监控指标——grad norm 分布、update/param 比值（健康范围约 10⁻³）、loss scale 值的变化趋势，这些指标往往在 loss 变 NaN 之前就已异常。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
System design trade-offs: (1) Epsilon magnitude: Setting $epsilon$ too small (e.g. $10^{-12}$ in FP16) results in underflow to zero, causing division by zero anyway; setting it too large (e.g. $10^{-2}$) distorts normalization variance. (2) BF16 vs FP16: BF16 trades 3 mantissa bits for 3 exponent bits, sacrificing precision ($10^{-3}$ relative error) for massive dynamic range ($10^{38}$), proving vastly superior for large model pretraining stability.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 只在 loss 变 NaN 后才排查（应监控 grad norm 提前预警）
- ⚠️ FP16 下沿用 FP32 的 ε 值（导致除零）
English Pitfalls:
– Placing epsilon outside the square root: torch.sqrt(var) + eps vs torch.sqrt(var + eps) (the former allows sqrt(0) whose gradient $frac{1}{2sqrt{0}}$ is inf).
– Clipping gradients element-wise instead of globally by norm, which alters the directional angle of the gradient vector.
六、高频深度面试追问与预测 (Follow-Up Questions)
- 如何定位是哪一层产生 NaN？
- Why is
torch.sqrt(var + eps)mathematically safe in backward autodiff whiletorch.sqrt(var) + epsproduces NaN gradients at zero? - 为什么残差与归一化能改善数值条件？
- How does weight decay decouple from adaptive gradients in AdamW to prevent numerical divergence?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
浮点运算、下溢/上溢、Log-Sum-Exp 稳定算子(Floating-Point, Underflow/Overflow & LogSumExp) - 🗺️ 知识图谱模块：
数理基础思维导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。