所属模块：
M8 · 系统架构、MLOps 与工程实战 (ML Systems, Engineering & Research)| 专题分类：研究能力：实验设计与消融 (Research: Experiment Design & Ablations)| 难度等级：Medium
一、核心一句话结论 (One-Sentence Summary)
先确定最小可检测效应、显著性水平与目标功效，再反推所需的样本量或随机种子数，避免功效不足导致假阴性或过度实验浪费资源。
Statistical power analysis predetermines the required sample size—evaluating minimum detectable effect (MDE), significance level ($alpha$), target power ($1-beta$), and metric variance—to guarantee that true performance improvements are reliably detected without underpowered false negatives or overpowered computational waste.
二、核心考点要义 (Key Insights)
- 📌 三个输入——最小可检测效应 δ、显著性水平 α、目标功效 1-β
- 📌 功效（power）——真实存在差异时能检测到的概率，常用 0.8
- 📌 样本量——n 随 δ 减小而增大、随 σ 增大而增大
- 📌 在 ML 中——’样本’常为随机种子数/评估样本数/用户数（A/B 实验）
- 📌 预注册——实验前定好协议与假设，避免事后调参
English Insights:
– Core mathematical parameters: Significance level $alpha$ (false positive rate, typically 0.05), Statistical Power $1-beta$ (true positive rate, typically 0.80), Minimum Detectable Effect $delta$ (MDE), and metric variance $sigma^2$.
– Sample size formulation: Required sample size scales inversely with the square of the effect size ($n propto frac{sigma^2}{delta^2}$)—detecting a $2times$ smaller improvement requires $4times$ more samples or seeds.
– Multi-context application: Sizing random seeds in offline deep learning benchmarks, determining test set size for evaluation benchmarks, and traffic allocation in online A/B testing.
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$napprox frac{2sigma^{2}(z_{1-alpha/2}+z_{1-beta})^{2}}{delta^{2}}$$
数学机理：统计功效分析（power analysis）——(1) 四个相关量——(a) 效应量 δ——希望检测到的最小差异（最小可检测效应 MDE）；(b) 显著性水平 α——假阳性率（通常 0.05）；(c) 功效 1-β——真阳性率（检测到真实差异的概率，常取 0.8）；(d) 样本量 n——四个量中给定三个可求第四个。(2) 样本量公式（两样本均值比较）——(a) n ≈ 2σ²(z_{1-α/2} + z_{1-β})² / δ²；(b) 含义——n 与 σ² 成正比（噪声大需更多样本）、与 δ² 成反比（想检测更小的差异需更多样本）；(c) z 分位数来自标准正态。(3) 在 ML 实验中的应用——(a) ‘样本’的多种含义——(i) 随机种子数——多种子平均降低方差；(ii) 评估集大小——测试样本越多，指标估计越稳；(iii) A/B 实验的用户数——在线实验的样本量；(b) 种子数——若种子间标准差为 s，想检测差异 δ，则种子数 n ≈ 2s²(z+z)²/δ²。(4) 功效不足的后果——(a) 假阴性——真实存在差异却检测不到（结论’无差异’是错的）；(b) 浪费——实验做了但结论不可靠；(c) 对策——实验前做功效分析，确保 n 足够。(5) 功效过高的代价——(a) 样本过多 → 检测到无实际意义的微小差异（统计显著但效应量小）；(b) 故——同时设定 MDE（最小可检测效应），只关心有实际意义的差异。(6) A/B 实验的样本量——(a) 公式扩展——涉及基线转化率、MDE、α、β、分流比例；(b) 实践——用在线计算器/工具；(c) 注意——方差估计、多重比较、新颖效应（novelty effect）会影响实际所需样本。(7) 预注册（pre-registration）——(a) 做法——实验前写下假设、指标、样本量、分析计划；(b) 作用——防止事后挑选有利结果（HARKing）与 p-hacking；(c) 在 ML——定好协议与种子数。(8) 实践建议——(a) 先估方差——用少量种子估计 s；(b) 定 MDE——基于实际意义（如提升 1% 才值得上线）；(c) 算 n——用公式或工具；(d) 记录——在论文/报告中说明功效分析。与其他问题的关系——(a) 与显著性检验；(b) 与方差处理（σ 的来源）；(c) 与在线实验设计（样本量）。度量——(a) 功效（1-β）；(b) MDE；(c) 实际种子数/样本量 vs 所需。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Mathematical Formulations & Sample Size Derivations:
(1) The 4 Coupled Statistical Quantities:
– 1. Significance Level ($alpha$): Probability of Type I error (rejecting $H_0$ when $H_0$ is true), typically $alpha = 0.05$ ($z_{1-alpha/2} approx 1.96$).
– 2. Statistical Power ($1-beta$): Probability of Type II error avoidance (correctly rejecting $H_0$ when $H_1$ is true), typically $1-beta = 0.80$ ($z_{1-beta} approx 0.84$).
– 3. Minimum Detectable Effect ($delta$ or MDE): The smallest absolute performance delta that is commercially or scientifically meaningful to detect.
– 4. Sample Size ($n$): Number of evaluation instances, random seeds, or A/B experiment users.
(2) Analytical Two-Sample Sample Size Derivation:
Comparing two independent groups with common variance $sigma^2$ under $H_0: mu_A = mu_B$ vs $H_1: mu_B – mu_A = delta$:
– Standard error of the difference: $text{SE} = sqrt{frac{sigma^2}{n} + frac{sigma^2}{n}} = sigma sqrt{frac{2}{n}}$.
– Equating critical boundaries under null and alternative distributions:
$$delta = (z_{1-alpha/2} + z_{1-beta}) cdot sigma sqrt{frac{2}{n}}$$
– Solving for $n$ per group:
$$n approx frac{2 sigma^2 (z_{1-alpha/2} + z_{1-beta})^2}{delta^2}$$
For standard parameters $alpha = 0.05, 1-beta = 0.80$, $(1.96 + 0.84)^2 = 7.84$, yielding:
$$n approx frac{15.7 cdot sigma^2}{delta^2}$$
(3) Application to Offline ML Evaluation:
– When estimating metric difference $bar{D}$ with inter-seed variance $s_D^2$, the required number of random seeds $N_{text{seeds}}$ to detect gain $delta$ is:
$$N_{text{seeds}} approx frac{s_D^2 (z_{1-alpha/2} + z_{1-beta})^2}{delta^2}$$
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① 功效不足会导致假阴性——’无差异’的结论可能是样本不够；面试中能指出这点是深度理解的标志。② n 与 δ² 成反比——检测更小差异需更多样本（平方关系）。③ n 与 σ² 成正比——噪声大需更多样本。④ 同时设 MDE——避免样本过多检测到无意义的微小差异。⑤ ML 中’样本’可为种子数或评估样本数——需明确。⑥ 预注册防 HARKing——实验前定协议。⑦ 面试要点——被问怎么定实验规模，应给出’确定 MDE、α、功效 → 估计方差 σ → 用公式算 n（种子数/评估样本/用户数）→ 预注册协议‘；能指出功效不足导致假阴性与 n∝1/δ² 是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
In-Depth Analysis & Engineering Trade-offs: ① Underpowered experiments produce disastrous false negatives—if an experiment is run with insufficient sample size or seeds, failing to reject the null hypothesis does not prove ‘no difference’; it simply means the experiment was blind to the improvement, causing teams to discard valuable algorithmic innovations. ② The quadratic penalty of small MDE ($n propto 1/delta^2$)—halving the effect size you want to detect requires quadrupling the sample size or experiment duration; product teams must establish a realistic threshold of practical significance rather than chasing infinitesimal gains. ③ Overpowered studies detect meaningless micro-effects—in massive A/B tests with $n = 10,000,000$ users, a $+0.001%$ difference will be statistically significant ($p < 0.001$), but has zero practical business value; pre-defining the MDE guarantees focus on impactful changes. ④ Variance reduction techniques (CUPED) amplify effective sample size—in online experimentation, using pre-experiment covariates to reduce metric variance $sigma^2$ cuts required sample size by $(1 – rho^2)$, allowing faster iterations. ⑤ Pre-registration locks the experimental contract—documenting MDE, sample size, and analysis plans prior to running tests prevents p-hacking, early stopping biases, and HARKing (Hypothesizing After Results are Known). ⑥ Interview takeaway—write out the sample size formula, explain the inverse-square relationship with MDE, describe how underpowered studies cause false negatives, and connect the concept to both offline seed counts and online A/B testing.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 不做功效分析就下’无差异’结论
- ⚠️ 只追统计显著而不设 MDE（检测到无意义差异）
English Pitfalls:
– Concluding that two models have ‘equivalent performance’ based on a statistically underpowered experiment with only 2 or 3 random seeds.
– Chasing miniscule MDEs in online experiments without considering the quadratic explosion in traffic volume and experimentation duration.
– Stopping an experiment early the moment $p < 0.05$ is observed, severely inflating false positive rates via continuous monitoring bias.
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么功效不足会导致假阴性？
- How does Controlled-experiment Using Pre-Experiment Data (CUPED) reduce metric variance and lower required sample size in A/B tests?
- 效应量越小为什么需要越多样本？
- How do sequential testing frameworks (such as mSPRT) enable continuous experiment monitoring without inflating Type I error rates?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
科学实验设计准则：严谨多随机种子消融、负结果分析与统计显著性验证(Rigorous Experiment Design: Multi-Seed Ablations & Significance) - 🗺️ 知识图谱模块：
算法研究科学家推导与实验导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。