所属模块：
M8 · 系统架构、MLOps 与工程实战 (ML Systems, Engineering & Research)| 专题分类：推理服务与部署 (Inference Serving & Deployment)| 难度等级：Easy
一、核心一句话结论 (One-Sentence Summary)
延迟（P50/P99 TTFT/TPOT）、吞吐（QPS/tokens/s）、可用性（SLA）、成本（每千 token）；以及 goodput。
Online inference serving is governed by tail latency percentiles (P95/P99), specialized LLM latency phases (Time To First Token vs. Time Per Output Token), concurrency throughput (QPS and token/s), availability SLAs, and cost-efficiency measured via Goodput—the maximum valid throughput delivered strictly within latency SLOs.
二、核心考点要义 (Key Insights)
- 📌 延迟：P50/P99、TTFT（首 token）、TPOT（每 token）
- 📌 吞吐：QPS / tokens/s；可用性：SLA/错误率
- 📌 成本：每千 token 成本；goodput（SLO 内的有效吞吐）
English Insights:
– Latency percentiles & decomposition: P50 (median) vs. P99 (tail experience); LLM decomposition: TTFT (prefill / compute-bound) vs. TPOT (decode / memory-bound).
– Throughput & availability: Requests per second (QPS), generation tokens per second, error budgets, and SLA/SLO compliance rates.
– Goodput & cost: Goodput defines productive throughput adhering to latency thresholds; unit economics measured via cost per 1,000 output tokens.
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{latency}: text{TTFT},text{TPOT};qquad text{throughput}: text{QPS};qquad text{goodput}=f(text{SLO})$$
数学机理：在线推理服务的核心指标——(1) 延迟（latency）——(a) 分位数——(i) P50（中位数——’典型体验’）；(ii) P99（尾部——’最差体验’，最关键，因为用户体验由尾部决定）；(iii) 均值不可靠（被长尾拉高但掩盖分布）；(b) LLM 特有——(i) TTFT（Time To First Token）——首 token 延迟（由 prefill 决定，∝ prompt 长度）；(ii) TPOT（Time Per Output Token）——每输出 token 延迟（由 decode 决定，∝ KV 读取）；(iii) 总延迟 ≈ TTFT + TPOT × 输出长度；(c) 端到端延迟（含排队、网络、预处理）。(2) 吞吐（throughput）——(a) QPS（每秒请求数）；(b) tokens/s（每秒 token 数——LLM 更常用）；(c) 与延迟的权衡——batch 越大吞吐越高但延迟越高（见 M7 的吞吐-延迟权衡题）。(3) 可用性（availability）——(a) SLA（如 99.9% 可用）；(b) 错误率（5xx/超时）；(c) 降级率（走了降级路径的比例）。(4) 成本（cost）——(a) 每千 token 成本；(b) 每请求成本；(c) GPU 利用率（利用率低则单位成本高）；(d) ‘成本-延迟-质量’三方权衡。(5) goodput——(a) 定义——在满足 SLO（延迟/错误率约束）的前提下能达到的有效吞吐；(b) 为什么重要——’最大吞吐’可能违反 SLO（延迟太高）；故 goodput 才是服务的真实目标；(c) 优化目标——max goodput s.t. 成本约束。(6) 其他——(a) 排队时间（与负载相关）；(b) 冷启动时间（新实例启动）；(c) 缓存命中率（前缀缓存/语义缓存）；(d) 批大小分布。监控实践——(a) 分位数而非均值（P50/P95/P99）；(b) TTFT/TPOT 分开监控（定位是 prefill 还是 decode 的问题）；(c) 按输入长度/输出长度分层（长 prompt 的延迟天然高）；(d) 与 SLO 对比（达标率）；(e) 成本归因（按请求/用户/场景）。与其他问题的关系——(a) 与’成本与延迟优化’（同一主题）；(b) 与’自动扩缩容’（延迟驱动扩缩容）；(c) 与’降级’（SLO 保障）。实践建议——(a) 监控 P99（而非均值）；(b) TTFT/TPOT 分开；(c) 按长度分层（公平对比）；(d) 定义 SLO 并监控达标率；(e) 优化 goodput（而非最大吞吐）；(f) 成本归因。度量——(a) P50/P99 的 TTFT/TPOT；(b) QPS/tokens per s；(c) SLA 达标率；(d) 每千 token 成本；(e) goodput。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Metrics Formalism & LLM Latency Breakdown:
(1) Latency Distribution & Percentiles:
– Average latency is deeply misleading because web traffic exhibits heavy-tailed, multi-modal distributions.
– P99 Latency: The 99th percentile represents the tail user experience; in microservice cascades, a single high-P99 dependency degrades the entire end-to-end response chain.
(2) LLM Latency Decomposition:
Total end-to-end inference latency is decomposed into two fundamentally distinct computational regimes:
$$T_{text{total}} = T_{text{queue}} + text{TTFT} + text{TPOT} times (N_{text{tokens}} – 1)$$
– Time To First Token (TTFT):
– Measures duration from request arrival to generation of the first token.
– Dominated by the Prefill Phase: processes the entire prompt sequence of length $S_{text{in}}$ in parallel.
– Compute-bound: high arithmetic intensity, saturating GPU Tensor Cores ($O(S_{text{in}}^2)$ attention + GEMM).
– Time Per Output Token (TPOT):
– Measures inter-token generation latency during autoregressive decoding.
– Dominated by the Decode Phase: generates tokens sequentially one by one.
– Memory-bandwidth-bound: low arithmetic intensity; every token requires reading all model weights and KV cache tensors from GPU HBM ($O(1)$ compute per byte transferred).
(3) Throughput Metrics:
– QPS (Queries Per Second): Traditional stateless service capacity metric.
– Tokens / Second: Aggregated generation throughput: $text{TPS} = sum_{i=1}^B frac{text{Tokens}_i}{Delta t}$.
(4) Goodput vs. Raw Throughput:
– Raw Throughput: Maximized by packing huge batches ($B=256$), which achieves high token throughput but inflates latency to dozens of seconds, violating user SLAs.
– Goodput Formalism:
$$text{Goodput} = sum_{i=1}^M mathbb{I}(text{Latency}_i le text{SLO}_{text{lat}} land text{Status}_i = 200) times text{Throughput}_i$$
Goodput measures the effective business throughput that genuinely satisfies contractual Service Level Objectives.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① ‘必须看 P99’——用户体验由尾部决定；面试中能指出是深度理解的标志。② ‘TTFT/TPOT 分开监控’——定位是 prefill 还是 decode 的问题（两者的优化手段完全不同）。③ ‘goodput 而非最大吞吐’——服务目标是’在 SLO 内最大化吞吐’。④ ‘按长度分层’——长 prompt 的 TTFT 天然高；不分层会误判。⑤ ‘均值掩盖问题’——必须用分位数。⑥ 面试要点——被问’推理服务的核心指标’，应给出’延迟（P50/P99 + TTFT/TPOT）+ 吞吐（QPS/tokens per s）+ 可用性（SLA）+ 成本（每千 token）+ goodput‘；能指出’TTFT/TPOT 分开’与’goodput’是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
In-Depth Analysis & Engineering Trade-offs: ① P99 over average latency—in multi-tenant architectures, averaging conceals queue starvation and tail latency spikes experienced by valuable users. ② TTFT and TPOT require separate optimization paths—improving TTFT requires compute scaling, FlashAttention, and chunked prefill; improving TPOT requires memory bandwidth optimizations, tensor parallelism, speculative decoding, and FP8 KV caching. ③ Batch size: Throughput vs. Latency trade-off—increasing batch size amortizes model weight loading across queries (improving tokens/sec and lowering GPU cost), but linearly increases individual token generation latency (degrading TPOT); dynamic continuous batching balances this frontier. ④ Length-stratified benchmarking—evaluating LLM serving performance without segmenting by prompt and generation length produces garbage metrics; long-context inputs (32k tokens) naturally inflate TTFT by orders of magnitude compared to conversational 100-token queries. ⑤ Cost per million tokens as the ultimate unit economic—cloud serving profitability is governed by maximizing Goodput per dollar of GPU hardware spend. ⑥ Interview takeaway—break down the latency equation into TTFT and TPOT, map them to prefill (compute-bound) vs. decode (memory-bound) phases, and contrast raw throughput with SLA-constrained Goodput.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 只看平均延迟（掩盖尾部）
- ⚠️ 不区分 TTFT 与 TPOT（无法定位瓶颈）
English Pitfalls:
– Evaluating serving systems using average latency instead of P95/P99 percentiles, completely hiding tail latency degradation.
– Conflating TTFT and TPOT into a single aggregate latency metric, making it impossible to identify whether prefill or decode is the bottleneck.
– Optimizing solely for peak raw tokens/second by saturating batch sizes, resulting in catastrophic latency SLO violations and zero actual Goodput.
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么必须看 P99 而非均值？
- How does Chunked Prefill prevent massive prefill requests from starving active autoregressive decode iterations in continuous batching engines?
- TTFT 与 TPOT 分别由什么决定？
- Why is the autoregressive decode phase strictly memory-bandwidth bound rather than compute bound on modern GPUs?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
大模型高并发推理服务架构：连续批处理 (Continuous Batching) 与 Triton(High-Concurrency Serving: Continuous Batching, vLLM & Triton) - 🗺️ 知识图谱模块：
AI 基础设施工程导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。