所属模块：
M8 · 系统架构、MLOps 与工程实战 (ML Systems, Engineering & Research)| 专题分类：成本与延迟优化 (Cost & Latency Optimization)| 难度等级：Medium
一、核心一句话结论 (One-Sentence Summary)
增大 batch 提高吞吐（摊薄权重读取与 kernel 启动开销）但增加单请求排队与首 token 延迟；连续批处理在 token 级动态组批，兼顾吞吐与延迟。
Static batching increases throughput by amortizing memory-bound model weight transfers across requests at the expense of head-of-line queue blocking, whereas continuous (iteration-level) batching dynamically injects and evicts sequences at every decoding iteration, maximizing GPU saturation while slashing tail latency.
二、核心考点要义 (Key Insights)
- 📌 静态批处理——固定 batch 内所有请求等最慢者完成，短请求被长请求拖累（队头阻塞）
- 📌 连续批处理——每个解码步动态移除已完成序列、加入新序列，显著降低平均延迟
- 📌 吞吐-延迟权衡——batch 越大吞吐越高但首 token 延迟（TTFT）与 TPOT 上升
- 📌 显存约束——batch 受 KV cache 显存限制，需分页（PagedAttention）与调度
- 📌 调度策略——先到先服务、最短作业优先、按 SLO 分层调度
English Insights:
– Static batching trade-off: Amortizes memory weight loading over $B$ requests, but forces fast, short sequences to remain trapped waiting for the longest sequence (head-of-line blocking).
– Continuous iteration-level batching: Evicts completed sequences and schedules newly arrived requests dynamically at every single autoregressive step.
– Hardware synergy: Continuous batching strictly requires non-contiguous virtual memory management (PagedAttention) to prevent physical VRAM fragmentation.
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$T_{text{step}}approx T_{text{mem}}+B,T_{text{flop}};qquad text{throughput}propto B, text{latency}propto B$$
数学机理：批处理的收益与代价——(1) 为何批处理提升吞吐——(a) 权重读取摊薄——decode 每步需读取全部权重（显存带宽受限），batch=B 时一次读取服务 B 个请求 → 单位请求的带宽成本降为 1/B；(b) kernel 启动摊薄——GPU kernel 启动有固定开销，批处理摊薄；(c) 算力利用率——batch 大时矩阵乘规模增大，算术强度提高，更接近算力上限（MFU 提升）。(2) 为何批处理增加延迟——(a) 排队——等待凑批与等待前序请求；(b) 单步变慢——T_step ≈ T_mem + B·T_flop，batch 越大每步耗时越长；(c) 故 TTFT 与 TPOT 随 B 上升。(3) 静态批处理的问题——(a) 队头阻塞（head-of-line blocking）——固定 batch 需等所有序列结束才释放，短请求被最长请求拖累；(b) 利用率低——序列陆续结束时 GPU 空闲（’尾部空洞’）。(4) 连续批处理（continuous batching / iteration-level scheduling）——(a) 机制——在每个解码步检查哪些序列已完成（遇 EOS 或达长度），立即移除并释放 KV，同时从队列加入新请求；(b) 效果——(i) 消除队头阻塞（短请求尽早返回）；(ii) 提高 GPU 利用率（步内始终接近满批）；(iii) 吞吐与延迟同时改善；(c) 前提——需要分页 KV cache（PagedAttention）以支持 KV 的非连续分配与释放；(d) 调度——需在步内决定’选哪些请求组成下一批’（FCFS / 最短作业优先 / 按 SLO 优先级）。(5) 显存约束——(a) batch 上限由 KV cache 显存决定（KV 随 batch 与长度线性增长）；(b) 分页——把 KV 切成固定大小的块，按需分配，减少碎片、支持抢占与共享前缀；(c) 抢占（preemption）——显存不足时换出低优先级序列的 KV。(6) 调度目标——(a) 吞吐最优——尽量满批；(b) 延迟 SLO——按 TTFT/TPOT 目标分配；(c) 公平/优先级——分层（付费/免费）。(7) 相关技术——(a) chunked prefill——把长 prefill 切块与 decode 混批，避免长 prefill 阻塞 decode；(b) PD 分离——prefill 与 decode 部署在不同实例，各自优化。与其他问题的关系——(a) 与推理成本（吞吐即成本）；(b) 与延迟分解（TTFT/TPOT）；(c) 与 KV cache 管理。度量——(a) 吞吐（token/s）；(b) TTFT / TPOT / 端到端延迟分位数；(c) GPU 利用率（MFU）；(d) 平均 batch 大小。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Batching Mechanics & Execution Scheduling:
(1) Why Batching Maximizes Throughput (Amortizing Memory Transfers):
– In autoregressive LLM decoding, generating a token requires reading all $N$ model parameters from high-bandwidth memory (HBM) to on-chip SRAM.
– For batch size $B=1$, memory transfer cost is $2N$ bytes for 1 token generated $implies$ severe memory-bandwidth saturation.
– For batch size $B$, reading the $N$ weights once serves $B$ concurrent tokens. Step execution duration scales as:
$$T_{text{step}}(B) approx T_{text{memory}} + B cdot T_{text{compute}}$$
Because $T_{text{memory}} gg B cdot T_{text{compute}}$ for moderate $B$, per-token generation cost drops by nearly $frac{1}{B}$, drastically boosting aggregate token throughput (Tokens/sec).
(2) Why Static Batching Destroys Latency (Head-of-Line Blocking):
– The Flaw: An entire batch of $B$ sequences must run until the single longest sequence finishes generation ($T_{max} = max_{i=1}^B L_i$).
– Tail Bubbles: If Sequence 1 finishes at iteration 10 while Sequence 2 finishes at iteration 500, Sequence 1’s memory sits locked and GPU compute slots run padded with zeros (idle waste) for 490 steps.
– Queuing Delay: Newly arriving queries must sit waiting in the queue until the entire legacy batch terminates.
(3) Continuous / Iteration-Level Batching (Orca / vLLM):
– Instead of scheduling at the request level, scheduling executes at the iteration step level:
– At step $t$, the engine checks which sequences emitted an end-of-sequence token ([EOS]) or reached max_tokens.
– Completed sequences are evicted immediately; their memory is reclaimed, and responses are returned to users.
– Newly arrived waiting requests are dynamically injected into the active batch slots.
– Result: Eliminates head-of-line blocking, maintains GPU utilization at near 100%, and cuts median user latency by $2-4times$.
(4) Chunked Prefill & Decode Colocation:
– A massive incoming prompt (e.g., 4096 tokens) monopolizes GPU compute during its prefill phase, stalling all active autoregressive decode sequences and causing massive TPOT latency spikes.
– Chunked Prefill partitions long prefills into chunks (e.g., 512 tokens), co-scheduling one prefill chunk alongside active decode iterations in the same batch step to maintain smooth inter-token latency SLAs.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① 批处理是吞吐与延迟的权衡旋钮——面试中能给出 T_step ≈ T_mem + B·T_flop 是深度理解的标志。② 静态批处理的队头阻塞是真实痛点——连续批处理是当前主流（vLLM/TGI 等）。③ 连续批处理依赖分页 KV——没有 PagedAttention 难以动态分配与释放。④ chunked prefill 解决长 prefill 阻塞——长输入会打断 decode 的稳定 TPOT。⑤ PD 分离是前沿实践——prefill 与 decode 的瓶颈不同，分离部署可各自优化。⑥ 显存是硬约束——batch 上限由 KV 显存决定，量化与 GQA 可放宽。⑦ 面试要点——被问怎么同时优化吞吐与延迟，应给出’连续批处理 + 分页 KV + chunked prefill + 按 SLO 调度‘，并指出’静态批处理有队头阻塞’；能给出 T_step 分解是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
In-Depth Analysis & Engineering Trade-offs: ① Continuous batching is impossible without PagedAttention—dynamic insertion and eviction of sequences causes severe physical memory fragmentation; without virtual memory paging, servers run out of memory despite having 50% free fragmented VRAM. ② Throughput vs. Latency Pareto knob—increasing max batch size from 16 to 128 maximizes token throughput and lowers dollar cost per token, but increases per-token generation time (TPOT) by 2-3x; platforms tune batch size strictly against contractual latency SLOs. ③ Chunked prefill preserves decode TPOT stability—un-chunked prefill requests inject 100-300ms delays into active streaming chats; chunked prefill caps prefill work per step, keeping decode iterations running like clockwork. ④ Prefill-Decode (PD) disaggregation as the ultimate separation—because prefill is compute-bound and decode is memory-bound, cutting-edge architectures deploy separate dedicated prefill GPU nodes and decode GPU nodes, transferring KV cache states over high-speed networks. ⑤ Scheduling policy trade-offs—First-Come First-Served (FCFS) ensures fairness, but Shortest-Job-First (SJF) minimizes average queuing latency across the cluster. ⑥ Interview takeaway—write out the $T_{text{step}} approx T_{text{mem}} + B cdot T_{text{comp}}$ equation, explain how static batching creates tail bubbles and head-of-line blocking, detail how continuous batching swaps sequences at every iteration step, and highlight Chunked Prefill.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 用静态批处理处理长短混合请求（队头阻塞）
- ⚠️ 只追吞吐不考虑 TTFT/TPOT 的 SLO
English Pitfalls:
– Using static request-level batching for serving mixtures of short and long prompt requests, causing extreme GPU compute idling on tail bubbles.
– Attempting continuous batching without PagedAttention, triggering fatal memory fragmentation and frequent OOM panics.
– Allowing large un-chunked prefill requests to monopolize batch steps, severely degrading active streaming TPOT for existing users.
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么静态批处理会有队头阻塞？
- How does PagedAttention allocate virtual memory table blocks to prevent physical GPU memory fragmentation during continuous batching?
- 连续批处理如何与分页 KV cache 配合？
- How does Chunked Prefill mathematically balance prompt processing throughput against autoregressive decode latency guarantees?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
端到端推理优化：TTFT 首字延迟、TPOT 吞吐优化与 GPU 算力成本核算(Latency & Cost Optimization: TTFT, TPOT & GPU Economics) - 🗺️ 知识图谱模块：
AI 基础设施工程导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。