所属模块：
M4 · 序列与 Transformer (Sequences & Transformers)| 专题分类：KV Cache 与推理优化 (KV Cache & Inference Optimizations)| 难度等级：Hard
一、核心一句话结论 (One-Sentence Summary)
把长 prompt 的 prefill 切成小块，与 decode 请求混在同一批中，兼顾 TTFT 与吞吐（算力与带宽互补）。
Chunked prefill splits lengthy prompt computations into fixed-size chunks and interleaves them with ongoing decode steps within the same batch iteration, preventing long prefills from causing decode latency spikes.
二、核心考点要义 (Key Insights)
- 📌 长 prompt 分块处理，避免阻塞 decode 请求
- 📌 prefill（算力密集）与 decode（带宽密集）混批互补
- 📌 是 Sarathi-Serve / vLLM 等引擎的关键调度技术
English Insights:
– Problem: Long prefills monopolize GPU compute for hundreds of milliseconds, stalling concurrent decode steps and causing severe Time-Per-Output-Token (TPOT) tail latency jitter
– Mechanism: Chunked prefill (Sarathi-Serve) slices long prompts into chunks of size $C$ (e.g., 512 tokens), co-scheduling one chunk with $B$ decode tokens in a single execution step
– Benefits: Normalizes execution latency per iteration, eliminates scheduling starvation, and improves GPU compute utilization via hybrid GEMM-GEMV batching
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{chunked}: text{prefill split into }c text{chunks};qquad text{mix with decode}Rightarrowtext{utilize both resources}$$
数学机理：问题——在连续批处理下，新请求的 prefill（处理长 prompt，长度可能数千 token）会独占算力，使正在进行的 decode 请求被阻塞、延迟抖动（decode 请求的 TPOT 变差）。同时，prefill 是 compute-bound（算力密集、访存少），而 decode 是 memory-bound（访存密集、算力空闲）——两者的资源需求互补，若分开执行则各自浪费一半资源。chunked prefill 的解法：把长 prompt 的 prefill 切成固定大小的小块（chunk，如 512 token），每次只处理一块，并把该块与同批中的 decode 请求一起执行。收益：(a) 不阻塞 decode——每个 chunk 的计算量受控，decode 请求的延迟抖动降低；(b) 资源互补——prefill 块提供算力密集的工作（填满张量核心）、decode 提供访存密集的工作（填满带宽），混批使 GPU 的两类资源都被利用；(c) TTFT 可控——chunk 大小决定’一个长 prompt 需要多少步完成 prefill’，从而控制 TTFT 与吞吐的权衡。与 Sarathi-Serve 的关系——该工作（Agrawal 等 2024）系统化了’chunked prefill + piggybacking（把 prefill 块’搭载’在 decode 批上）’，并证明其能同时改善吞吐与延迟抖动（称为’stall-free batching’）。chunk 大小的权衡——chunk 越大则 prefill 越快完成（TTFT 低）但单步延迟抖动大；chunk 越小则抖动小但 TTFT 高（需更多步）。实践中按目标 SLO 选择。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Mathematical Mechanism: Latency Jitter in Naive Continuous Batching: When a long prompt of length $L = 4096$ arrives, executing its prefill in one step takes $T_{text{prefill}} gg T_{text{decode}}$ (e.g., $150text{ ms}$ vs $15text{ ms}$). All ongoing decode streams pause for $150text{ ms}$, creating an unacceptable $10times$ spike in P99 TPOT. Chunked Prefill Formulation: The prompt is partitioned into $M = lceil L / C rceil$ chunks: $P = [C_1, C_2, dots, C_M]$. At step $k$, chunk $C_k$ is executed. The attention computation for chunk $C_k$ attends to: $$Q_{C_k} in mathbb{R}^{C times d}, quad K = [K_{text{cached}(1:k-1)}; K_{C_k}], quad V = [V_{text{cached}(1:k-1)}; V_{C_k}]$$ This is causal cross-attention against historical KV cache plus causal self-attention within $C_k$. Hybrid Batch Scheduling: In each iteration, the scheduler constructs a batch containing $N_{text{tokens}} = C + B_{text{decode}}$, where $C$ is the prefill chunk size and $B_{text{decode}}$ is the number of active decode tokens. The unified tensor $X in mathbb{R}^{(C + B_{text{decode}}) times d}$ is processed in a single forward pass, keeping per-iteration execution time tightly bounded.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① ‘算力与带宽互补’是核心洞察——这是理解 chunked prefill 的钥匙；在 roofline 视角下，prefill 位于 compute-bound 区、decode 位于 memory-bound 区，混批使工作点更接近’屋顶’的拐点（两者资源都被利用）。② 与 PD 分离的对比——chunked prefill 是’同实例混批‘（简单、无 KV 传输开销）；PD 分离是’异实例分离‘（各阶段可独立优化与扩缩容，但有 KV 传输开销）。两者是同一目标（消除两阶段互相干扰）的两种实现，可结合（如 PD 分离 + 各自的 chunked 调度）。③ 调度复杂度——需在每步决定’哪些 prefill 块与哪些 decode 请求组成批’，且需保证 (a) 不超出显存（KV 与激活）、(b) 满足 TTFT/TPOT 的 SLO；这是推理引擎调度器的核心算法。④ 与投机解码的交互——投机解码使 decode 每步的 token 数可变（变长），进一步增加批组装的复杂度。⑤ 实测收益——Sarathi-Serve 报告在同等吞吐下可显著降低 TPOT 的 P99 抖动（从数十倍降到几倍），这对’在线服务’的用户体验很关键（平均延迟好但尾延迟差是常见问题）。⑥ 面试要点——被问’如何同时优化 TTFT 与吞吐’，应给出’chunked prefill + 与 decode 混批（算力/带宽互补）+ chunk 大小权衡‘，并说明’与 PD 分离是同一目标的两种实现’；能提到’尾延迟（P99 TPOT）’是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
Deep Dive & Engineering Trade-offs: ① TTFT vs TPOT Trade-off: Chunked prefill slightly extends Time-to-First-Token (TTFT) for the incoming long prompt (due to multiple chunk steps and kernel overhead) in exchange for dramatic reductions in TPOT jitter for all active decode requests. ② Kernel Complexity: The attention kernel must support mixed ragged batches where some sequences perform chunked cross-attention over variable-length caches while others perform single-token decoding (supported in FlashAttention-2/3 and vLLM). ③ Compute Efficiency: Merging a prefill chunk ($C=512$) with decode tokens converts small GEMVs into larger GEMMs, raising GPU Tensor Core utilization without saturating memory bandwidth. ④ Chunk Size Tuning: If $C$ is too small (e.g., 64), kernel launch overhead and low arithmetic intensity reduce throughput; if $C$ is too large (e.g., 2048), decode latency jitter reappears. Typical values are $C in [256, 1024]$. ⑤ Interview Strategy: Explain the root cause of TPOT jitter, sketch the chunked attention formulation, and demonstrate how balancing token counts achieves both predictable latency and high hardware utilization.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 把 prefill 与 decode 分开执行（浪费互补的算力/带宽）
- ⚠️ chunk 大小设得过小导致 TTFT 过高
English Pitfalls:
– Believing chunked prefill speeds up TTFT (it slightly slows down TTFT for the individual request to protect global TPOT)
– Ignoring the requirement for causal cross-attention kernels capable of handling past KV history during chunk processing
– Setting the chunk size arbitrarily without considering GPU compute saturation thresholds
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么 prefill 与 decode 混批能互补？
- How does FlashAttention support mixed ragged sequences containing both prefill chunks and decode tokens?
- chunk 大小如何影响 TTFT 与吞吐？
- What is the mathematical relationship between chunk size $C$ and GPU Tensor Core saturation?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
KV Cache 显存占用公式、Prefill/Decode 阶段与 PagedAttention(KV Cache Memory, Prefill/Decode & PagedAttention) - 🗺️ 知识图谱模块：
AI 基础设施工程导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。