所属模块：
M5 · NLP 与大语言模型 (NLP & Large Language Models)| 专题分类：高效微调 PEFT (PEFT (LoRA / QLoRA / Prefix Tuning))| 难度等级：Medium
一、核心一句话结论 (One-Sentence Summary)
Prompt/Prefix Tuning 在输入/每层加可学习前缀（改激活）；LoRA 改权重（加低秩增量）；后者更通用且可合并。
Prompt/Prefix Tuning prepends learnable virtual tokens or key-value activations to transformer layers, whereas LoRA directly modulates weight matrices with low-rank deltas, ensuring zero inference latency and weight-folding capability.
二、核心考点要义 (Key Insights)
- 📌 Prompt Tuning：只在输入加可学习嵌入（参数极少）
- 📌 Prefix Tuning：在每层的 KV 前加可学习前缀（更强）
- 📌 LoRA：改权重（低秩增量）——更通用、可合并、无推理开销
English Insights:
– Prompt Tuning: injects trainable continuous embedding vectors directly at the input sequence layer; minimal parameter footprint but weak expressivity on smaller models
– Prefix Tuning: prepends trainable key-value prefixes to every transformer self-attention layer; more expressive but permanently consumes sequence context window capacity
– LoRA structural advantage: modifies linear weight parameters rather than activations; can be permanently merged into weights ($W = W_0 + Delta W$), incurring zero latency overhead and zero context penalty
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{prefix}: hleftarrow[h_{text{prefix}};h] text{per layer};qquad text{LoRA}: Wleftarrow W+BA (text{weights})$$
数学机理：三类 PEFT 的机制差异。(1) Prompt Tuning（Lester 等 2021）——在输入层前加若干个可学习的嵌入向量（’软提示’），只训练这些嵌入（参数量极小，如 100 个 × d）。特点——参数量最少；但表达力最弱（只影响输入层）；缺点——在小模型上效果差（需模型足够大才有’软提示’能力）、占用上下文长度（前缀占用 token 位置）。(2) Prefix Tuning（Li & Liang 2021）——在每一层的注意力计算前，给 K 和 V 拼接可学习的前缀（即每层都加’虚拟的 KV 对’）。特点——比 Prompt Tuning 强（影响每层）；缺点——(a) 占用注意力计算（每层多算前缀长度）、(b) 减少可用的上下文长度（因为前缀占位置）、(c) 推理时有额外计算（不像 LoRA 可合并）。(3) LoRA——修改权重（W + BA）。特点——(a) 表达力强（直接改权重）；(b) 可合并（推理时无额外开销）；(c) 不占上下文（不占用 token 位置）；(d) 在大/小模型上都有效。对比总结——(a) 参数量：Prompt Tuning < Prefix Tuning < LoRA（但都远小于全参）；(b) 表达力：LoRA > Prefix > Prompt（一般规律）；(c) 推理开销：LoRA 可合并（0）；Prefix/Prompt 有额外计算（且占上下文）；(d) 小模型适用性：LoRA 好；Prompt/Prefix 在小模型上差（研究表明 <10B 时 Prompt Tuning 明显不如 LoRA，但模型越大差距越小）；(e) 易用性：LoRA 生态最成熟。为什么 Prefix/Prompt 在小模型上差——它们依赖’模型能理解软提示’的能力，这需要模型足够大（有足够的’通用性’）；小模型的容量不足以从软提示中’解读’任务。其他 PEFT——(a) Adapter（在层间插入小 MLP 模块，有推理延迟）；(b) IA³（对激活做逐元素缩放，参数极少）；(c) BitFit（只训 bias，参数极少但表达力有限）。实践选择——LoRA 是默认（综合最优）；Prefix/Prompt 用于’参数极致受限’或’需要动态切换多个任务提示’的场景；Adapter 在部分场景仍有使用。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Mathematical Mechanism: 1. Prompt Tuning (Lester et al., 2021): Prepends $L$ virtual token embeddings $P in mathbb{R}^{L times d}$ to input embedding matrix $X in mathbb{R}^{N times d}$: $$tilde{X} = [P ; X] in mathbb{R}^{(L + N) times d}$$ Base model weights are completely frozen; only continuous prompts $P$ are trained. As model parameter size scales past 10B, prompt tuning approaches full fine-tuning quality, but fails on sub-3B models due to limited prompt steerability. 2. Prefix Tuning (Li & Liang, 2021): Prepends learnable prefixes directly to the key and value representations at every attention layer: $$K_{text{new}} = [P_K ; K(X)], quad V_{text{new}} = [P_V ; V(X)], quad P_K, P_V in mathbb{R}^{L times d}$$ Output hidden state at layer $l$: $$h_i = text{Softmax}left( frac{q_i K_{text{new}}^T}{sqrt{d}} right) V_{text{new}}$$ 3. LoRA Formulation (Hu et al., 2021): Modifies the linear weight mapping directly: $$y = x W_0 + frac{alpha}{r} x B A$$ 4. Architectural Comparison Matrix: begin{array}{l|c|c|c} textbf{Dimension} & textbf{Prompt Tuning} & textbf{Prefix Tuning} & textbf{LoRA} \ hline text{Intervention Site} & text{Input Embeddings} & text{KV Tensors per Layer} & text{Linear Layer Weights} \ text{Context Window Consumption} & text{Yes ($-L$ tokens)} & text{Yes ($-L$ tokens)} & textbf{None (0 tokens)} \ text{Inference Compute Overhead} & mathcal{O}(L cdot N) text{ attention} & mathcal{O}(L cdot N) text{ per layer} & textbf{Zero (foldable)} \ text{Small Model Compatibility} & text{Poor (<10B)} & text{Moderate} & textbf{Excellent (all scales)} end{array}
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① ‘LoRA 可合并、Prefix 不可’是关键工程差异——LoRA 合并后是普通模型（无额外开销）；Prefix 每层都有额外计算且占用上下文。故生产环境 LoRA 更优。② ‘Prefix 占用上下文长度’——因为它占用 token 位置（每层的前缀相当于额外的 KV）；这对长上下文场景不利。③ ‘小模型上 Prompt/Prefix 效果差’——这是’模型容量不足’的体现；故小模型应用 LoRA。④ ‘LoRA 生态最成熟’——工具支持（PEFT 库、vLLM 的 multi-LoRA）、社区适配器丰富；这降低了使用门槛。⑤ ‘多任务管理’——LoRA 的适配器可多任务共存（multi-LoRA 服务）；Prompt/Prefix 也可（不同的软提示）；但 LoRA 可合并（多任务能力合并），Prefix 不行。⑥ 面试要点——被问’Prompt/Prefix Tuning 与 LoRA 的差异’，应给出’作用位置（输入/每层 KV vs 权重）+ 表达力 + 是否可合并 + 是否占上下文 + 小模型适用性‘的对比，并给出’LoRA 是默认选择‘的结论；能指出’Prefix 占上下文且不可合并’是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
Deep Dive & Engineering Trade-offs: ① The Context Window Tax: Both Prompt Tuning and Prefix Tuning allocate $L$ virtual tokens (typically $L=20text{–}100$) that occupy precious positional slots in the KV cache across every layer. In long-context retrieval-augmented generation (RAG) or multi-turn conversational agents, sacrificing context length for adapter steering is unacceptable. LoRA introduces zero context consumption. ② Weight Merging and Serving Simplicity: LoRA can be statically folded into model weights ($W = W_0 + Delta W$), converting the fine-tuned model into standard weights with zero serving dependency on specialized frameworks. Prefix Tuning modifies the core attention execution path permanently, requiring specialized serving kernels across its entire deployment lifecycle. ③ Cross-Task Adapter Arithmetic: LoRA adapters reside in parameter vector space $mathbb{R}^{d times k}$ and natively support task vector addition, subtraction, TIES, and DARE merging. Prefix activations do not form linear weight spaces and cannot be algebraically merged across multi-task domains. ④ Historical Context and Paradigm Shift: Prompt and Prefix tuning were pioneering early PEFT milestones (2021); however, LoRA and its derivatives have completely superseded them in production due to superior expressivity, modularity, and operational simplicity. ⑤ Interview Strategy: Contrast the intervention sites (input tokens vs KV activations vs linear weights), detail the context window consumption penalty of prefix tuning, explain why LoRA enables zero-overhead weight folding, and highlight task arithmetic compatibility.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 在小模型上用 Prompt Tuning（效果差）
- ⚠️ 认为 Prefix Tuning 可合并进权重（不可）
English Pitfalls:
– Selecting Prompt Tuning for small models (< 7B), where frozen transformer representations lack the capacity to be steered by virtual embeddings
– Assuming Prefix Tuning can be folded directly into transformer weights; prefix tokens modify runtime activations and cannot be merged into static matrices
– Overlooking the context window length reduction incurred by prefix tokens in long-document understanding tasks
六、高频深度面试追问与预测 (Follow-Up Questions)
- 为什么 Prefix Tuning 会’占用上下文长度’？
- Why does Prompt Tuning degrade drastically in performance when applied to smaller language models compared to 100B+ models?
- 哪种方法在小模型上更差？
- How does the attention computation in Prefix Tuning differ mathematically from standard multi-head self-attention?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
参数高效微调全解：LoRA 低秩矩阵推导、QLoRA NF4 量化与梯度检查点(PEFT Deep Dive: LoRA Math, QLoRA NF4 & Activation Checkpointing) - 🗺️ 知识图谱模块：
大语言模型全景图谱
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。