所属模块：
M6 · 多模态与生成模型 (Multimodal & Generative Models)| 专题分类：VLM 训练与评估 (VLM Training & Evaluation)| 难度等级：Hard
一、核心一句话结论 (One-Sentence Summary)
多图需评’跨图比较/关系’；视频需评’时序/因果/长程’；两者都受 token 预算与采样策略影响。
Evaluating multi-image and video VLMs is complicated by frame sampling discrepancies, single-frame shortcut cheating, long-horizon temporal reasoning failures, and severe visual token context explosion.
二、核心考点要义 (Key Insights)
- 📌 多图：跨图比较、指代（’图 2 中的物体’）、图像数量泛化
- 📌 视频：时序理解、动作识别、因果推理、长程依赖
- 📌 难点：token 预算（帧/图多了放不下）、采样策略影响结果
English Insights:
– The single-frame shortcut trap: over 50% of questions in popular video benchmarks can be answered from a single static frame without temporal reasoning
– Frame sampling inconsistency: varying frame sampling rates (e.g., uniform 8 vs 32 frames) creates non-comparable benchmark scores across research publications
– Multi-image reasoning hurdles: requires maintaining distinct visual representations simultaneously, testing cross-image comparison, temporal sequence order, and counterfactual changes
三、核心数学原理与机理推导 (Mathematical Principles & Derivation)
$$text{multi-image}: text{cross-image reasoning};qquad text{video}: text{temporal}, text{causal}, text{long-horizon}$$
数学机理：多图评估的难点——(a) 跨图推理——需理解’图 1 与图 2 的关系’（比较、变化、相同/不同）；这类题目要求模型同时保持多图的信息（而非只看一张）；(b) 指代清晰性——’第二张图里的物体’需要模型正确索引图像；故需图像索引机制（如 标记）；(c) 图像数量泛化——训练时 2 张图、推理时 8 张，性能可能显著下降（因为注意力分布与 token 预算变化）；(d) token 预算——M 张图 × N token 易超上下文，故需压缩（但压缩可能损害跨图细节）；(e) 基准稀缺——多图基准较少（NLVR2、部分 MMMU 题、以及专门的’多图 QA’）；评估覆盖不足。视频评估的难点——(a) 时序理解——’先发生什么’（动作顺序）、’持续多久’（时长）；需模型理解时间维；(b) 因果推理——’为什么这个动作导致那个结果’；(c) 长程依赖——长视频（分钟级）的关键信息可能相隔很远（如’开头的人物在结尾出现’）；(d) 采样策略的影响——视频需抽帧（帧数有限）；抽帧方式（均匀/关键帧/密集）显著影响结果——同一模型不同抽帧策略的分数差异可能很大，导致结果不可比；(e) 时序分辨率与 token 预算的矛盾——帧数多则 token 多（成本高）；帧数少则漏掉关键动作；(f) 基准的局限——现有视频基准多测’单帧可答’的问题（如’视频里有什么’），而非真正的时序推理；(g) 评估成本——视频的 token 数远大于图像（数十帧 × 每帧数百 token），评估成本高。评估实践——(a) 多图——用’指代明确’的题目（避免歧义）、测’图像数量泛化’（不同数量）、报告 token 预算；(b) 视频——固定抽帧策略（否则不可比）、报告帧数与采样方式、区分’单帧可答’与’需时序’的题目（如 TempCompass 的时序子集）、用专门基准（Video-MME、MVBench、TempCompass）；(c) 两者都需——报告’因预算不足的失败’、与’单图/单帧基线’对比。改进方向——(a) 时序位置编码（3D RoPE 的时间维）；(b) 稀疏/分层采样（先粗筛关键帧、再密集处理）；(c) 视频摘要（把长视频压成关键帧 + 文本摘要）；(d) 专门的时序注意力设计（时空分离注意力）。
📖 查看英文严格数学推导 (English Mathematical Derivation)
Mathematical Mechanism: 1. Single-Frame Shortcut Cheating Diagnostic: Let video $V = {f_1, f_2, dots, f_T}$ and query $q$. Define the Temporal Grounding Dependency metric: $$Delta_{text{temporal}} = text{Acc}(f_{text{all}}, q) – max_{t in {1, dots, T}} text{Acc}(f_t, q)$$ If $Delta_{text{temporal}} approx 0$, the question is a static perception problem (e.g., ‘What color is the car?’) rather than a temporal reasoning problem (e.g., ‘Did the car turn left before or after the pedestrian crossed?’). Video-ChatGPT and early ActivityNet benchmarks suffered from severe static frame leakage. 2. Visual Token Scaling in Video Sequences: For video of duration $D$ seconds sampled at frame rate $R$ (total $T = D cdot R$ frames), with spatial patch count $N_p$ per frame: $$N_{text{video}} = T times N_p = (D cdot R) times frac{HW}{P^2}$$ For a 60-second video at 1 FPS ($T=60$) with $N_p = 256$ tokens/frame: $$N_{text{video}} = 60 times 256 = 15,360 text{ tokens}$$ Squeezing 15K visual tokens into the LLM drives attention FLOPs to $mathcal{O}(T^2 N_p^2 D)$. 3. Frame Sampling Protocols: (a) Uniform Frame Sampling: $t_k = lfloor k cdot frac{T}{K} rfloor$. (b) Dynamic Scene-Change Sampling: Evaluates frame perceptual difference $|f_{t} – f_{t-1}|_2 > tau$ to allocate frames strictly across scene transitions.
四、工业级落地权衡与工程考量 (Industrial Trade-offs)
深度剖析与工程权衡：① ‘抽帧策略影响结果’是视频评估的关键问题——不同论文用不同抽帧方式，导致分数不可比；故必须报告抽帧配置（帧数、采样方式）。② ‘单帧可答’的问题掩盖时序能力——很多视频基准的问题用单帧就能答（如’视频里有什么’）；故需 (a) 时序专门子集、(b) 与’单帧基线’对比。③ ‘图像数量泛化’是多图的常见失败——训练时数量固定会导致推理时数量变化即失效；故需训练时随机化数量。④ ‘token 预算是最硬的约束’——多图/视频的 token 需求远超图像；故需 (a) 固定 K 压缩、(b) 稀疏采样、(c) 分层处理。⑤ ‘指代明确性’影响多图评估的公平性——若题目中的’那张图’指代不明，则模型答错不是能力问题；故基准设计需明确索引。⑥ 面试要点——被问’多图/视频怎么评估’，应给出’多图（跨图推理/指代/数量泛化/预算）+ 视频（时序/因果/长程/抽帧策略）‘与’必须报告抽帧配置、区分单帧可答与时序题‘；能指出’抽帧策略导致结果不可比’是深度理解的标志。
⚙️ 查看英文落地权衡分析 (English Systems & Trade-offs)
Deep Dive & Engineering Trade-offs: ① The Frame Rate vs Token Budget Trade-off: Video models cannot ingest every raw frame. Selecting 8 frames per video saves compute but misses fleeting actions (e.g., dropping a key); selecting 64 frames captures fine temporal dynamics but exhausts LLM KV cache memory. State-of-the-art architectures (Video-LLaVA, Qwen2-VL) utilize spatio-temporal pooling, compressing each frame’s spatial tokens by $4times$ to enable packing 64+ temporal frames into the context window. ② Benchmarking with Strict Temporal Suites: Modern benchmarks (Video-ChatGPT, PerceptionTest, MVBench, Video-MME) isolate temporal reasoning by including questions on action sequence order, velocity, duration estimation, and event reversal detection. ③ Image Indexing in Multi-Image Contexts: When comparing two high-resolution images (‘Identify the differences between Image A and Image B’), the model must maintain coordinate grounding across independent visual streams without mixing up patch coordinates. Dedicated image index tokens (“, “) are essential. ④ Streaming Video Inference: Offline batch video evaluation differs fundamentally from real-time streaming video serving, which requires rolling KV caches and sliding-window attention. ⑤ Interview Strategy: Formulate the single-frame baseline subtraction $Delta_{text{temporal}}$, quantify video token context explosion ($T times N_p$), contrast uniform vs scene-adaptive frame sampling, and describe spatial token pooling mitigations.
五、常见面试避坑陷阱 (Common Pitfalls & Traps)
- ⚠️ 不报告抽帧策略（结果不可比）
- ⚠️ 用’单帧可答’的题评估时序能力
English Pitfalls:
– Evaluating video understanding models without reporting a single-frame baseline, masking the fact that questions fail to test temporal reasoning
– Comparing video models evaluated with different frame counts (e.g., 8 frames vs 64 frames), confounding model capability with visual sampling density
– Feeding raw uncompressed high-frame-rate video streams directly into LLM contexts, triggering immediate out-of-memory crashes
六、高频深度面试追问与预测 (Follow-Up Questions)
- 视频的’时序’如何评？
- Why do many video question answering benchmarks fail to genuinely test temporal reasoning, and how do modern benchmarks isolate temporal dependency?
- 为什么视频评估结果不可比？
- How does spatio-temporal token pooling compress video representations to allow processing 64+ frames within standard context windows?
七、知识图谱对齐 (Knowledge Graph Anchor)
- 🔗 关联底层卡片：
大视觉语言模型预训练与指令对齐流水线、MMBench 评测(VLM Pretraining, Multimodal SFT & MMBench Evaluation) - 🗺️ 知识图谱模块：
多模态与扩散模型导图
🔬 算法科学家与机器学习深度考察全量题库 (Science Depth)
本题收录于 TalentMe 算法科学家深度考察真题库 (Science Depth)。全库共 856 道硬核考点，深度覆盖数学统计、经典ML、深度学习、Transformer、大语言模型、多模态、推荐系统与 MLOps。支持 Jev 面经智能匹配、一键离线单文件 HTML 手册导出并直连 Obsidian 本地记忆。