Identifying instruction-relevant regions for information-rich image understanding.
1Korea University 2KT Corporation 3Soongsil University 4Kakao Mobility · †Corresponding authors
Token pruning throws away tokens the attention map thinks are boring, and it is often wrong. PinPoint instead asks where the instruction is pointing, crops those regions, and re-encodes them at full detail. The token budget goes down; the answers get better.
over the vanilla LLaVA-NeXT baseline, using 65.3% of its FLOPs. Against SparseVLM, the strongest prior method, +24.5% ANLS with 4.9% fewer FLOPs.
Only the guidance queries and two MLPs are trained. The LLM, vision encoder and projector stay frozen — 1.33 GPU-hours to converge on InfoVQA.
Answer, evidence and encompass boxes for InfoVQA, SPDocVQA and MPDocVQA — with an automated pipeline that keeps manual labelling under 10%.
Feed LLaVA-NeXT only the tokens inside the ground-truth region and it answers best. Dilute them with instruction-irrelevant tokens and accuracy falls monotonically. Extra context is not neutral — it is noise you pay for twice, in FLOPs and in accuracy.
Hover the line to read each point. Left edge: only ground-truth-region tokens. Right edge: every token in the image, undifferentiated.
Attention weights in the decoding layers are noisy and can sink onto uninformative positions, so pruning by attention score removes tokens the answer actually needed and the model hallucinates a plausible number instead.
Text spans many tokens, so per-token decisions cut phrases in half. And because self-attention is global, tokens inside the answer region already carry irrelevant context, blurring exactly the distinction pruning depends on.
The same thing you do with a Where's Wally page. Stage 1 compares the instruction against region-level features to pick candidate windows. Stage 2 crops those windows and encodes them again in isolation, so the surviving tokens are dense, fine-grained, and free of global contamination.
Visual tokens are reshaped into a 2D grid and swept with a 10×10 window at stride 7. Learnable guidance queries E attend to each region and to the text, giving both modalities a shared space to be compared in — no CLS token needed, no BPE mismatch. Regions are ranked by cosine similarity to the instruction and taken until they cover a ratio r of the image.
The selected crops are pushed back through the vision encoder on their own. Stripped of global context, the result is a smaller and denser set of tokens that carries genuinely fine-grained detail — which is what reading a 4.9%-of-the-page number in an infographic actually requires.
An inter-modal loss aligns instructions with their positive region across the batch. An intra-image loss separates the positive region from the irrelevant regions inside the same image — the distinction that actually matters at selection time. Adding it lifts region accuracy on GQA from 93% to 98%.
Selection and refinement add 1.67 TFLOPs on LLaVA-NeXT — 7.1% of the total — and 5.68 TFLOPs on Qwen2-VL. Unlike ViCrop, there is no iterative decoding: one lightweight query pass finds the region.
Pick a question, then move r — the share of the image PinPoint is allowed to keep. Windows are selected by similarity to the instruction until that budget is spent. Numbers below the canvas are the measured results from the ablation on r (Table A7).
The canvas is a schematic stand-in for a real infographic — window geometry (10×10, stride 7) and the greedy top-k rule are the ones from the paper. Metrics are read from Table A7 for the selected backbone, dataset and r.
Accuracy on the vertical axis, compute on the horizontal. Every competing method trades one for the other; PinPoint moves toward the corner where both are better. The dashed line is the vanilla model at 100% FLOPs.
| Method | ANLS ↑ | FLOPs (T) ↓ | % of vanilla | Latency (ms) ↓ |
|---|
GQA is reported as accuracy rather than ANLS. Latency is measured end-to-end, from input processing to generated text.
Grounding the answer in a located region lowers hallucination across both discriminative and generative benchmarks: POPE-adversarial 85.6 → 86.0, CHAIRS 26.1 → 25.6, AMBER Hal 48.9 → 42.4 — at 22.18 TFLOPs against the baseline's 34.32.
Trained exclusively on GQA, PinPoint still beats train-free baselines on MMMU (102.6% of vanilla), MMMU-Pro (106.8%) and TextVQA (136.9%), at 61.1% of vanilla FLOPs. It learns to localize, not to memorize a distribution.
Answering “what is the object the man is pointing at?” means finding the man first. Prior benchmarks annotate one box around the answer; we annotate the answer, every supporting evidence element, and an encompass box that covers them together — the region that actually grounds the instruction.
| Dataset | Split | answer | evidence | encompass | Images | Pages | QA pairs |
|---|---|---|---|---|---|---|---|
| InfoVQA | train | 1.06 | 1.64 | 1.0 | 4,162 | 1.0 | 17,887 |
| InfoVQA | val | 1.09 | 1.53 | 1.0 | 500 | 1.0 | 2,801 |
| SPDocVQA | train | 1.00 | 0.30 | 1.0 | 10,194 | 1.0 | 39,463 |
| SPDocVQA | val | 1.00 | 0.28 | 1.0 | 1,286 | 1.0 | 5,349 |
| MPDocVQA | train | 1.00 | 1.05 | 1.0 | 5,131 | 9.1 | 36,230 |
| MPDocVQA | val | 1.00 | 1.02 | 1.0 | 927 | 5.6 | 5,187 |
Average number of boxes per QA pair. Manual self-annotation stays under 10% on every benchmark. Download and license details will be posted in the repository.
@inproceedings{kwon2026pinpoint,
title = {Focus, Don't Prune: Identifying Instruction-Relevant Regions
for Information-Rich Image Understanding},
author = {Kwon, Mincheol and Lee, Minseung and Choi, Seonga and Choi, Miso
and Oh, Kyeongjin and Lee, Hyunyoung and Park, Cheonyoung
and Song, Yongho and Park, Seunghyun and Kim, Jinkyu},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)},
year = {2026}
}