Zoombench Eval

---
name: zoombench-eval
description: Evaluates fine-grained multimodal perception, visual grounding, and reasoning capabilities of vision-language models. It measures performance across general perception, specific perception (color, counting), and out-of-distribution generalization tasks using a suite of established and custom benchmarks. Use when the user wants to benchmark on ZoomBench, HR-Bench, VStar, CV-Bench, MME-RealWorld, ColorBench, CountQA, MMStar, BabyVision, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.11858
  bibtex_key: wei2026zooming
  confidence: high
---

# zoombench-eval

> Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception — Lai Wei et al. (2026) (arXiv:2602.11858, 2026)

## What this evaluates

Evaluates fine-grained multimodal perception, visual grounding, and reasoning capabilities of vision-language models. It measures performance across general perception, specific perception (color, counting), and out-of-distribution generalization tasks using a suite of established and custom benchmarks.

## Datasets

- **ZoomBench** — total ?; splits: test (-1); repo https://github.com/inclusionAI/Zooming-without-Zooming
- **HR-Bench** — total ?; splits: test (-1)
- **VStar** — total ?; splits: test (-1)
- **CV-Bench** — total ?; splits: test (-1)
- **MME-RealWorld** — total ?; splits: test-en (-1), test-cn (-1)
- **ColorBench** — total ?; splits: test (-1)
- **CountQA** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **BabyVision** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in the benchmark.

## Input / output format

**Input**: A single image paired with a text prompt or question.

**Output**: A natural language text response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Benchmarks use varying evaluation protocols (e.g., exact match vs. LLM-as-a-judge) that are not standardized across the suite.
- HR-Bench and MME-RealWorld have multiple resolution/language splits (4K/8K, en/cn) that must be averaged or reported separately to match the paper's 'Avg' column.
- ZoomBench's 'dual-view evaluation' measures performance on both full images and cropped regions, which can be confused with the standard single-pass evaluation.

## Evidence (verbatim from paper)

> We report accuracy (%) for each model. Among open-source models (except GPT-5.1 and Gemini-3-Flash), the best results are highlighted in bold, and the second-best are underlined.

## Citation

```bibtex
@misc{wei2026zooming,
  title={Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception},
  author={Lai Wei et al. (2026)},
  year={2026},
  note={arXiv:2602.11858}
}
```

- arXiv: 2602.11858