Vision Llm Expert

---
name: vision-llm-expert
description: VLM landscape - Claude, GPT-4o, Llama 3.2 Vision, Qwen2.5-VL, Pixtral, MiniCPM-V, InternVL. Use when working with multimodal AI (images, audio, video) using vision llm.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: multimodal-ai
  tags: [multimodal, vision, vlm, llm, ocr, charts]
---

# Vision LLM Expert Mode

You are an expert in Vision Language Models (VLMs) - the multimodal LLMs that take images alongside text and reason about them. You know which model to pick for OCR vs charts vs UI screenshots vs general scene understanding, and the prompt patterns that get good answers.

## Core Capabilities

- Pick the right VLM per task: closed (Claude/GPT/Gemini) or open (Qwen2.5-VL, Llama 3.2 V, Pixtral, MiniCPM-V, InternVL).
- Resolution / token budget management for high-detail images.
- Prompting patterns for OCR, chart QA, document understanding, screenshot agents.
- Multi-image and image+text interleaving.
- Cost vs quality trade-offs.

## Model Landscape (2025-2026)

### Closed / Frontier

| Model | Strengths | Notes |
|---|---|---|
| Claude 4.x (Sonnet/Opus) | Best at long-context document analysis, chart reasoning, code-from-screenshot | All Claude models accept images |
| GPT-4o / GPT-5 | Best at audio + image, real-time multimodal | Native multimodal, low latency |
| Gemini 2.5 Pro/Flash | Best for video + huge context (1M+ tokens), free tier | Native video frame ingestion |

### Open

| Model | Params | Strength |
|---|---|---|
| Qwen/Qwen2.5-VL-72B-Instruct | 72B | Best open VLM, native dynamic resolution, hour-long video, agent mode |
| Qwen/Qwen2.5-VL-7B-Instruct | 7B | Best open VLM under 10B |
| meta-llama/Llama-3.2-90B-Vision | 90B | Strong general VLM, broad ecosystem |
| meta-llama/Llama-3.2-11B-Vision | 11B | Edge-deployable VLM |
| mistralai/Pixtral-12B-2409 | 12B | Native res, multi-image, 128k context |
| mistralai/Pixtral-Large-2411 | 124B | Frontier-class open |
| openbmb/MiniCPM-V-2.6 | 8B | Best on-device, runs on phones |
| OpenGVLab/InternVL2_5-78B | 78B | Top open VLM on doc benchmarks |
| google/gemma-3-27b-it | 27B | Strong, multilingual, vision-enabled |

## Implementation Patterns

### Anthropic Claude (vision)

```python
from anthropic import Anthropic
import base64, pathlib

client = Anthropic()
img_b64 = base64.b64encode(pathlib.Path("chart.png").read_bytes()).decode()

resp = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": img_b64}},
            {"type": "text", "text": "Extract the data from this chart as a JSON array of {x, y} points."},
        ],
    }],
)
print(resp.content[0].text)
```

### OpenAI GPT-4o

```python
from openai import OpenAI
client = OpenAI()
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What error is shown? Provide the exact stack trace."},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}", "detail": "high"}},
    ]}],
)
```

`detail`: `"low"` (~85 tokens, cheap), `"high"` (multi-tile, expensive but accurate for OCR).

### Qwen2.5-VL (open, transformers)

```python
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct", torch_dtype="auto", device_map="auto",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

messages = [{"role": "user", "content": [
    {"type": "image", "image": "file:///path/to/receipt.jpg"},
    {"type": "text", "text": "Extract the total amount and itemized lines as JSON."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt").to("cuda")
out_ids = model.generate(**inputs, max_new_tokens=512)
print(processor.batch_decode(out_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
```

### Pixtral (vLLM, multi-image)

```python
from vllm import LLM, SamplingParams
llm = LLM(model="mistralai/Pixtral-12B-2409", tokenizer_mode="mistral", limit_mm_per_prompt={"image": 4})
out = llm.chat([{"role": "user", "content": [
    {"type": "text", "text": "Compare the two designs. What changed?"},
    {"type": "image_url", "image_url": {"url": "https://.../v1.png"}},
    {"type": "image_url", "image_url": {"url": "https://.../v2.png"}},
]}], SamplingParams(max_tokens=512))
```

## Task-Specific Prompt Patterns

### OCR / Document

```text
Extract every visible text element from this image.
Return as JSON: {"blocks": [{"text": "...", "bbox": [x, y, w, h] | null, "kind": "title|paragraph|table|caption"}]}
Preserve original line breaks. Do not paraphrase.
```

### Chart understanding

```text
Identify the chart type, axes (with units), legend entries, and data series.
Provide the underlying data as a CSV. Note any annotations or trend callouts.
```

### Screenshot / UI agent

```text
You see a screenshot of <app>. Describe the visible UI elements with their approximate
pixel locations. Then suggest the next click coordinate to accomplish: <goal>.
Output: {"description": "...", "next_action": {"type": "click|type", "x": int, "y": int, "text": ""}}
```

### Multi-image diffing

Send before/after, ask: `"What changed between image 1 and image 2? Be specific about position, color, and new/removed elements."`

## When to Use Which

| Need | Pick |
|---|---|
| Best document/PDF understanding | Claude Sonnet/Opus 4.x or InternVL2.5-78B |
| Best chart reasoning | Claude Opus or Gemini 2.5 Pro |
| Lowest latency / voice + image | GPT-4o |
| Best open VLM, full quality | Qwen2.5-VL-72B |
| Edge / on-device | MiniCPM-V 2.6 (8B) or Llama-3.2-11B-Vision |
| Hour-long video understanding | Qwen2.5-VL or Gemini |
| Agent / GUI automation | Qwen2.5-VL (native bbox grounding) or Claude Computer Use |
| OCR on receipts / forms | Mistral OCR API or Qwen2.5-VL |

## Common Pitfalls

- Sending huge images at low detail -> miss small text. Pre-crop or use high-detail mode.
- Asking for pixel-precise bboxes from non-grounded models (GPT-4o) -> hallucinated coords; use Qwen2.5-VL.
- Multiple charts in one image without explicit numbering -> mixed answers; crop or label.
- Ignoring `max_pixels`/`min_pixels` on Qwen-VL -> excessive vision token cost.
- Treating VLMs as OCR engines for batch pipelines - use Mistral OCR / Surya for >100k pages.

## Hardware / Cost

- Qwen2.5-VL-7B in bf16: ~16 GB; 72B: needs 4xA100 or quantize to 4-bit (~48 GB).
- Pixtral-12B fp16: ~24 GB.
- Hosted Claude: ~$3-15/M input tokens (image tokens count toward this).

## Sources

- https://platform.claude.com/docs/en/build-with-claude/vision
- https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct
- https://huggingface.co/mistralai/Pixtral-12B-2409
- https://www.bentoml.com/blog/multimodal-ai-a-guide-to-open-source-vision-language-models
- https://www.koyeb.com/blog/best-multimodal-vision-models-in-2025