V1.1: Grading the Benchmark with Gemini 3.1 Pro
The Capability score — whether a model did what the prompt asked — is only as good as the judge that grades it. ImageBench V1.1 upgrades that judge on two fronts at once: from local open VLMs to Gemini 3.1 Pro, and from a generic question template to a specialized question written for every prompt. The result is a stricter, more consistent grader. Capability scores fall by about 22 points across the board — but the ranking, and the story it tells, largely holds.
What V1.0 did, and why we changed it
In V1.0, every one of the 192 images was scored PASS/FAIL by a routed panel of local open-weight VLMs — qwen3-vl, qwen3.5-122b, gemma — each asked a generic question with the original prompt pasted in: “the prompt asked for X; does the image match? PASS or FAIL.” It worked, and it was free to run. But it had two blind spots.
- The judge had to interpret the prompt itself. Handing a model a paragraph and asking “does this match?” makes it do two jobs — parse intent, then check the image — and it can get the first one wrong.
- Open VLMs were lenient. On close calls — an extra finger, a miscount, a subtly wrong reflection — the local judges tended to wave the image through.
Change one: specialized questions
Instead of one template reused across every prompt, each prompt now has a standalone question with the ground truth baked in. We do the interpretation once, carefully, so the judge answers a single concrete thing. Take the counting prompt “Exactly 9 goldfish swimming in a large round fishbowl” — here is exactly what the judge is asked, before and after:
Count the objects in this image. The prompt asked for: “Exactly 9 goldfish swimming in a large round fishbowl”. How many of each object type do you see? Does the count exactly match what was requested? Rate: PASS or FAIL.
How many goldfish are in the image? PASS if there are exactly 9; FAIL for any other number.
The generic version makes the judge do two jobs — read the prompt to work out what “correct” means, then check the image. The specialized version has already done the first job, so the judge just counts. That removes a whole class of interpretation errors, and it is a clear win for the objective categories — text rendering, counting, negation, spatial relations, world knowledge. For the more subjective tests (faces, studio lighting, style) the question is tightened but the call is still a judgement.
Change two: a stricter judge
The specialized questions go to Gemini 3.1 Pro, a frontier vision model. It is noticeably stricter than the local panel and than smaller commercial models: on a controlled comparison it agreed with the local judges only about two-thirds of the time, and where it disagreed it was usually catching a defect the others missed. Hands are the clearest example — asked to count fingers, Pro reliably flags the six-fingered grips and fused digits that a lenient judge passes.
We also validated the other direction: a large open VLM (Qwen3-VL-235B) run through the same pipeline clustered with the lenient local judges, not with Pro — confirming the strictness is a real property of the frontier model, not just of “a bigger model.”
What it does to the scores
Averaged over all 32 models, Capability drops from 76.1 to 54.1 — about 22 points. The drop is remarkably uniform, which is the point: a stricter judge applied evenly re-scales the board without scrambling it. The strongest models stay on top; the gaps compress.
| Model | Capability V1.0 | Capability V1.1 | Overall |
|---|---|---|---|
| gpt-image-2 | 95.3 | 83.9 | 78.5 |
| nano-banana-2 | 95.3 | 73.4 | 73.0 |
| nano-banana-pro | 92.2 | 73.4 | 66.2 |
| flux-2-pro | 83.9 | 63.0 | 60.1 |
| flux-2-max | 91.1 | 62.5 | 59.6 |
| seedream-v4 | 84.4 | 62.5 | 59.4 |
The headline Overall score moves less than Capability alone. Overall is 0.5 × Capability + 0.5 × EPS, and the Estimated Preference Score — the aesthetic axis — is unchanged by the judge swap. So a model’s taste score anchors it while its capability is re-graded.
Reading the new numbers
V1.1 Capability is not comparable to V1.0: a 63% today is harder-earned than a 63% was last week, because the bar moved. Treat the switch as a re-baselining of the capability axis. If you were tracking a model over time, re-read it against the new board, not the old number.
Everything else about the benchmark is unchanged: the same 192 prompts across six capability categories, the same EPS aesthetic axis, the same Overall blend. Only the grader is new.
Honest caveats
- The judge is a commercial model. Gemini 3.1 Pro is a hosted, evolving model — grading is no longer fully reproducible on local hardware, and it costs money to run. We accept that trade for the accuracy gain, but it is a real change in the benchmark’s character.
- Strict is a choice, not a law. A stricter judge is only “better” if its stricter calls are correct. We spot-checked disagreements and found Pro was usually right, but on genuinely subjective tests reasonable judges still differ.
- Specialized questions can be demanding. Baking the ground truth in makes the objective tests sharp, but a hard question judged strictly will fail images a looser reading would pass. That is intended — but worth knowing when a score looks low.