V1.2: A Better Judge for Hands
When we moved to Gemini 3.1 Pro in V1.1, we said a stricter judge was catching the six-fingered hands a lenient one waved through. On hands specifically, that was wrong in the other direction: Gemini fails almost every hand — including good ones. V1.2 fixes just that one test by routing the Hands subcategory to Qwen3.5-122B, the judge that best matches human labels. Human-realism scores climb about 18 points; nothing else changes and the ranking holds.
The problem: Gemini can’t see good hands
Hands are the hardest thing for an image model to draw and one of the hardest to grade. Under V1.1, Gemini 3.1 Pro graded the 12 hands prompts for every model — and it flagged nearly all of them as failures. Across the board it passed only 0–1 of 12 hands for most models, frontier models included. Read literally, that says even the best generators can barely draw a hand. Look at the images and that isn’t true: plenty of the flagged hands are anatomically fine.
The judge wasn’t being usefully strict — it was over-flagging. A grader that fails everything can’t rank anything: if the frontier and the weakest model both score 1/12, the test has stopped measuring hands.
The evidence: a human-labeled hands set
To settle it we hand-labeled a 24-image calibration set — every hand marked PASS or FAIL by a human reviewer — and scored each candidate judge against those labels, treating a bad hand as the thing to detect.
| Judge | F1 vs human | Recall | Precision |
|---|---|---|---|
| Qwen3.5-122B | 80% | 67% | 100% |
| Gemma4-26B | 71% | 100% | 55% |
| Qwen3.6-27B | 67% | 75% | 60% |
| Gemini 3.1 Pro (V1.1) | 65% | 92% | 50% |
| Qwen3-VL | 0% | 0% | 0% |
Gemini’s 92% recall / 50% precision is the signature of over-flagging: it catches every bad hand because it fails nearly all of them, so half its “fails” are good hands. Qwen3.5-122B lands closest to the human at F1 80%, precision 100% — when it flags a hand, the human agreed every time. (Qwen3-VL sits at the opposite extreme: it passes everything, so it detects nothing.)
Per-image: what the two judges said
The 24 human-labeled images, graded PASS/FAIL by each judge. Cells that disagree with the human are tinted red. On Nano Banana Pro — mostly good hands — Gemini’s column is a wall of red: 11 FAILs out of 12, and 10 of them are hands the human passed. Qwen tracks the human almost exactly. On Bonsai Image 4B — mostly bad hands — both flag most of them, so Gemini looks fine here; it fails everything either way, which is exactly why it can’t tell the two models apart.
| Model | Test | Hand | Human | Gemini 3.1 Pro · V1.1 | Qwen3.5-122B · V1.2 |
|---|---|---|---|---|---|
| Nano Banana Pro | Easy · p1 |  | PASS | FAIL | FAIL |
| Easy · p2 |  | PASS | FAIL | PASS | |
| Easy · p3 |  | FAIL | FAIL | PASS | |
| Medium · p1 |  | PASS | FAIL | PASS | |
| Medium · p2 |  | PASS | FAIL | PASS | |
| Medium · p3 |  | PASS | FAIL | PASS | |
| Hard · p1 |  | PASS | FAIL | PASS | |
| Hard · p2 |  | PASS | PASS | PASS | |
| Hard · p3 |  | PASS | FAIL | PASS | |
| Extreme · p1 |  | PASS | FAIL | PASS | |
| Extreme · p2 |  | PASS | FAIL | PASS | |
| Extreme · p3 |  | PASS | FAIL | PASS | |
| Bonsai Image 4B | Easy · p1 |  | FAIL | FAIL | FAIL |
| Easy · p2 |  | PASS | FAIL | PASS | |
| Easy · p3 |  | FAIL | FAIL | PASS | |
| Medium · p1 |  | FAIL | FAIL | FAIL | |
| Medium · p2 |  | FAIL | FAIL | PASS | |
| Medium · p3 |  | FAIL | FAIL | FAIL | |
| Hard · p1 |  | FAIL | FAIL | FAIL | |
| Hard · p2 |  | FAIL | PASS | PASS | |
| Hard · p3 |  | FAIL | FAIL | FAIL | |
| Extreme · p1 |  | FAIL | FAIL | FAIL | |
| Extreme · p2 |  | FAIL | FAIL | FAIL | |
| Extreme · p3 |  | FAIL | FAIL | PASS |
Across all 24 images Qwen3.5-122B agrees with the human 18 times; Gemini only 12 — and mostly by failing the bad-hand set, not by judging well. The tinted Nano column is the over-flagging made concrete.
The change: one judge, one subcategory
V1.2 is V1.1 with a single routing change. The 12 Hands tests are now judged by Qwen3.5-122B; the other 180 tests stay with Gemini 3.1 Pro. Same 192 generation prompts, same specialized questions, same PASS/FAIL protocol, same images — only who grades hands is different. (The original V1.0 spec already routed hands to Qwen3.5-122B; V1.2 restores that on top of the V1.1 Gemini baseline.)
What it does to the scores
Every model’s Human realism category rises — on average +18.1 points (from +9.5 to +26.2), because the hands that were being wrongly failed now pass. Overall Capability moves about +4 points, and the headline Overall score about +2. As with the V1.1 change, the shift is broad and even, so the ranking barely moves — gpt-image-2 stays on top, sana stays last.
| Model | Human realism V1.1 | Human realism V1.2 | Capability | Overall |
|---|---|---|---|---|
| gpt-image-2 | 54.8 | 69.0 | 87.0 | 80.0 |
| nano-banana-2 | 38.1 | 54.8 | 77.1 | 74.9 |
| nano-banana-pro | 33.3 | 57.1 | 78.6 | 68.8 |
| seedream-v5-pro | 35.7 | 57.1 | 75.0 | 68.0 |
| seedream-v4 | 33.3 | 59.5 | 68.2 | 62.2 |
| flux-2-pro | 31.0 | 50.0 | 67.2 | 62.2 |
| sana-1.5 | 7.1 | 21.4 | 34.9 | 34.5 |
Overall is 0.5 × Capability + 0.5 × EPS; the aesthetic axis is untouched by the judge swap, so the Overall move is muted relative to Capability.
Honest caveats
- “Better” means closer to humans, not lenient. Qwen3.5-122B passes more hands than Gemini, but the point is agreement with human labels (F1 80% vs 65%), not a higher pass rate. On a 24-image set with ~12 bad hands, that is a directional result, not a proof.
- Only hands changed. Every other category is still graded by Gemini 3.1 Pro exactly as in V1.1. If you compare a model to V1.1, only its Human-realism line (and the totals it feeds) should differ.
- Different judges for different tests. V1.2 is explicitly a mixed panel — a frontier commercial model for most categories, a large open model for hands. That is a deliberate trade of uniformity for accuracy on the one test where the uniform judge was wrong.