Can an Open Model Grade Our Benchmark?
Most of ImageBench is graded by Gemini 3.1 Pro, a closed model that nobody can freeze or rerun forever. So when Tencent released WeVisOne-VL, an open model built to find and count things, we tested it as a judge. With blind human labels as the reference, it tied Gemini on counting and on simple negation, and lost on text and attributes. A tie is a better result than we expected, but not enough to change the benchmark. V1.2 stays, and the full dataset is public.
Why try another judge
A judge is part of the measurement. If the judge changes, every score can drift. An open model with published weights can be pinned and rerun years from now, and a detector gives something a chat model does not: boxes you can look at. If it says nine goldfish, you can see which nine.
WeVisOne-VL comes in 2B and 4B versions under Apache-2.0. We tested 4B, the largest published. It does not reason in prose. It answers with geometry: boxes, dots, masks and OCR words. That makes it a poor fit for “does this look like a photo?” and a plausible fit for “are there exactly three apples?”
First attempt: the obvious prompt was the wrong one
We started with the model’s built-in count prompts on all 728 Counting images, from 81 models. The pass rates looked close: Gemini 47.4%, WeVisOne 45.6% with dots and 45.3% with boxes. A reviewer then labeled the images blind. On the 44 confident images where the judges disagreed, Gemini was right 70.5% of the time, the dot prompt 61.4% and the box prompt 38.6%. All three got the 17 confident control images right. Gemini won, and we nearly stopped there.
We also tried finger counting on the 24 human-labeled images from our hands study, cropping each hand and counting fingers. The best rule reached F1 71% and precision 58%, against F1 80% and precision 100% for Qwen3.5-122B. Crops often held more than one hand, and grips hide fingers. Hands stay with Qwen.
But the count prompt is only one of the model’s outputs, so a loss there did not tell us what the model could do.
What each output actually sees
We ran every method we could build on 10 hard images, picked by hand. That sample is deliberately biased, so read this table for how methods fail, not as a ranking. Rows with fewer than 10 images are methods that do not apply to every prompt.
| Method | Matches the human |
|---|---|
| Gemini 3.1 Pro (live judge) | 8 / 10 |
| Plain detection | 7 / 10 |
| Referring prompt | 7 / 10 |
| Detection, singular wording | 7 / 10 |
| Detection, upscaled image | 7 / 10 |
| Generic noun mask + color from mask pixels | 6 / 7 |
| Joint detection, all classes in one call | 4 / 5 |
| Pointing dots | 6 / 10 |
| Count prompt, boxes (our first test) | 6 / 10 |
| Segmentation masks | 6 / 10 |
| Majority vote of dots, detection and masks | 6 / 10 |
| Count prompt, dots (our first test) | 4 / 10 |
| Visual prompt | 4 / 10 |
The dot-count prompt we had leaned on was among the weakest. Each output also had its own blind spot. Boxes missed a sliver of an orange that dots found:
The model also counts literally. Ask for yellow rubber ducks in front of a mirror and it counts the reflections too. Masking the generic word “ducks” happened to return only the real five:
Gemini has its own failure. On this GPT Image 2 fishbowl it stated “exactly 9 goldfish” and passed the image. WeVisOne found 10, and our reviewer failed it. Boxes make that kind of error easy to check.
The real test: every disagreement, labeled blind
We picked two candidates and wrote the rules down before running them: plain detection, and masks of a generic noun with color read from the mask pixels. Both ran on all 728 Counting images. Most verdicts matched Gemini, so only the disagreements could separate them: 36 images for detection, 43 for masks.
Each disagreement got a human label, with the verdicts hidden until the reviewer had answered. The 10 images used to choose the methods were left out of the score, and a method needed p < 0.025 on a McNemar test to claim a win.
Neither won. Detection came out at 11 right to Gemini’s 13, which is a statistical tie. Masks, the method we had expected the most from, lost 11 to 19; the gap is not significant, but it went the wrong way. Pass rates across the board were also close: Gemini 345 of 728, detection 331, masks 328.
The two judges are good at different images. On the goldfish prompt detection beat Gemini 5 to 1. On every other prompt combined it lost 6 to 12. Balloons barely separated them: across 81 images they disagreed once, and Gemini was right.
Every labeled disagreement (40 images)
Cells that disagree with the human are tinted red. Marked images were used to choose the methods and are not in the score. Images the reviewer marked unsure are left out.
| Image | Prompt asks for | Human | Gemini | Detection | Masks |
|---|---|---|---|---|---|
 | 3 red apples | FAIL | FAIL | PASS | PASS |
 | 3 red apples | FAIL | FAIL | PASS | PASS |
 | 3 red apples | PASS | PASS | PASS | FAIL |
 | 3 red apples | PASS | FAIL | PASS | PASS |
 | 5 yellow rubber ducks | PASS | PASS | FAIL | PASS |
 | 5 yellow rubber ducks (tuning image) | PASS | PASS | FAIL | PASS |
 | 4 white candles | PASS | PASS | FAIL | PASS |
 | 9 goldfish | FAIL | PASS | FAIL | FAIL |
 | 9 goldfish | FAIL | PASS | FAIL | FAIL |
 | 9 goldfish | FAIL | PASS | FAIL | FAIL |
 | 9 goldfish (tuning image) | PASS | PASS | FAIL | FAIL |
 | 9 goldfish | PASS | FAIL | PASS | PASS |
 | 9 goldfish | FAIL | FAIL | PASS | PASS |
 | 9 goldfish (tuning image) | FAIL | PASS | FAIL | FAIL |
 | 9 goldfish | FAIL | PASS | FAIL | FAIL |
 | 9 goldfish (tuning image) | FAIL | PASS | FAIL | FAIL |
 | 3 bananas, 4 oranges, 2 apples (tuning image) | PASS | PASS | FAIL | FAIL |
 | 3 bananas, 4 oranges, 2 apples | PASS | PASS | FAIL | FAIL |
 | 3 bananas, 4 oranges, 2 apples | PASS | PASS | FAIL | FAIL |
 | 3 bananas, 4 oranges, 2 apples | FAIL | PASS | FAIL | FAIL |
 | 11 balloons | PASS | PASS | FAIL | FAIL |
 | 4 red roses, 2 white daisies | PASS | PASS | PASS | FAIL |
 | 4 red roses, 2 white daisies | PASS | PASS | PASS | FAIL |
 | 4 red roses, 2 white daisies | FAIL | PASS | FAIL | FAIL |
 | 4 red roses, 2 white daisies | PASS | PASS | PASS | FAIL |
 | 4 red roses, 2 white daisies | PASS | FAIL | PASS | PASS |
 | 4 red roses, 2 white daisies | PASS | PASS | PASS | FAIL |
 | 4 red roses, 2 white daisies | PASS | PASS | PASS | FAIL |
 | 4 red roses, 2 white daisies | PASS | PASS | PASS | FAIL |
 | 4 red roses, 2 white daisies | PASS | PASS | PASS | FAIL |
 | 4 red roses, 2 white daisies | PASS | PASS | PASS | FAIL |
 | 1 blue butterflies, 3 yellow butterflies | PASS | PASS | FAIL | PASS |
 | 1 blue butterflies, 3 yellow butterflies | PASS | PASS | FAIL | FAIL |
 | 1 blue butterflies, 3 yellow butterflies | PASS | PASS | PASS | FAIL |
 | 1 blue butterflies, 3 yellow butterflies | PASS | FAIL | PASS | PASS |
 | 1 blue butterflies, 3 yellow butterflies | PASS | PASS | FAIL | PASS |
 | 1 blue butterflies, 3 yellow butterflies | PASS | PASS | FAIL | FAIL |
 | 1 blue butterflies, 3 yellow butterflies | PASS | FAIL | PASS | PASS |
 | 1 blue butterflies, 3 yellow butterflies (tuning image) | PASS | PASS | FAIL | PASS |
 | 1 blue butterflies, 3 yellow butterflies | PASS | PASS | FAIL | FAIL |
Could it grade anything else?
We first asked it every question in the benchmark directly, on 8 models and 1,533 images. As a general judge it does not work: it answered FAIL only 16 times and gave no verdict on 321. Where agreement looked high, it was just passing images that mostly pass.
So we built tests around what it can see, then labeled disagreements blind again. For Writing accuracy, OCR words had to contain the expected words. For Negation, zero boxes on the forbidden thing meant PASS. For Attributes Binding, every phrase needed a box.
| Test | Images | Disagreements | WeVisOne right | Live judge right | Unsure | p | Result |
|---|---|---|---|---|---|---|---|
| Counting, detection | 728 | 36 | 11 | 13 | 6 | 0.84 | Tie |
| Counting, mask + color | 728 | 43 | 11 | 19 | 9 | 0.20 | Tie |
| Negation Easy + Medium, detection | 486 | 25 | 9 | 13 | 3 | 0.52 | Tie |
| Writing Easy + Medium, OCR | 486 | 42 | 11 | 27 | 4 | 0.014 | Gemini |
| Writing Hard + Extreme, OCR (sample of 20) | 484 | 132 | 2 | 17 | 1 | 0.0007 | Gemini |
| Attributes Binding, detection (sample of 20) | 729 | 104 | 3 | 13 | 4 | 0.021 | Gemini |
Counting rows leave out the 10 tuning images, so their scores cover 30 and 39 disagreements. Sample rows label 20 random disagreements, so they show a direction, not a final score.
Negation is the cleanest fit, and it tied. The two judges failed differently: WeVisOne wrongly failed 13 images, often from one false box on a person or on food, and Gemini wrongly failed the other 9. Writing accuracy lost: OCR misses short words such as “a”, and word matching passes text a person would reject. Attributes Binding leaned clearly against it. In 94 of 104 disagreements WeVisOne passed what Gemini failed, and the sample suggests a box on a “small blue triangle” is weak evidence that it is small or blue.
Why we are not switching
Moving Counting and easy Negation to detection would cost no measurable accuracy. It would flip 61 of 1,214 verdicts, and no model would move more than 3 images out of 192. We still decided against it. A benchmark judge should change because the new one is better, and nothing here beat Gemini. The gain would also be partial: 15 of 192 prompts would become reproducible, Gemini would still grade most of the benchmark, and those 15 prompts would lose their written explanations. Every new model would need a third judge.
So the leaderboard stays on V1.2. What changes is what we know: on these labels we could not measure a difference between a 4B open detector and a frontier model on counting.
Honest caveats
- A tie is not equivalence. With 24 scored disagreements on Counting, the test can only detect a large gap. “No detectable difference” is the claim, not “equally accurate”.
- One reviewer, and some images are ambiguous. All labels come from one person. Unsure images are excluded, and they were not rare on the Counting disagreements. Refracted fish in a bowl are a judgment call for a person too.
- Only disagreements were fully labeled. Where both judges agree, both may be wrong. These tests compare the judges; they do not measure absolute accuracy.
- Masks were chosen on a small hard set. Our own favorite lost on fresh images, and the 10-image table above flattered it. That is why we keep tuning images out of the score.
- The goldfish result is post hoc. We noticed the 5 to 1 after seeing the results, on 6 images. It is a lead for later, not a finding.
- Some steps ran outside our normal pipeline. Masks need Tencent’s own package, so the mask runs and the 10-image methods study ran directly on our GPU machine. The full-board detection, OCR and direct questions ran through the benchmark’s batch server. Plain detection matched across the two routes on 9 of the 10 images.
The data is open
The dataset on Hugging Face has the 203 human labels, 4,932 rows of judge verdicts with raw answers, the methods study, the finger counts and the plans we froze before each test. If you think a prompt, rule or label is wrong, you can check it image by image.