ImageBench

ImageBench V1 —

192 evaluations across 6 categories

Benchmark V1 verdicts are produced by VLM judges and can contain mistakes. Treat PASS/FAIL labels as machine-assisted assessments, and inspect the images yourself. Learn more about the methodology.

Generation Details

Source-backed model context, size, cost, and request settings for this ImageBench V1 run.

local/qwen-image-2.1-7b-nf4

Local

Qwen-Image-2.1 is Alibaba Qwen's open-weight text-to-image and image-edit model, released 2026-09-20 under the non-commercial Qwen Research License. It pairs a 32-layer single-stream 7B DiT with a Qwen3-VL 8B text encoder and a 64-channel RGBA autoencoder, so transparent cutouts need no matting pass. This entry benchmarks the NF4 configuration: the 7B diffusion transformer runs at full BF16 precision and only the 8B text encoder is quantised to NF4 (bitsandbytes). Quantising just the encoder cuts the pipeline from roughly 31 GiB to 20.5 GiB, which fits a 32 GB consumer card without CPU offload. Image quality therefore comes from an unquantised denoiser; NF4 affects only how the prompt is encoded.

Maker
Alibaba / Qwen
Family
Qwen-Image
Model Size
7B DiT (BF16) + 8B text encoder (NF4)
high
Cost
local run; no API price
not_applicable
Run Target
5090/Qwen/Qwen-Image-2.1
Effective Request
response_format: b64_json · size: 1024x1024 · num_inference_steps: 40 · true_cfg_scale: 1 · seed: 7
54.6
Overall
64%
Capability
45.1
Est. Preference
123
Pass
69
Fail
14.6s
Avg Latency
14.4s
Min Latency
15.3s
Max Latency
Text Rendering93%Spatial Reasoning67%Human realism48%Truthfulness52%Professional Studio85%Graphical design58%Preference45%Latency21%

All 192 generations

Text Rendering93%

Typography Style100%

Writing accuracy92%

Spatial Reasoning67%

Attributes Binding67%

Compositionality89%

Counting56%

Negation44%

Relative Position75%

Scale & Proportions67%

Human realism48%

Faces & Expressions67%

Full Body0%

Hands83%

Multi-Subject33%

Truthfulness52%

Photorealism33%

Physics & Reflections67%

World Knowledge42%

Professional Studio85%

Camera & Lighting83%

Color Precision92%

Photorealism67%

Graphical design58%

Data Visualisation0%

Layout & Design33%

Style Diversity92%