VGI-Bench Leaderboard

Model comparison

Most Accurate

ByteDance Seed: Seed 1.6

66.2%

Best Value

ByteDance Seed: Seed 1.6

$0.039/question

Fastest

ByteDance Seed: Seed 1.6

64s

Accuracy

Representative-run accuracy, best first.

Cost per question

Average cost per question, cheapest first.

Time per question

Average wall-clock time per question, fastest first.

Cost efficiency

Accuracy vs. cost (Pareto frontier)

One point per model, using default routing (not pinned to a provider) when available. The line is the Pareto frontier: no model beats these on both accuracy and cost.

Leaderboard

Accuracy over the public split (439 questions, runs need at least 395 answered to be listed), with cost, time, and input and output tokens per question for each provider endpoint the model was run on. Scores aggregate successful runs from the newest 90 days.

VGI-Bench leaderboard: model accuracy, cost, time, and input and output tokens per question by provider
#	Model		Std dev
1	ByteDance Seed: Seed 1.6 Pareto	66.2%	±1.0pp	$0.039	64s	141k	2.03k
2	Qwen: Qwen3.8 Omni Flash Alibaba	65.5%	--	$0.046	4.6m	412k	11.5k
3	ByteDance Seed: Seed-2.0-Lite Seed	63.7%	±4.8pp	$0.073	2.5m	241k	6.5k
4	Qwen: Qwen3.8 Max (0902) Alibaba	63.7%	--	$0.55	8.6m	613k	20.1k
5	Meta: Muse Wonder Meta	63.6%	--	$1.03	3.1m	233k	8.62k
6	Google: Gemini 2.5 Pro Google AI Studio	61.4%	±0.7pp	$0.061	1.8m	341	6.04k
7	Meta: Muse Spark 1.3 Meta	61.1%	--	$0.22	4.2m	227k	14.3k
8	Qwen: Qwen3.8 Max (0803) Alibaba	58.0%	--	$0.33	8.2m	579k	13.6k
9	Meta: Muse Ember Pareto Meta	57.4%	--	$0.023	1.6m	231k	9.18k
10	Crouton Alpha Stealth	56.8%	--	--	1.9m	313	102
11	Qwen: Qwen3.8 Flash Alibaba	56.8%	--	$0.11	6.2m	1.52M	24.1k
12	Google: Gemini 3 Flash Preview Google AI Studio	54.8%	±0.5pp	$0.14	4.7m	111k	40.2k
13	Qwen: Qwen3.7 Flash Alibaba	54.8%	±0.5pp	$0.27	5.6m	1.48M	10.9k
14	Meta: Muse Spark 1.1 Meta	54.2%	±0.5pp	$0.24	2.6m	151k	21.8k
15	Meta: Muse Spark Open	52.8%	±2.3pp	$0.16	3.7m	98.6k	14.1k
16	Google: Gemma 4 31B Pareto	51.3%	--	$0.008	6.0m	7.53k	13.7k
17	Meta: Muse Spark 1.2 Meta	51.0%	±0.6pp	$0.16	2.0m	150k	15.3k
18	Qwen: Qwen3.6 Plus Alibaba	50.6%	±0.4pp	$0.21	7.1m	532k	20.9k
19	Google: Gemini 2.5 Flash Google AI Studio	50.0%	±0.3pp	$0.024	1.9m	341	9.42k
20	Google: Gemini 2.5 Flash Lite Google AI Studio	48.0%	--	$0.012	2.4m	340	29.7k
21	Qwen: Qwen3.8 27B	47.6%	±1.9pp	$0.026	2.3m	36.7k	7.15k
22	Xiaomi: MiMo-V2.5 Xiaomi	46.5%	±2.9pp	$0.018	6.4m	272k	13.2k
23	Perceptron: Perceptron Mk1 Perceptron	46.2%	±0.1pp	$0.008	2.7m	46.4k	817
24	Qwen: Qwen3.5-Flash Alibaba	46.2%	±1.2pp	$0.10	4.9m	1.49M	20.2k
25	Xiaomi: MiMo-X-Pro Preview Xiaomi	45.6%	--	$0.027	4.6m	77.4k	14.7k
26	Qwen: Qwen3.5 397B A17B	45.5%	±1.0pp	$0.95	8.6m	1.45M	21.4k
27	Qwen: Qwen3.5 Plus 2026-02-15 Alibaba	45.5%	±0.7pp	$0.17	7.5m	532k	21.3k
28	Xiaomi: MiMo-X-Flash Preview Xiaomi	45.4%	--	$0.034	5.2m	337k	11.9k
29	Qwen: Qwen3.6 Flash Alibaba	45.2%	±0.8pp	$1.14	6.7m	1.49M	19.4k
30	Qwen: Qwen3.5 Plus 2026-04-20 Alibaba	45.1%	±0.6pp	$0.17	4.9m	440k	18.6k
31	ByteDance Seed: Seed-2.0-Mini Seed	43.4%	±2.9pp	$0.027	1.9m	232k	9.34k
32	Xiaomi: MiMo-V2.6-Pro-UltraSpeed Xiaomi	43.4%	--	$0.34	3.2m	77k	21.1k
33	Z.ai: GLM 5.3 FlashX Novita	42.3%	--	$0.19	3.5m	515k	7.53k
34	Xiaomi: MiMo-V2.6-Pro	40.9%	±7.2pp	$0.70	4.4m	38.3k	81k
35	Xiaomi: MiMo-V2.6-Flash	34.8%	±2.9pp	$0.021	7.5m	199k	16.4k
36	Z.ai: GLM 5.3 Flash	34.6%	±2.7pp	$0.028	8.3m	58.2k	43.9k
37	MiniMax: MiniMax M3	30.1%	±2.6pp	$0.081	9.8m	53.4k	75.3k
38	Perceptron: Perceptron Mk1.5 Perceptron	28.9%	--	$0.019	1.6m	49k	7.48k
39	NVIDIA: Nemotron 3.5 Super VL	28.5%	±0.8pp	$0.016	1.9m	14.6k	30.3k
40	Google: Gemma 4 26B A4B Pareto	24.7%	±7.3pp	$0.004	2.8m	4.59k	9.91k
41	MiniMax: MiniMax M3.1 Flash	22.2%	--	$0.006	44s	130k	3.99k
42	Reka Edge Pareto Reka	21.4%	±0.2pp	$0.00015	31s	1.49k	30.5
43	inclusionAI: Ling 3.0 Flash VL	19.2%	--	$0.001	1.8m	5.71k	9.46k
44	Google: Gemini 4 Argon	15.5%	--	$0.050	44s	1.58k	4.7k
45	Google: Gemini 4 Carbon (EAP) Google AI Studio	14.7%	--	$0.32	3.0m	1.63k	15.5k
46	Google: Gemini 3.8 Flash	14.4%	--	$0.039	76s	1.42k	10.3k
47	Elorian: Glimpse Together	13.8%	--	$0.022	47s	6.28k	6.46k
48	Qwen: Qwen3.5-122B-A10B Alibaba	12.8%	±0.4pp	$0.023	3.9m	38k	6.22k
49	Google: Gemini 4 Hydrogen	12.7%	±0.0pp	$0.068	51s	1.58k	8.24k
50	Google: Gemini 3.7 Flash Google	12.4%	--	$0.013	24s	1.19k	3.29k
51	Qwen: Qwen3.5-35B-A3B	12.4%	±0.4pp	$0.015	56s	38.3k	6.56k
52	Z.ai: GLM 5V Turbo Z.AI	12.3%	±0.5pp	$0.13	6.9m	124k	2.42k
53	Qwen: Qwen3.6 27B	11.8%	±0.4pp	$0.031	3.9m	32.7k	5.87k
54	MoonshotAI: Kimi K3.1 Pro	11.8%	--	$0.041	82s	5.06k	3.17k
55	Google: Gemini 3.6 Flash Google AI Studio	11.5%	--	$0.016	24s	1.39k	3.97k
56	Google: Gemini 4 Carbon	11.4%	--	$0.11	1.9m	1.58k	10.6k
57	Google: Gemini 3.1 Pro Preview Google AI Studio	10.9%	--	$0.098	59s	1.39k	7.96k
58	Qwen: Qwen3.6 35B A3B	10.8%	±0.3pp	$0.005	44s	4.57k	5.08k
59	Google: Gemini 3.1 Pro Preview Custom Tools Google AI Studio	10.6%	--	$0.098	60s	1.39k	8.02k
60	Google: Gemini 3.5 Flash Google AI Studio	10.1%	--	$0.088	39s	1.38k	9.59k
61	Google: Gemini 3.5 Flash Lite Google AI Studio	8.4%	--	$0.006	13s	1.39k	2.08k
62	Google: Gemini 3.1 Flash Lite Google AI Studio	8.2%	--	$0.038	58s	1.39k	25.1k
63	Google: Gemini 3.1 Flash Lite Preview Google AI Studio	8.2%	--	$0.038	58s	1.38k	25.1k
64	Google: Gemini 4 Hydrogen (EAP) Google AI Studio	6.4%	--	$0.26	57s	1.57k	12.9k
65	MoonshotAI: Kimi K3	6.3%	±5.7pp	$0.025	36s	5.46k	1.69k
66	Z.ai: GLM 4.6V Z.AI	5.6%	±4.3pp	$0.017	1.9m	69.2k	804
67	Qwen: Qwen3.5-9B	3.3%	±1.5pp	$0.003	7.4m	612	21.4k

Trajectories

The reasoning trace is the model's perception: the moments it says it saw and the timestamps it attaches to them. Reading it against the video shows how the answer was reached, and on the failed runs, where the description parts from the footage.

correct

Which window the pointer hovers over most

Happy path: The model locates the moment and reads it correctly.

Question 1187 global-graph/interaction_partner Qwen3.8 27B

The full video the model was sent, served from the OpenRouter mirror used by the harness.

Which option does the mouse pointer hover over most often?

A)Windhawk application windowkey, model
B)Install button
C)Adobe Premiere Pro window
D)Windows taskbar

answered A ✓key A

The trace tracks the pointer across the whole recording rather than one frame, names the Premiere Pro cameo near the end as a distractor, and lands on the window that dominates the session.

The question asks which option the mouse pointer hovers over most often. Let me think about the video content. The video is a tutorial about Windhawk, a customization tool for Windows. Throughout the video, the user is constantly interacting with the Windhawk application window - browsing mods, installing them, configuring settings, etc. Let me consider each option: - A) Windhawk application window - This is the main focus of the video. The user opens Windhawk, browses mods, clicks on details, settings, installs mods, etc. The mouse is frequently over this window. - B) Install button - The install button is clicked a few times but not hovered over most often. - C) Adobe Premiere Pro window - This appears only briefly near the end (around 13:20-13:40) as a joke about the title bar color. - D) Windows taskbar - The taskbar is shown but the mouse isn't constantly hovering over it. The dominant interaction is with the Windhawk application window. The mouse pointer is over it for the majority of the video as the user navigates through mods, settings, etc. So the answer is A. Let me double check by reviewing timestamps. From 1:20 onwards, the Windhawk window is open and the mouse moves around it constantly - clicking Mods, Explore, scrolling, clicking Details, Settings, etc. This continues through most of the video. The taskbar is shown at times but the mouse hovers over the Windhawk window more. Thus A is correct.

incorrect

Object co-present with a garden shed

Failure mode: Salient object over the key

Question 1862 global-graph/co_presence Qwen3.8 27B

The full video the model was sent, served from the OpenRouter mirror used by the harness.

Which option is repeatedly visible at the same time as the wooden garden shed during the video?

A)presenter with short curly hairkey
B)metal shed hinge
C)wagon wheelmodel
D)glass doors

answered C ✗key A

The trace cites eight timestamps for a wagon wheel leaning against the shed and dismisses the presenter as incidental. The human-validated key is the presenter. The model chose the most distinctive object in the frame over the one the question was scored on.

The question asks which option is repeatedly visible at the same time as the wooden garden shed during the video. Let me look at the frames where a wooden garden shed appears. At around 3:57-4:00, there's a wooden garden shed with a wagon wheel leaning against it, and the presenter is sitting in front of it. At 4:57-5:15, the same wooden shed with the wagon wheel is visible while the presenter talks about the slab base. So the wooden garden shed appears together with the wagon wheel in multiple segments. Let me check the other options: - A) presenter with short curly hair - the presenter is visible in many places, but the question is about what's visible at the same time as the wooden shed. The presenter is there too, but the wagon wheel is a distinctive object attached to the shed scene. - B) metal shed hinge - not particularly visible. - C) wagon wheel - yes, clearly visible leaning against the wooden shed in multiple frames (3:57, 4:00, 4:57, 5:00, 5:03, 5:07, 5:11, 5:15). - D) glass doors - those are on a different building (the dark shed with white doors). The wagon wheel is repeatedly visible alongside the wooden garden shed. The answer is C. Let me double check the frames. At 3:57 the presenter sits against a wooden shed with a large wagon wheel on the right. At 4:57-5:15 same scene. So wagon wheel is repeatedly visible with the wooden shed. Thus answer C.

incorrect

A false-premise question about a measuring spoon

Failure mode: Narrates a moment that never happens

Question 1096 contrastive-hard-negative/false-premise-visual-attribute Qwen3.8 27B

The full video the model was sent, served from the OpenRouter mirror used by the harness.

When the speaker says she used a full tablespoon of salt, where is the red measuring spoon positioned relative to the cheese curds?

A)This does not happen in the video.key
B)It is resting beside the bowl of cheese curds.
C)It is positioned just in front of the cheese curds.model
D)It is laid across the top of the cheese curds.

answered C ✗key A

The key says this moment does not occur. The trace narrates a 2:38 to 2:44 sequence with the spoon set down in front of the curds, rejects the 'does not happen' option outright, and answers with detail the key does not support.

The user wants me to look at the video and determine the position of the red measuring spoon relative to the cheese curds when the speaker mentions using a full tablespoon of salt. Let me trace through the video. Around 2:38-2:42, the speaker is adding salt to the cheese curds in a cloth-lined pot. She holds a red measuring spoon with salt in it. At 2:42, she sets the red spoon down. Let me look at where it ends up. At 2:42-2:44, after adding the salt, she places the red spoon down in front of the bowl/curds. Looking at the frame at 2:44, the red spoon is lying on the counter just in front of the cheese curds (the cloth-lined pot). Then she picks up the wooden spoon to stir. Let me check the options: A) Does not happen - but it does happen, she uses a tablespoon of salt. B) Resting beside the bowl - "beside" could be left/right. C) Positioned just in front of the cheese curds. D) Laid across the top. Looking at the frame at 2:44, the red spoon is at the bottom of the frame, in front of the curds. So "just in front of the cheese curds" matches C. Let me double check the timestamp. At 2:38 she holds the red spoon with salt. At 2:42 she tips it in. At 2:44 the red spoon is resting on the counter at the front edge, in front of the curds. Yes. Thus answer C. I need to provide reasoning then final answer format.

incorrect

The sound of an M.2 heatsink locking

Failure mode: Answers an audio question from world knowledge

Question 2852 contrastive-hard-negative/visual_audio Qwen3.8 27B

The full video the model was sent, served from the OpenRouter mirror used by the harness.

When the two M.2 slots are highlighted in purple, what click is heard as the top heatsink locks back into place?

A)This does not happen in the videokey
B)A sharp metallic snap
C)A firm double-click
D)A soft plastic clickmodel

answered D ✗key A

The trace says 'I need to recall the audio' and then reasons from what PC build videos 'typically' sound like, never from this one. The key is 'does not happen'. The final response presents the guessed click as something heard in the video.

The user is asking about a specific audio detail in the video. Let me look at the relevant section. Around 02:20-02:30, the video shows the M.2 slots being highlighted in purple (the diagram at 02:20 shows the M.2 slots). Then the person is working with the heatsink. Let me think about the sequence. At 02:20 there's a diagram showing M.2 slots highlighted. Then at 02:30 the person is holding a black heatsink piece. At 02:40-02:50 they're working on the motherboard with the M.2 area. The question asks: when the two M.2 slots are highlighted in purple, what click is heard as the top heatsink locks back into place? I need to recall the audio. In these PC build videos, when you press down on the M.2 heatsink, there's typically a plastic click as it locks. The options are: sharp metallic snap, firm double-click, soft plastic click, or doesn't happen. Given the context of an M.2 heatsink locking into place, the typical sound is a soft plastic click. The heatsink is plastic/metal but the locking mechanism is usually a plastic clip. Let me consider the diagram at 02:20 - it shows the M.2 slots with purple/magenta highlighting. Then the person installs the SSD and the heatsink. The click when the heatsink locks would be a soft plastic click. I'll go with D, soft plastic click. Let me double check by reasoning about the video flow. The diagram with purple highlights appears at 02:20. Then at 02:30-02:50 the person is handling the heatsink and the M.2 area. The locking click of an M.2 heatsink is generally a soft plastic click. Thus answer D.

incorrect

Where a whiskey bottle sits when a line is spoken

Failure mode: Never locates the spoken cue

Question 2894 contrastive-hard-negative/transcript_visual Qwen3.8 27B

The full video the model was sent, served from the OpenRouter mirror used by the harness.

When he says, “This whole time he's back there drinking whiskey,” is the whiskey bottle to the left or right of the laptop on Craig’s bench?

A)This does not happen in the video.key
B)To the left of the laptop.model
C)To the right of the laptop.
D)Directly behind the laptop.

answered B ✗key A

The cue is a spoken line. The trace never finds it, says it 'probably occurs during the conversation at the bench', and answers from a frame at 60 to 70 seconds. The key is 'does not happen'. Without the transcript anchor, the model substituted the nearest matching scene.

The user is asking about a specific moment in the video where someone says "This whole time he's back there drinking whiskey." I need to find this moment and look at the position of the whiskey bottle relative to the laptop on Craig's bench. Let me look at the early part of the video. Around 60-70 seconds, there's a scene with a man sitting at a bench with a laptop, and there's a bottle on the bench. Let me look carefully. At around 60-70 seconds, there's a man in a light blue shirt sitting in a chair at a bench with a laptop. On the bench, to the left of the laptop (from the viewer's perspective), there's a bottle (looks like a whiskey bottle) and a mug. Wait, I need to find the audio line "This whole time he's back there drinking whiskey." This is likely during the scene where two people are at the bench with the laptop. Let me look at the timestamps around 60-130 seconds. At 60-70s: man in blue shirt at bench with laptop, bottle on left side of bench. At 120-130s: close up of laptop showing a 3D model of a table, with a hand typing. The line about drinking whiskey probably occurs during the conversation at the bench. The bottle is on the bench. Looking at the frame at 60-70s, the bottle is to the left of the laptop (viewer's left). The laptop is on the right side of the bench, the bottle and mug are on the left side. So from the viewer's perspective, the whiskey bottle is to the left of the laptop. Thus the answer is B. Let me double check the orientation. The question asks "to the left or right of the laptop on Craig's bench?" In the image at 60-70s, the laptop is on the right portion of the bench, the bottle is on the left portion. So the bottle is to the left of the laptop. Therefore answer B. I'm fairly confident. The line "This whole time he's back there drinking whiskey" matches the scene where one person is at the laptop and another is "back there" drinking. The bottle visible on the bench is to the left of the laptop. Thus B.

VGI-Bench

Model comparison

Cost efficiency

Leaderboard

Trajectories

Which window the pointer hovers over most

Perception (reasoning trace)

Response

Object co-present with a garden shed

Perception (reasoning trace)

Response

A false-premise question about a measuring spoon

Perception (reasoning trace)

Response

The sound of an M.2 heatsink locking

Perception (reasoning trace)

Response

Background behind a PSU cable being plugged in

Perception (reasoning trace)

Response

Where a whiskey bottle sits when a line is spoken

Perception (reasoning trace)

Response