Benchmark rankings

80 models. Latest benchmark run: 2026-09-10. This snapshot is available without JavaScript.

All benchmark models, ordered by F1. Confidence intervals are calculated in the interactive view.
Rank	Model / route / shots	F1	95% CI	Precision	Recall	Eval date	Date released
1	GPT-6 Astra · OpenRouter · 2-shot	0.7778	See interactive view	0.7559	0.8010	2026-09-06	2026-09-03
2	Claude Fable 5.1* · OpenRouter · 2-shot	0.7316	See interactive view	0.7000	0.7662	2026-09-06	2026-09-01
3	Claude Opus 5* · OpenRouter · 2-shot	0.6904	See interactive view	0.6645	0.7185	2026-09-06	2026-07-24
4	Claude Fable 5* · 2-shot	0.6579	See interactive view	0.6274	0.6915	2026-07-23	2026-06-09
5	Kimi K3 · OpenRouter · 2-shot	0.6496	See interactive view	0.6684	0.6318	2026-07-23	2026-07-16
6	GPT-5.4 · 2-shot	0.6219	See interactive view	0.6452	0.6002	2026-03-05	2026-03-05
7	Gemini 3.6 Flash · OpenRouter · 2-shot	0.6197	See interactive view	0.6166	0.6228	2026-07-23	2026-07-21
8	GPT-5.5 Pro · OpenRouter · 2-shot	0.6078	See interactive view	0.5990	0.6169	2026-04-26	2026-04-24
9	Muse Spark 1.3 · OpenRouter · 2-shot	0.6078	See interactive view	0.5990	0.6169	2026-09-10	2026-09-02
10	Gemini 3.8 Flash* · OpenRouter · 2-shot	0.5954	See interactive view	0.6094	0.5821	2026-09-06	2026-09-02
11	GPT-5.5 · 2-shot	0.5820	See interactive view	0.5777	0.5864	2026-04-26	2026-04-23
12	GPT-5.6 Sol · 2-shot	0.5811	See interactive view	0.5660	0.5970	2026-07-23	2026-07-09
13	Kimi K2.6 Nitro · OpenRouter · 2-shot	0.5714	See interactive view	0.6317	0.5217	2026-04-26	2026-04-20
14	GPT-5.3 Codex · OpenRouter · 2-shot	0.5699	See interactive view	0.6105	0.5344	2026-03-05	2026-02-05
15	Grok 4.6 · OpenRouter · 2-shot	0.5521	See interactive view	0.5377	0.5672	2026-09-06	2026-08-12
16	Claude Sonnet 5* · 2-shot	0.5347	See interactive view	0.5475	0.5225	2026-07-23	2026-06-30
17	GPT-5.6 Terra · 2-shot	0.5284	See interactive view	0.5467	0.5113	2026-07-23	2026-07-09
18	Gemini 3.5 Flash · OpenRouter · 2-shot	0.5250	See interactive view	0.5277	0.5224	2026-07-23	2026-05-19
19	GPT-5.6 Luna · 2-shot	0.5076	See interactive view	0.5181	0.4975	2026-07-23	2026-07-09
20	Muse Spark 1.1 · OpenRouter · 2-shot	0.4875	See interactive view	0.4937	0.4815	2026-07-23	2026-07-16
21	Claude Opus 4.7* · 2-shot	0.4805	See interactive view	0.4882	0.4730	2026-04-26	2026-04-16
22	GPT-5.2 · OpenRouter · 2-shot	0.4791	See interactive view	0.5297	0.4374	2026-02-22	2025-12-11
23	Kimi K2.5 · OpenRouter · 2-shot	0.4751	See interactive view	0.5092	0.4453	2026-02-22	2025-07-11
24	Kimi K2.7 Code · OpenRouter · 2-shot	0.4611	See interactive view	0.5506	0.3966	2026-07-23	2026-06-12
25	Grok 4.5 · OpenRouter · 2-shot	0.4416	See interactive view	0.4450	0.4382	2026-07-23	2026-07-08
26	Claude Opus 4.5* · 2-shot	0.4141	See interactive view	0.4090	0.4194	2026-03-08	Nov 2025
27	GPT-5 · 2-shot	0.4067	See interactive view	0.4838	0.3508	2026-02-18	2025-08-07
28	Qwen3.6 Plus Nitro · OpenRouter · 2-shot	0.3797	See interactive view	0.4678	0.3195	2026-04-26	2026-04-02
29	Qwen3.8 Max (0902)* · OpenRouter · 2-shot	0.3725	See interactive view	0.3640	0.3814	2026-09-06	2026-09-03
30	Muse Glimmer 30B · OpenRouter · 2-shot	0.3663	See interactive view	0.4164	0.3269	2026-09-10	2026-08-09
31	Qwen3.7 Plus · OpenRouter · 2-shot	0.3575	See interactive view	0.3653	0.3500	2026-07-23	2026-05-31
32	Claude Opus 4.6* · 2-shot	0.3269	See interactive view	0.3244	0.3294	2026-03-08	2026-02-05
33	Qwen3.5 397B A17B · OpenRouter · 2-shot	0.3081	See interactive view	0.3248	0.2930	2026-02-22	2026-02-17
34	Claude Sonnet 4.6* · 2-shot	0.2882	See interactive view	0.3130	0.2670	2026-03-08	2026-02-17
35	Gemini 3.1 Pro Preview · OpenRouter · 2-shot	0.2493	See interactive view	0.2434	0.2554	2026-02-22	2026-02-19
36	Claude Opus 4.1* · 2-shot	0.2318	See interactive view	0.2236	0.2407	2026-03-08	2025-08-05
37	Claude Opus 4.8* · 2-shot	0.2150	See interactive view	0.2113	0.2188	2026-07-23	2026-05-28
38	GPT-4o · 2-shot	0.2123	See interactive view	0.2589	0.1799	2026-02-22	2024-05-13
39	GPT-4o (2024-11-20 OR) · OpenRouter · 2-shot	0.2085	See interactive view	0.2662	0.1713	2026-02-22	2024-11-20
40	Inkling · OpenRouter · 2-shot	0.1962	See interactive view	0.2248	0.1741	2026-07-23	2026-07-17
41	GPT-5.3 Chat · 2-shot	0.1958	See interactive view	0.2121	0.1818	2026-03-05	2026-03-03
42	GPT-o4 Mini · 2-shot	0.1895	See interactive view	0.2382	0.1573	2026-02-22	2025-04-16
43	GPT-4.1 · 2-shot	0.1773	See interactive view	0.2243	0.1465	2026-02-22	2025-04-14
44	ChatGPT-4o · OpenRouter · 2-shot	0.1726	See interactive view	0.2043	0.1494	2026-02-22	2024-05-13
45	Step 3.7 Flash · OpenRouter · 2-shot	0.1652	See interactive view	0.1587	0.1723	2026-07-23	2026-05-28
46	GPT-4.5 Preview · OpenRouter · 2-shot	0.1443	See interactive view	0.1424	0.1462	2026-02-22	2025-02-27
47	Gemini 3.1 Flash Lite Preview · OpenRouter · 2-shot	0.1354	See interactive view	0.1515	0.1224	2026-03-08	2026-03-03
48	GPT-4.5 Preview · 2-shot	0.1225	See interactive view	0.1209	0.1240	2026-02-22	2025-02-27
49	Gemma 4 31B · OpenRouter · 2-shot	0.1200	See interactive view	0.1616	0.0955	2026-04-04	2026-04-02
50	Gemini 3.5 Flash Lite · OpenRouter · 2-shot	0.1187	See interactive view	0.1243	0.1136	2026-07-23	2026-07-21
51	Claude Sonnet 4.5* · 2-shot	0.0986	See interactive view	0.0874	0.1130	2026-03-08	2025-09-29
52	Mistral Medium 3.5 · OpenRouter · 2-shot	0.0871	See interactive view	0.0733	0.1072	2026-07-23	2026-04-30
53	GLM 4.6V · OpenRouter · 2-shot	0.0837	See interactive view	0.1447	0.0588	2026-02-22	2025-12-08
54	MiniMax M3 · OpenRouter · 2-shot	0.0734	See interactive view	0.0941	0.0602	2026-07-23	2026-05-28
55	DeepSeek V4 Flash Vision Exp* · OpenRouter · 2-shot	0.0578	See interactive view	0.1055	0.0398	2026-09-06	2026-08-21
56	Llama 4 Maverick · OpenRouter · 2-shot	0.0485	See interactive view	0.0525	0.0451	2026-02-22	2025-04-05
57	Claude Haiku 4.5* · 2-shot	0.0459	See interactive view	0.0554	0.0392	2026-03-08	2025-10-15
58	Mistral Small 3.1 24B · OpenRouter · 2-shot	0.0443	See interactive view	0.0511	0.0391	2026-02-22	2025-03-17
59	GPT-4 Turbo · 2-shot	0.0437	See interactive view	0.0561	0.0358	2026-02-22	2023-11-06
60	GPT-4o · OpenRouter · 2-shot	0.0397	See interactive view	0.0496	0.0331	2026-02-22	2023-11-06
61	GPT-4.1 Mini · 2-shot	0.0354	See interactive view	0.0380	0.0332	2026-02-22	2025-04-14
62	Gemma 3 27B · OpenRouter · 2-shot	0.0341	See interactive view	0.0355	0.0329	2026-02-22	2025-03-12
63	Qwen VL Max · OpenRouter · 2-shot	0.0335	See interactive view	0.0434	0.0273	2026-02-22	2024-01-25
64	GPT-4o Mini · 2-shot	0.0299	See interactive view	0.0314	0.0285	2026-02-22	2024-07-18
65	Qwen2.5 VL 72B Instruct · OpenRouter · 2-shot	0.0208	See interactive view	0.0271	0.0169	2026-02-22	2025-01-26
66	Google: Gemini 2.0 Flash · OpenRouter · 2-shot	0.0183	See interactive view	0.0236	0.0149	2026-02-22	2025-02-05
67	Llama 4 Scout · OpenRouter · 2-shot	0.0169	See interactive view	0.0227	0.0134	2026-02-22	2025-04-05
68	Gemma 4 26B A4B IT · OpenRouter · 2-shot	0.0126	See interactive view	0.0157	0.0105	2026-04-04	2026-04-02
69	GPT-4o-mini · OpenRouter · 2-shot	0.0076	See interactive view	0.0085	0.0069	2026-02-22	2024-07-18
70	Mistral Large 3 2512 · OpenRouter · 2-shot	0.0072	See interactive view	0.0065	0.0080	2026-02-22	2025-12-02
71	GPT-4.1 Nano · 2-shot	0.0051	See interactive view	0.0058	0.0045	2026-02-22	2025-04-14
72	DeepSeek R1 · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-01-20
73	DeepSeek V3 0324 · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-03-25
74	Gemini 2.5 Pro Preview 3/25 · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-03-25
75	Grok 4 · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-07-09
76	Grok 4 Fast · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-09-25
77	Grok 4.1 Fast · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2026-01-29
78	Llama 3.2 90B Vision Instruct · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2024-09-25
79	Phi 4 Multimodal Instruct · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	Feb 2025
80	Qwen VL Plus · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2024-01-25

Scores pool edit-level true positives, false positives, and false negatives across evaluated pages and runs. Higher is better. Small score differences may not be meaningful; use the interactive F1 confidence intervals when comparing models.

* Claude-family images are resized and recompressed as needed for Anthropic’s image limit. Other model-specific caveats appear in the interactive view.

Model-specific caveats

Claude Fable 5.1* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 5* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Fable 5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Gemini 3.8 Flash* · OpenRouter · 2-shot: Images are resized/recompressed when needed to stay below the provider's request payload limit.
Claude Sonnet 5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 4.7* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Qwen3.8 Max (0902)* · OpenRouter · 2-shot: Images are resized/recompressed when needed to stay below the provider's request payload limit.
Claude Opus 4.6* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Sonnet 4.6* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 4.1* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 4.8* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit. Opus 4.8's score is depressed by unreliable line counting: it identifies most edits correctly (166 of ~200 match ground truth by text) but reports noisy line numbers that drift progressively down the page and vary between runs, so the judge rejects the matches.
Claude Sonnet 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
DeepSeek V4 Flash Vision Exp* · OpenRouter · 2-shot: Incomplete run: 35 of 36 evaluation predictions completed; one request repeatedly timed out at the provider.
Claude Haiku 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.

Interactive leaderboard · Rankings and pricing in Markdown · Raw results JSON · Pricing JSON · Technical report