Benchmark rankings

94 models. Latest benchmark run: 2026-10-13. This snapshot is available without JavaScript.

All benchmark models, ordered by F1. Confidence intervals are calculated in the interactive view.
Rank	Model / route / shots	F1	95% CI	Precision	Recall	Eval date	Date released
1	GPT-6 Astra · OpenRouter · 2-shot	0.7778	See interactive view	0.7559	0.8010	2026-09-06	2026-09-03
2	Claude Opus 5.5* · OpenRouter · 2-shot	0.7524	See interactive view	0.7215	0.7861	2026-10-12	2026-09-22
3	GPT-6.1 Sol · OpenRouter · 2-shot	0.7416	See interactive view	0.7143	0.7711	2026-10-12	2026-09-29
4	Claude Fable 5.1* · OpenRouter · 2-shot	0.7316	See interactive view	0.7000	0.7662	2026-09-06	2026-09-01
5	Claude Opus 5* · OpenRouter · 2-shot	0.6904	See interactive view	0.6645	0.7185	2026-09-06	2026-07-24
6	Gemini 4 Argon* · OpenRouter · 2-shot	0.6843	See interactive view	0.6636	0.7065	2026-10-13	2026-10-13
7	Claude Fable 5* · 2-shot	0.6579	See interactive view	0.6274	0.6915	2026-07-23	2026-06-09
8	Kimi K3 · OpenRouter · 2-shot	0.6496	See interactive view	0.6684	0.6318	2026-07-23	2026-07-16
9	GPT-5.4 · 2-shot	0.6219	See interactive view	0.6452	0.6002	2026-03-05	2026-03-05
10	Gemini 3.6 Flash · OpenRouter · 2-shot	0.6197	See interactive view	0.6166	0.6228	2026-07-23	2026-07-21
11	Claude Sonnet 5.5* · OpenRouter · 2-shot	0.6125	See interactive view	0.6051	0.6202	2026-10-12	2026-09-28
12	GPT-5.5 Pro · OpenRouter · 2-shot	0.6078	See interactive view	0.5990	0.6169	2026-04-26	2026-04-24
13	Muse Spark 1.3 · OpenRouter · 2-shot	0.6078	See interactive view	0.5990	0.6169	2026-09-10	2026-09-02
14	Muse Wonder · OpenRouter · 2-shot	0.6077	See interactive view	0.5853	0.6318	2026-10-12	2026-10-12
15	Gemini 3.8 Flash* · OpenRouter · 2-shot	0.5954	See interactive view	0.6094	0.5821	2026-09-06	2026-09-02
16	GPT-5.5 · 2-shot	0.5820	See interactive view	0.5777	0.5864	2026-04-26	2026-04-23
17	GPT-5.6 Sol · 2-shot	0.5811	See interactive view	0.5660	0.5970	2026-07-23	2026-07-09
18	Kimi K2.6 Nitro · OpenRouter · 2-shot	0.5714	See interactive view	0.6317	0.5217	2026-04-26	2026-04-20
19	GPT-5.3 Codex · OpenRouter · 2-shot	0.5699	See interactive view	0.6105	0.5344	2026-03-05	2026-02-05
20	GPT-6 Luna · OpenRouter · 2-shot	0.5653	See interactive view	0.6092	0.5274	2026-10-12	2026-09-22
21	Grok 4.7 · OpenRouter · 2-shot	0.5640	See interactive view	0.5901	0.5401	2026-10-12	2026-09-21
22	Grok 4.6 · OpenRouter · 2-shot	0.5521	See interactive view	0.5377	0.5672	2026-09-06	2026-08-12
23	MiMo V2.6 Pro · OpenRouter · 2-shot	0.5473	See interactive view	0.5632	0.5323	2026-10-12	2026-09-21
24	Claude Sonnet 5* · 2-shot	0.5347	See interactive view	0.5475	0.5225	2026-07-23	2026-06-30
25	GPT-5.6 Terra · 2-shot	0.5284	See interactive view	0.5467	0.5113	2026-07-23	2026-07-09
26	Gemini 3.5 Flash · OpenRouter · 2-shot	0.5250	See interactive view	0.5277	0.5224	2026-07-23	2026-05-19
27	GPT-5.6 Luna · 2-shot	0.5076	See interactive view	0.5181	0.4975	2026-07-23	2026-07-09
28	Muse Spark 1.1 · OpenRouter · 2-shot	0.4875	See interactive view	0.4937	0.4815	2026-07-23	2026-07-16
29	Claude Opus 4.7* · 2-shot	0.4805	See interactive view	0.4882	0.4730	2026-04-26	2026-04-16
30	GPT-5.2 · OpenRouter · 2-shot	0.4791	See interactive view	0.5297	0.4374	2026-02-22	2025-12-11
31	Kimi K2.5 · OpenRouter · 2-shot	0.4751	See interactive view	0.5092	0.4453	2026-02-22	2025-07-11
32	GLM 5.3 FlashX · OpenRouter · 2-shot	0.4651	See interactive view	0.4736	0.4569	2026-10-12	2026-09-18
33	Kimi K2.7 Code · OpenRouter · 2-shot	0.4611	See interactive view	0.5506	0.3966	2026-07-23	2026-06-12
34	Grok 4.5 · OpenRouter · 2-shot	0.4416	See interactive view	0.4450	0.4382	2026-07-23	2026-07-08
35	Claude Opus 4.5* · 2-shot	0.4141	See interactive view	0.4090	0.4194	2026-03-08	Nov 2025
36	GPT-5 · 2-shot	0.4067	See interactive view	0.4838	0.3508	2026-02-18	2025-08-07
37	Qwen3.6 Plus Nitro · OpenRouter · 2-shot	0.3797	See interactive view	0.4678	0.3195	2026-04-26	2026-04-02
38	Qwen3.8 Max (0902)* · OpenRouter · 2-shot	0.3725	See interactive view	0.3640	0.3814	2026-09-06	2026-09-03
39	Mistral Large 4 Preview · OpenRouter · 2-shot	0.3682	See interactive view	0.4297	0.3221	2026-10-12	2026-10-06
40	Muse Glimmer 30B · OpenRouter · 2-shot	0.3663	See interactive view	0.4164	0.3269	2026-09-10	2026-08-09
41	Qwen3.7 Plus · OpenRouter · 2-shot	0.3575	See interactive view	0.3653	0.3500	2026-07-23	2026-05-31
42	Claude Opus 4.6* · 2-shot	0.3269	See interactive view	0.3244	0.3294	2026-03-08	2026-02-05
43	Qwen3.5 397B A17B · OpenRouter · 2-shot	0.3081	See interactive view	0.3248	0.2930	2026-02-22	2026-02-17
44	Claude Sonnet 4.6* · 2-shot	0.2882	See interactive view	0.3130	0.2670	2026-03-08	2026-02-17
45	Claude Haiku 5.5* · OpenRouter · 2-shot	0.2692	See interactive view	0.2712	0.2672	2026-10-12	2026-10-07
46	DeepSeek V4.1 Flash · OpenRouter · 2-shot	0.2621	See interactive view	0.2881	0.2403	2026-10-12	2026-09-10
47	Gemini 3.1 Pro Preview · OpenRouter · 2-shot	0.2493	See interactive view	0.2434	0.2554	2026-02-22	2026-02-19
48	Qwen3.8 Omni Flash · OpenRouter · 2-shot	0.2459	See interactive view	0.2446	0.2472	2026-10-12	2026-09-21
49	Claude Opus 4.1* · 2-shot	0.2318	See interactive view	0.2236	0.2407	2026-03-08	2025-08-05
50	Claude Opus 4.8* · 2-shot	0.2150	See interactive view	0.2113	0.2188	2026-07-23	2026-05-28
51	GPT-4o · 2-shot	0.2123	See interactive view	0.2589	0.1799	2026-02-22	2024-05-13
52	GPT-4o (2024-11-20 OR) · OpenRouter · 2-shot	0.2085	See interactive view	0.2662	0.1713	2026-02-22	2024-11-20
53	Inkling · OpenRouter · 2-shot	0.1962	See interactive view	0.2248	0.1741	2026-07-23	2026-07-17
54	GPT-5.3 Chat · 2-shot	0.1958	See interactive view	0.2121	0.1818	2026-03-05	2026-03-03
55	GPT-o4 Mini · 2-shot	0.1895	See interactive view	0.2382	0.1573	2026-02-22	2025-04-16
56	GPT-4.1 · 2-shot	0.1773	See interactive view	0.2243	0.1465	2026-02-22	2025-04-14
57	ChatGPT-4o · OpenRouter · 2-shot	0.1726	See interactive view	0.2043	0.1494	2026-02-22	2024-05-13
58	Step 3.7 Flash · OpenRouter · 2-shot	0.1652	See interactive view	0.1587	0.1723	2026-07-23	2026-05-28
59	GPT-4.5 Preview · OpenRouter · 2-shot	0.1443	See interactive view	0.1424	0.1462	2026-02-22	2025-02-27
60	Gemini 3.1 Flash Lite Preview · OpenRouter · 2-shot	0.1354	See interactive view	0.1515	0.1224	2026-03-08	2026-03-03
61	GPT-4.5 Preview · 2-shot	0.1225	See interactive view	0.1209	0.1240	2026-02-22	2025-02-27
62	Gemma 4 31B · OpenRouter · 2-shot	0.1200	See interactive view	0.1616	0.0955	2026-04-04	2026-04-02
63	Gemini 3.5 Flash Lite · OpenRouter · 2-shot	0.1187	See interactive view	0.1243	0.1136	2026-07-23	2026-07-21
64	Claude Sonnet 4.5* · 2-shot	0.0986	See interactive view	0.0874	0.1130	2026-03-08	2025-09-29
65	Step 5 Preview · OpenRouter · 2-shot	0.0952	See interactive view	0.0960	0.0945	2026-10-12	2026-09-20
66	Mistral Medium 3.5 · OpenRouter · 2-shot	0.0871	See interactive view	0.0733	0.1072	2026-07-23	2026-04-30
67	GLM 4.6V · OpenRouter · 2-shot	0.0837	See interactive view	0.1447	0.0588	2026-02-22	2025-12-08
68	MiniMax M3 · OpenRouter · 2-shot	0.0734	See interactive view	0.0941	0.0602	2026-07-23	2026-05-28
69	DeepSeek V4 Flash Vision Exp* · OpenRouter · 2-shot	0.0578	See interactive view	0.1055	0.0398	2026-09-06	2026-08-21
70	Llama 4 Maverick · OpenRouter · 2-shot	0.0485	See interactive view	0.0525	0.0451	2026-02-22	2025-04-05
71	Claude Haiku 4.5* · 2-shot	0.0459	See interactive view	0.0554	0.0392	2026-03-08	2025-10-15
72	Mistral Small 3.1 24B · OpenRouter · 2-shot	0.0443	See interactive view	0.0511	0.0391	2026-02-22	2025-03-17
73	GPT-4 Turbo · 2-shot	0.0437	See interactive view	0.0561	0.0358	2026-02-22	2023-11-06
74	GPT-4o · OpenRouter · 2-shot	0.0397	See interactive view	0.0496	0.0331	2026-02-22	2023-11-06
75	GPT-4.1 Mini · 2-shot	0.0354	See interactive view	0.0380	0.0332	2026-02-22	2025-04-14
76	Gemma 3 27B · OpenRouter · 2-shot	0.0341	See interactive view	0.0355	0.0329	2026-02-22	2025-03-12
77	Qwen VL Max · OpenRouter · 2-shot	0.0335	See interactive view	0.0434	0.0273	2026-02-22	2024-01-25
78	GPT-4o Mini · 2-shot	0.0299	See interactive view	0.0314	0.0285	2026-02-22	2024-07-18
79	Qwen2.5 VL 72B Instruct · OpenRouter · 2-shot	0.0208	See interactive view	0.0271	0.0169	2026-02-22	2025-01-26
80	Google: Gemini 2.0 Flash · OpenRouter · 2-shot	0.0183	See interactive view	0.0236	0.0149	2026-02-22	2025-02-05
81	Llama 4 Scout · OpenRouter · 2-shot	0.0169	See interactive view	0.0227	0.0134	2026-02-22	2025-04-05
82	Gemma 4 26B A4B IT · OpenRouter · 2-shot	0.0126	See interactive view	0.0157	0.0105	2026-04-04	2026-04-02
83	GPT-4o-mini · OpenRouter · 2-shot	0.0076	See interactive view	0.0085	0.0069	2026-02-22	2024-07-18
84	Mistral Large 3 2512 · OpenRouter · 2-shot	0.0072	See interactive view	0.0065	0.0080	2026-02-22	2025-12-02
85	GPT-4.1 Nano · 2-shot	0.0051	See interactive view	0.0058	0.0045	2026-02-22	2025-04-14
86	DeepSeek R1 · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-01-20
87	DeepSeek V3 0324 · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-03-25
88	Gemini 2.5 Pro Preview 3/25 · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-03-25
89	Grok 4 · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-07-09
90	Grok 4 Fast · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2025-09-25
91	Grok 4.1 Fast · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2026-01-29
92	Llama 3.2 90B Vision Instruct · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2024-09-25
93	Phi 4 Multimodal Instruct · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	Feb 2025
94	Qwen VL Plus · OpenRouter · 2-shot	0.0000	See interactive view	0.0000	0.0000	2026-02-22	2024-01-25

Scores pool edit-level true positives, false positives, and false negatives across evaluated pages and runs. Higher is better. Small score differences may not be meaningful; use the interactive F1 confidence intervals when comparing models.

* Claude-family images are resized and recompressed as needed for Anthropic’s image limit. Other model-specific caveats appear in the interactive view.

Model-specific caveats

Claude Opus 5.5* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Fable 5.1* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 5* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Gemini 4 Argon* · OpenRouter · 2-shot: Images are resized/recompressed when needed to stay below the provider's request payload limit.
Claude Fable 5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Sonnet 5.5* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Gemini 3.8 Flash* · OpenRouter · 2-shot: Images are resized/recompressed when needed to stay below the provider's request payload limit.
MiMo V2.6 Pro · OpenRouter · 2-shot: Native reasoning enabled; reasoning effort is controlled by the provider.
Claude Sonnet 5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 4.7* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Qwen3.8 Max (0902)* · OpenRouter · 2-shot: Images are resized/recompressed when needed to stay below the provider's request payload limit.
Claude Opus 4.6* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Sonnet 4.6* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Haiku 5.5* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Qwen3.8 Omni Flash · OpenRouter · 2-shot: Native reasoning enabled; reasoning effort is controlled by the provider.
Claude Opus 4.1* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
Claude Opus 4.8* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit. Opus 4.8's score is depressed by unreliable line counting: it identifies most edits correctly (166 of ~200 match ground truth by text) but reports noisy line numbers that drift progressively down the page and vary between runs, so the judge rejects the matches.
Claude Sonnet 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
DeepSeek V4 Flash Vision Exp* · OpenRouter · 2-shot: Incomplete run: 35 of 36 evaluation predictions completed; one request repeatedly timed out at the provider.
Claude Haiku 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.

Interactive leaderboard · Rankings and pricing in Markdown · Raw results JSON · Pricing JSON · Technical report