# Little Dorrit benchmark rankings

80 models. Latest benchmark run: 2026-09-10.

Scores pool edit-level TP, FP, and FN across evaluated pages and runs. Precision = TP / (TP + FP); recall = TP / (TP + FN); F1 is their harmonic mean. Higher is better. Zero-denominator scores are zero. F1 confidence intervals are calculated in the interactive site; small differences may not be meaningful.

Pricing snapshot: 2026-09-11 (UTC). Prices are USD per million tokens for the exact benchmark route. Standard uncached text rates; OpenRouter starting rates may vary by provider. Unknown means unverified, not free. These are not measured costs per page. Images, caching, context tiers, reasoning, and fees can change actual bills. Historical usage is unknown when not recorded.

| Rank | Model / route / shots | F1 | Precision | Recall | Input / 1M | Output / 1M | Evaluated | Released |
|---:|:---|---:|---:|---:|---:|---:|:---|:---|
| 1 | GPT-6 Astra · OpenRouter · 2-shot | 0.7778 | 0.7559 | 0.8010 | $10.00 | $50.00 | 2026-09-06 | 2026-09-03 |
| 2 | Claude Fable 5.1* · OpenRouter · 2-shot | 0.7316 | 0.7000 | 0.7662 | $10.00 | $50.00 | 2026-09-06 | 2026-09-01 |
| 3 | Claude Opus 5* · OpenRouter · 2-shot | 0.6904 | 0.6645 | 0.7185 | $5.00 | $25.00 | 2026-09-06 | 2026-07-24 |
| 4 | Claude Fable 5* · 2-shot | 0.6579 | 0.6274 | 0.6915 | $10.00 | $50.00 | 2026-07-23 | 2026-06-09 |
| 5 | Kimi K3 · OpenRouter · 2-shot | 0.6496 | 0.6684 | 0.6318 | $1.89 | $9.48 | 2026-07-23 | 2026-07-16 |
| 6 | GPT-5.4 · 2-shot | 0.6219 | 0.6452 | 0.6002 | $2.50 | $15.00 | 2026-03-05 | 2026-03-05 |
| 7 | Gemini 3.6 Flash · OpenRouter · 2-shot | 0.6197 | 0.6166 | 0.6228 | $0.75 | $3.75 | 2026-07-23 | 2026-07-21 |
| 8 | GPT-5.5 Pro · OpenRouter · 2-shot | 0.6078 | 0.5990 | 0.6169 | $30.00 | $180.00 | 2026-04-26 | 2026-04-24 |
| 9 | Muse Spark 1.3 · OpenRouter · 2-shot | 0.6078 | 0.5990 | 0.6169 | $1.25 | $4.25 | 2026-09-10 | 2026-09-02 |
| 10 | Gemini 3.8 Flash* · OpenRouter · 2-shot | 0.5954 | 0.6094 | 0.5821 | $0.75 | $3.75 | 2026-09-06 | 2026-09-02 |
| 11 | GPT-5.5 · 2-shot | 0.5820 | 0.5777 | 0.5864 | Unknown | Unknown | 2026-04-26 | 2026-04-23 |
| 12 | GPT-5.6 Sol · 2-shot | 0.5811 | 0.5660 | 0.5970 | Unknown | Unknown | 2026-07-23 | 2026-07-09 |
| 13 | Kimi K2.6 Nitro · OpenRouter · 2-shot | 0.5714 | 0.6317 | 0.5217 | Unknown | Unknown | 2026-04-26 | 2026-04-20 |
| 14 | GPT-5.3 Codex · OpenRouter · 2-shot | 0.5699 | 0.6105 | 0.5344 | $1.75 | $14.00 | 2026-03-05 | 2026-02-05 |
| 15 | Grok 4.6 · OpenRouter · 2-shot | 0.5521 | 0.5377 | 0.5672 | $2.00 | $6.00 | 2026-09-06 | 2026-08-12 |
| 16 | Claude Sonnet 5* · 2-shot | 0.5347 | 0.5475 | 0.5225 | Unknown | Unknown | 2026-07-23 | 2026-06-30 |
| 17 | GPT-5.6 Terra · 2-shot | 0.5284 | 0.5467 | 0.5113 | Unknown | Unknown | 2026-07-23 | 2026-07-09 |
| 18 | Gemini 3.5 Flash · OpenRouter · 2-shot | 0.5250 | 0.5277 | 0.5224 | $1.50 | $9.00 | 2026-07-23 | 2026-05-19 |
| 19 | GPT-5.6 Luna · 2-shot | 0.5076 | 0.5181 | 0.4975 | Unknown | Unknown | 2026-07-23 | 2026-07-09 |
| 20 | Muse Spark 1.1 · OpenRouter · 2-shot | 0.4875 | 0.4937 | 0.4815 | $1.25 | $4.25 | 2026-07-23 | 2026-07-16 |
| 21 | Claude Opus 4.7* · 2-shot | 0.4805 | 0.4882 | 0.4730 | Unknown | Unknown | 2026-04-26 | 2026-04-16 |
| 22 | GPT-5.2 · OpenRouter · 2-shot | 0.4791 | 0.5297 | 0.4374 | $1.75 | $14.00 | 2026-02-22 | 2025-12-11 |
| 23 | Kimi K2.5 · OpenRouter · 2-shot | 0.4751 | 0.5092 | 0.4453 | $0.45 | $2.25 | 2026-02-22 | 2025-07-11 |
| 24 | Kimi K2.7 Code · OpenRouter · 2-shot | 0.4611 | 0.5506 | 0.3966 | $0.71 | $3.50 | 2026-07-23 | 2026-06-12 |
| 25 | Grok 4.5 · OpenRouter · 2-shot | 0.4416 | 0.4450 | 0.4382 | $2.00 | $6.00 | 2026-07-23 | 2026-07-08 |
| 26 | Claude Opus 4.5* · 2-shot | 0.4141 | 0.4090 | 0.4194 | Unknown | Unknown | 2026-03-08 | Nov 2025 |
| 27 | GPT-5 · 2-shot | 0.4067 | 0.4838 | 0.3508 | Unknown | Unknown | 2026-02-18 | 2025-08-07 |
| 28 | Qwen3.6 Plus Nitro · OpenRouter · 2-shot | 0.3797 | 0.4678 | 0.3195 | Unknown | Unknown | 2026-04-26 | 2026-04-02 |
| 29 | Qwen3.8 Max (0902)* · OpenRouter · 2-shot | 0.3725 | 0.3640 | 0.3814 | Unknown | Unknown | 2026-09-06 | 2026-09-03 |
| 30 | Muse Glimmer 30B · OpenRouter · 2-shot | 0.3663 | 0.4164 | 0.3269 | $0.30 | $1.10 | 2026-09-10 | 2026-08-09 |
| 31 | Qwen3.7 Plus · OpenRouter · 2-shot | 0.3575 | 0.3653 | 0.3500 | $0.32 | $1.28 | 2026-07-23 | 2026-05-31 |
| 32 | Claude Opus 4.6* · 2-shot | 0.3269 | 0.3244 | 0.3294 | Unknown | Unknown | 2026-03-08 | 2026-02-05 |
| 33 | Qwen3.5 397B A17B · OpenRouter · 2-shot | 0.3081 | 0.3248 | 0.2930 | $0.55 | $3.50 | 2026-02-22 | 2026-02-17 |
| 34 | Claude Sonnet 4.6* · 2-shot | 0.2882 | 0.3130 | 0.2670 | Unknown | Unknown | 2026-03-08 | 2026-02-17 |
| 35 | Gemini 3.1 Pro Preview · OpenRouter · 2-shot | 0.2493 | 0.2434 | 0.2554 | $2.00 | $12.00 | 2026-02-22 | 2026-02-19 |
| 36 | Claude Opus 4.1* · 2-shot | 0.2318 | 0.2236 | 0.2407 | Unknown | Unknown | 2026-03-08 | 2025-08-05 |
| 37 | Claude Opus 4.8* · 2-shot | 0.2150 | 0.2113 | 0.2188 | Unknown | Unknown | 2026-07-23 | 2026-05-28 |
| 38 | GPT-4o · 2-shot | 0.2123 | 0.2589 | 0.1799 | Unknown | Unknown | 2026-02-22 | 2024-05-13 |
| 39 | GPT-4o (2024-11-20 OR) · OpenRouter · 2-shot | 0.2085 | 0.2662 | 0.1713 | $2.50 | $10.00 | 2026-02-22 | 2024-11-20 |
| 40 | Inkling · OpenRouter · 2-shot | 0.1962 | 0.2248 | 0.1741 | $1.00 | $4.05 | 2026-07-23 | 2026-07-17 |
| 41 | GPT-5.3 Chat · 2-shot | 0.1958 | 0.2121 | 0.1818 | Unknown | Unknown | 2026-03-05 | 2026-03-03 |
| 42 | GPT-o4 Mini · 2-shot | 0.1895 | 0.2382 | 0.1573 | Unknown | Unknown | 2026-02-22 | 2025-04-16 |
| 43 | GPT-4.1 · 2-shot | 0.1773 | 0.2243 | 0.1465 | Unknown | Unknown | 2026-02-22 | 2025-04-14 |
| 44 | ChatGPT-4o · OpenRouter · 2-shot | 0.1726 | 0.2043 | 0.1494 | Unknown | Unknown | 2026-02-22 | 2024-05-13 |
| 45 | Step 3.7 Flash · OpenRouter · 2-shot | 0.1652 | 0.1587 | 0.1723 | $0.20 | $1.15 | 2026-07-23 | 2026-05-28 |
| 46 | GPT-4.5 Preview · OpenRouter · 2-shot | 0.1443 | 0.1424 | 0.1462 | Unknown | Unknown | 2026-02-22 | 2025-02-27 |
| 47 | Gemini 3.1 Flash Lite Preview · OpenRouter · 2-shot | 0.1354 | 0.1515 | 0.1224 | $0.25 | $1.50 | 2026-03-08 | 2026-03-03 |
| 48 | GPT-4.5 Preview · 2-shot | 0.1225 | 0.1209 | 0.1240 | Unknown | Unknown | 2026-02-22 | 2025-02-27 |
| 49 | Gemma 4 31B · OpenRouter · 2-shot | 0.1200 | 0.1616 | 0.0955 | $0.09 | $0.34 | 2026-04-04 | 2026-04-02 |
| 50 | Gemini 3.5 Flash Lite · OpenRouter · 2-shot | 0.1187 | 0.1243 | 0.1136 | $0.30 | $2.50 | 2026-07-23 | 2026-07-21 |
| 51 | Claude Sonnet 4.5* · 2-shot | 0.0986 | 0.0874 | 0.1130 | Unknown | Unknown | 2026-03-08 | 2025-09-29 |
| 52 | Mistral Medium 3.5 · OpenRouter · 2-shot | 0.0871 | 0.0733 | 0.1072 | $1.50 | $7.50 | 2026-07-23 | 2026-04-30 |
| 53 | GLM 4.6V · OpenRouter · 2-shot | 0.0837 | 0.1447 | 0.0588 | $0.30 | $0.90 | 2026-02-22 | 2025-12-08 |
| 54 | MiniMax M3 · OpenRouter · 2-shot | 0.0734 | 0.0941 | 0.0602 | $0.30 | $1.20 | 2026-07-23 | 2026-05-28 |
| 55 | DeepSeek V4 Flash Vision Exp* · OpenRouter · 2-shot | 0.0578 | 0.1055 | 0.0398 | $0.22 | $0.66 | 2026-09-06 | 2026-08-21 |
| 56 | Llama 4 Maverick · OpenRouter · 2-shot | 0.0485 | 0.0525 | 0.0451 | $0.20 | $0.70 | 2026-02-22 | 2025-04-05 |
| 57 | Claude Haiku 4.5* · 2-shot | 0.0459 | 0.0554 | 0.0392 | Unknown | Unknown | 2026-03-08 | 2025-10-15 |
| 58 | Mistral Small 3.1 24B · OpenRouter · 2-shot | 0.0443 | 0.0511 | 0.0391 | $0.35 | $0.55 | 2026-02-22 | 2025-03-17 |
| 59 | GPT-4 Turbo · 2-shot | 0.0437 | 0.0561 | 0.0358 | Unknown | Unknown | 2026-02-22 | 2023-11-06 |
| 60 | GPT-4o · OpenRouter · 2-shot | 0.0397 | 0.0496 | 0.0331 | $10.00 | $30.00 | 2026-02-22 | 2023-11-06 |
| 61 | GPT-4.1 Mini · 2-shot | 0.0354 | 0.0380 | 0.0332 | Unknown | Unknown | 2026-02-22 | 2025-04-14 |
| 62 | Gemma 3 27B · OpenRouter · 2-shot | 0.0341 | 0.0355 | 0.0329 | $0.08 | $0.45 | 2026-02-22 | 2025-03-12 |
| 63 | Qwen VL Max · OpenRouter · 2-shot | 0.0335 | 0.0434 | 0.0273 | Unknown | Unknown | 2026-02-22 | 2024-01-25 |
| 64 | GPT-4o Mini · 2-shot | 0.0299 | 0.0314 | 0.0285 | Unknown | Unknown | 2026-02-22 | 2024-07-18 |
| 65 | Qwen2.5 VL 72B Instruct · OpenRouter · 2-shot | 0.0208 | 0.0271 | 0.0169 | $0.80 | $1.00 | 2026-02-22 | 2025-01-26 |
| 66 | Google: Gemini 2.0 Flash · OpenRouter · 2-shot | 0.0183 | 0.0236 | 0.0149 | Unknown | Unknown | 2026-02-22 | 2025-02-05 |
| 67 | Llama 4 Scout · OpenRouter · 2-shot | 0.0169 | 0.0227 | 0.0134 | $0.10 | $0.30 | 2026-02-22 | 2025-04-05 |
| 68 | Gemma 4 26B A4B IT · OpenRouter · 2-shot | 0.0126 | 0.0157 | 0.0105 | $0.04 | $0.22 | 2026-04-04 | 2026-04-02 |
| 69 | GPT-4o-mini · OpenRouter · 2-shot | 0.0076 | 0.0085 | 0.0069 | $0.15 | $0.60 | 2026-02-22 | 2024-07-18 |
| 70 | Mistral Large 3 2512 · OpenRouter · 2-shot | 0.0072 | 0.0065 | 0.0080 | $0.50 | $1.50 | 2026-02-22 | 2025-12-02 |
| 71 | GPT-4.1 Nano · 2-shot | 0.0051 | 0.0058 | 0.0045 | Unknown | Unknown | 2026-02-22 | 2025-04-14 |
| 72 | DeepSeek R1 · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | $0.70 | $2.50 | 2026-02-22 | 2025-01-20 |
| 73 | DeepSeek V3 0324 · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | $0.29 | $1.14 | 2026-02-22 | 2025-03-25 |
| 74 | Gemini 2.5 Pro Preview 3/25 · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | Unknown | Unknown | 2026-02-22 | 2025-03-25 |
| 75 | Grok 4 · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | Unknown | Unknown | 2026-02-22 | 2025-07-09 |
| 76 | Grok 4 Fast · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | Unknown | Unknown | 2026-02-22 | 2025-09-25 |
| 77 | Grok 4.1 Fast · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | Unknown | Unknown | 2026-02-22 | 2026-01-29 |
| 78 | Llama 3.2 90B Vision Instruct · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | Unknown | Unknown | 2026-02-22 | 2024-09-25 |
| 79 | Phi 4 Multimodal Instruct · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | Unknown | Unknown | 2026-02-22 | Feb 2025 |
| 80 | Qwen VL Plus · OpenRouter · 2-shot | 0.0000 | 0.0000 | 0.0000 | Unknown | Unknown | 2026-02-22 | 2024-01-25 |

## Caveats

* marks an image-preprocessing caveat. Claude-family images are resized and recompressed for Anthropic’s image limit. See each model’s notes below.

- Claude Fable 5.1* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Claude Opus 5* · OpenRouter · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Claude Fable 5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Gemini 3.8 Flash* · OpenRouter · 2-shot: Images are resized/recompressed when needed to stay below the provider's request payload limit.
- Claude Sonnet 5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Claude Opus 4.7* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Claude Opus 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Qwen3.8 Max (0902)* · OpenRouter · 2-shot: Images are resized/recompressed when needed to stay below the provider's request payload limit.
- Claude Opus 4.6* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Claude Sonnet 4.6* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Claude Opus 4.1* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- Claude Opus 4.8* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit. Opus 4.8's score is depressed by unreliable line counting: it identifies most edits correctly (166 of ~200 match ground truth by text) but reports noisy line numbers that drift progressively down the page and vary between runs, so the judge rejects the matches.
- Claude Sonnet 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.
- DeepSeek V4 Flash Vision Exp* · OpenRouter · 2-shot: Incomplete run: 35 of 36 evaluation predictions completed; one request repeatedly timed out at the provider.
- Claude Haiku 4.5* · 2-shot: Claude results use resized/recompressed images when needed to stay below Anthropic's 5 MiB per-image limit.

## Sources

- [Raw results](https://dorrit.pairsys.ai/results.json)
- [Pricing and per-model verification/source URLs](https://dorrit.pairsys.ai/pricing.json)
- [Technical report](https://dorrit.pairsys.ai/technical-report.pdf)
- [Interactive leaderboard](https://dorrit.pairsys.ai/)