Can an LLM solve a Redactle?

Redactle is a puzzle game where a redacted Wikipedia article is displayed. Guessing a word reveals all matches in the article, including variations like 1 and one. Once the entire title is revealed, the game is won.
One day I was pasting Redactle text into LLM chats from different providers and got wildly different results. I got nerd sniped and built a leaderboard to fairly evaluate as many as I could. $16.20 later, I now have this dashboard to answer that question.
The results are surprising. Reasoning effort did not help as much as I expected. Gemini and Grok are the best, not OpenAI and Anthropic models as I had expected. Gemini 3.7 Flash does exceedingly well, managing to snipe all 12 puzzles while being the cheapest and fastest.
Redactle results
Published Sep 3, 2026 · 35/36 attempts
- One-shot
- No-hint
- Failed
The rewritten passages keep the same topics and factual basis but remove exact Wikipedia sentences and substantially change word lengths. Lower scores here are evidence that familiar source phrasing helps, not proof that a model searched the web or memorized an article.
| # | Model · reasoning | Attempts | Quality | Solve rate | Score | Cost / run | Time / run |
|---|---|---|---|---|---|---|---|
| 1 | 12 | 4/7/1 | 92% | 8.8 | $0.039 | 61s | |
| 2 | 12 | 1/10/1 | 92% | 13.5 | $0.092 | 107s | |
| — | 11/12 | Partial results are shown without a rank. | |||||
Scroll sideways to see every measure.
Original text versus rewritten prose
Lower is better. Only configurations with all 12 synthetic attempts appear here.
| Model · reasoning | Original score | Rewritten score | Change |
|---|---|---|---|
| 0.0 | 8.8 | +8.8 | |
| 1.5 | 13.5 | +12.0 |
Trade-offs
Each label is a complete model configuration. Lower and farther left is better; the shaded quadrant is below the median on both measures.
Cost versus score
Speed versus score
Method notes
A model must make Redactle word guesses until every meaningful title word is revealed. Scores measure accepted guesses above the article title's word-count par, so a model that reveals every title word with exactly one guess per title word scores 0. In the hint-enabled evaluations, every hint adds 50 points.
The 500-word evaluation stops after 30 accepted guesses. If a model is still solving at that point, the attempt receives a score of 60 guesses. That is a cost-saving shortcut, not an estimate of how many guesses the model would have needed to finish. Other configurations use the same rule at twice their own guess limit.
Results from incomplete configurations remain visible but are not ranked. Article titles, excerpts, and per-article outcomes are deliberately omitted so this page does not spoil Redactle puzzles.
View evaluation code on GitHub (opens in a new tab)