What a Hebrew Cryptic Crossword Taught Me About AI Evals

One of my hobbies is solving תשבץ היגיון, the Hebrew cryptic crossword. The weekly puzzle by Yoram HaRoeh in Haaretz is reputedly the hardest of them all: anagrams, hidden words, homographs, and cultural references stacked into clues that read like tiny riddles. As a side project, in the same vibe coding spirit I bring to marketing systems, I spent a few weeks asking a serious question: can an LLM agent solve it honestly?

The result is a research note and an interactive solving assistant, live at tashbetz.gtmascode.dev, with the full methodology in the open tashbetz-solver repository. This post is the technical story, including the part where my solver scored 96% and the only correct response was suspicion.

The setup: corpus first, solver second

You cannot evaluate what you cannot measure, so the project started as a data pipeline, not a solver:

• 52 puzzles by the main setter, with 1,457 answered and explained clues collected from a community solutions site
• 310 easier puzzles from other setters, 6,792 clues, used only to learn solving tactics
• Clue text and grid geometry transcribed from newspaper images and validated by letter count
• Learned artifacts: a mechanism playbook, an indicator lexicon of trigger words, and 1,102 recurring "crosswordese" answers that act as priors

On top of that sits the solving toolkit: a grid model that knows every slot and crossing, a 129k-word Hebrew lexicon with pattern and anagram lookup, retrieval over similar past clues, and a consensus layer that merges multiple independent solve runs. The solver never sees answers at solve time. The grid layer is airtight: every filled answer is length-valid with zero crossing conflicts.

Eight versions, honestly logged

The iteration log is the heart of the project, and I kept it honest on purpose:

• v1 crashed on output limits before writing a single answer. Fix: incremental checkpointing.
• v2 hit 25% and 57% on the two dev puzzles. Real progress.
• v3 added anagram and pattern tools and regressed to 4% and 14%. The lesson: a word merely existing is not evidence it is the answer. The tools manufactured false confidence, and grid crossings amplified every wrong anchor.
• v4 introduced best-of-3 consensus, which beat every individual run. The key signal: answers that two runs agreed on were 64% correct, single-run answers only 24%.
• v5 allowed controlled fact lookup for cultural references and set a new single-run high of 61%, then regressed hard on a puzzle with no easy entry point. Bootstrappability of the puzzle itself, not solver mechanics, became the binding constraint.

Then came the audit.

The 96% that was too good to be true

A v6 run scored 96% on a puzzle where every previous version scored between 4% and 25%. Nobody celebrates a jump like that; you investigate it. The audit found that the lexicon had been built from the parsed answers file, which included the held-out evaluation puzzles. The solver never read an answers file directly, but its own pattern-lookup tool ranked the gold answers above ordinary dictionary words. The tool handed it the answer.

Every result from v3 to v6 was declared contaminated and thrown out. After sealing the leak and fixing a second bug (reversed letter enumerations that had silently halved accuracy), the clean v8 run landed at 43% per individual run and 64% with confidence-weighted consensus, against a clean baseline of 25%. The consensus math is the satisfying part: each run got 12 answers right, but only 5 overlapped. The union was 19, and the merge captured 18 of them. Independent runs fail on different clues, so merging recovers nearly everything either one knew.

What transfers beyond crosswords

I came for the puzzle and left with four lessons that apply directly to evaluating any AI system:

Contamination hides in tools, not just training data

My solver never "cheated" in the obvious sense. The leak lived in a helper tool built from the wrong file. If your agent has retrieval, search, or lookup tools, your eval is only as clean as every data source those tools touch.

Implausible scores are alarms, not achievements

The 96% run was the most useful failure of the project. Any evaluation pipeline should treat a discontinuous jump as a bug until proven otherwise.

Confidence calibration beats candidate generation

The v3 regression showed that generating more candidates makes things worse when the model cannot tell strong evidence from weak. Encoding confidence discipline, including the right to leave a cell empty, was worth more than any new solving tool.

Consensus works when runs are truly independent

Two runs agreeing was the strongest correctness signal in the whole system. That maps cleanly to production LLM systems: independent sampling plus agreement checks is cheap insurance.

Where it stands

The engine plateaus around 64% on the hardest setter with a fully clean protocol, gated by tightly interlocked grids and deep cultural references. The honest ceiling is the finding. The methodology also ships as a Claude Code skill, so anyone can drop a puzzle image into a session and get a solving partner that refuses to guess: the same discipline shown on the interactive solving site.

The meta-lesson for anyone building with AI: the interesting number is never the best score. It is the score you still trust after you have tried to kill it.