Leaderboard
Means over each doctor's consultations. Click a column to sort; click a row to highlight that model in every chart.
Charts
Colour is the lab. Filled circles are closed-weight models, rings are open-weight, the square is the GP registrar. Hover for numbers; click a doctor to outline it, fade the rest and split its point into its consultations.
Score against cost
Questions asked against red flags caught
Where the points came from
Mean score by case, or by the four parts of the bill, each out of its own maximum.
The traps
Each case hides something the clinic record doesn't show. Filled dots are consultations that walked into it.
Two runs: 4 October and 21 October
The same five cases, played again on the game as it stands. Each line runs from a model's 4 October result to its 21 October one.
What changed between the runs
So a change in a model's score mixes three things: the patient, the rules and the model's own run-to-run spread. The comparison shows how robust the ranking is, not which model improved.
Every consultation
Each button is one consultation, shaded by its score. Open one for the transcript, the itemised bill and the move log.
Method and limits
The doctor is the model under test. It acts only through tools: say to the patient, list_exams and search to find exams, tests, drugs, referrals and diagnoses, then examine, order_test, prescribe, refer, diagnose and end_consult. It sees what a human player sees: the patient's name, age and booking, the clinic record, and its own findings and results. It never sees the case file or the points.
The patient is the game's own patient model, playing a person from a private chart; it does not know its diagnosis. A separate classifier call decides which history items each question asks about, and only those facts are released to the patient. A question it misses gets a denial, as it would for a human player.
The score is decided by code against a hand-written key: red flags asked about, key history, indicated exams and tests, a safe plan and a defensible diagnosis. It scores the process, not the outcome. Unnecessary tests cost points, more for each further one, and harmful actions cost a lot. Spoken advice is judged at the end from the doctor's own words, with code checking that each quote was really said.
The budget is 50 questions a consultation. Explanations, the plan, examining, tests and prescriptions are free.
The GP registrar is one of the paper's authors, a practising GP registrar in Australia. They played each free case once on the live game, briskly (about four questions a consultation), and their consultations were rescored under the 21 October rules. Five consultations by one person are a reference point, not a human baseline.
Limits
- All five cases are drafts written by the game's author, a GP registrar, from standard references, and follow Australian practice (eTG, PBS).
- The patient, the classifier and the judge are small language models. They miss some questions and sometimes credit or miss a spoken item. Every doctor faces the same patient, but the noise is real. On 21 October the patient was a Gemini model, and Gemini 4 Argon was among the doctors.
- Three consultations a case is a small sample; the ± is one standard deviation of a doctor's consultations.
- It is a benchmark of a game. It says nothing about whether any model can or should practise medicine.
Cite
@misc{woodburn2026crookbench,
title = {Crook Bench: Language Models as the Doctor in a Deterministically Scored General-Practice Consultation Game},
author = {Woodburn, Michael and Keenan, Rory},
year = {2026},
note = {https://woodytwoshoes.github.io/crook-bench/}
}
Code, cases, every transcript and the known issues: github.com/woodytwoshoes/crook-bench. The game: doctorfoo.ai.