Crook bench
Crook bench

How AI models do as the GP in a consultation game

Each model plays the doctor in the five free cases of Crook, an Australian general-practice game, three times each. It talks to an AI patient in its own words, examines, orders tests, prescribes and refers through tools, then commits to a diagnosis. The game's own code scores every consultation against a hand-written key, exactly as it scores a human player. A score here measures play of a game, not fitness to practise medicine.

Run
Weights
Click any model to highlight it everywhere and open up its consultations.

Leaderboard

Means over each doctor's consultations. Click a column to sort; click a row to highlight that model in every chart.

Charts

Colour is the lab. Filled circles are closed-weight models, rings are open-weight, the square is the GP registrar. Hover for numbers; click a doctor to outline it, fade the rest and split its point into its consultations.

Overall score: the mean share of each case's maximum. The whisker is one standard deviation of the doctor's consultations either side.

Score against cost

Cost per consultation of the doctor model through OpenRouter, log scale. Up and to the left is better value. The patient's cost is the same for everyone and not included.

Questions asked against red flags caught

Red flags are the history items that change management, such as sweating with chest pain. Models had a 50-question budget.

Where the points came from

Mean score by case, or by the four parts of the bill, each out of its own maximum.

The traps

Each case hides something the clinic record doesn't show. Filled dots are consultations that walked into it.

Two runs: 4 October and 21 October

The same five cases, played again on the game as it stands. Each line runs from a model's 4 October result to its 21 October one.

What changed between the runs

    So a change in a model's score mixes three things: the patient, the rules and the model's own run-to-run spread. The comparison shows how robust the ranking is, not which model improved.

    Every consultation

    Each button is one consultation, shaded by its score. Open one for the transcript, the itemised bill and the move log.

    Pick a consultation above.

    Method and limits

    The doctor is the model under test. It acts only through tools: say to the patient, list_exams and search to find exams, tests, drugs, referrals and diagnoses, then examine, order_test, prescribe, refer, diagnose and end_consult. It sees what a human player sees: the patient's name, age and booking, the clinic record, and its own findings and results. It never sees the case file or the points.

    The patient is the game's own patient model, playing a person from a private chart; it does not know its diagnosis. A separate classifier call decides which history items each question asks about, and only those facts are released to the patient. A question it misses gets a denial, as it would for a human player.

    The score is decided by code against a hand-written key: red flags asked about, key history, indicated exams and tests, a safe plan and a defensible diagnosis. It scores the process, not the outcome. Unnecessary tests cost points, more for each further one, and harmful actions cost a lot. Spoken advice is judged at the end from the doctor's own words, with code checking that each quote was really said.

    The budget is 50 questions a consultation. Explanations, the plan, examining, tests and prescriptions are free.

    The GP registrar is one of the paper's authors, a practising GP registrar in Australia. They played each free case once on the live game, briskly (about four questions a consultation), and their consultations were rescored under the 21 October rules. Five consultations by one person are a reference point, not a human baseline.

    Limits

    • All five cases are drafts written by the game's author, a GP registrar, from standard references, and follow Australian practice (eTG, PBS).
    • The patient, the classifier and the judge are small language models. They miss some questions and sometimes credit or miss a spoken item. Every doctor faces the same patient, but the noise is real. On 21 October the patient was a Gemini model, and Gemini 4 Argon was among the doctors.
    • Three consultations a case is a small sample; the ± is one standard deviation of a doctor's consultations.
    • It is a benchmark of a game. It says nothing about whether any model can or should practise medicine.

    Cite

    @misc{woodburn2026crookbench,
      title  = {Crook Bench: Language Models as the Doctor in a Deterministically Scored General-Practice Consultation Game},
      author = {Woodburn, Michael and Keenan, Rory},
      year   = {2026},
      note   = {https://woodytwoshoes.github.io/crook-bench/}
    }

    Code, cases, every transcript and the known issues: github.com/woodytwoshoes/crook-bench. The game: doctorfoo.ai.