Introducing MDI-Bench: the same dictation has a different correct answer in every office

Medicolegal death investigation has no benchmark.

Medical examiner and coroner offices sit where medicine, public health and the justice system meet. What they write carries consequences for families and in court.¹ As AI enters medicine, forensic pathology has to define what trustworthy AI looks like in its own practice, rather than inherit tools designed for outpatient visits.

AI already writes clinical documentation, and its failure modes are documented: omitted findings, fabricated content and misattribution that survive review.^2,3 A system that scores well on general medical benchmarks can still fail at the work itself.⁴

Reporting is the first place where it makes the most sense to build a benchmark for AI in medicolegal death investigation.

Why existing evaluations do not answer it

Documentation benchmarks score a note against a single reference note.^5,6,7 Legal work-product benchmarks⁸ and forensic evaluations^9,10,11 are examination focused. All of them assume one correct answer, which breaks down where informed judgment varies.^12-16

In an autopsy report, the template, the pathologist and the jurisdiction together define what correct looks like. MDI-Bench was built to close that gap, on three design choices:

• Office grounded. Every task is defined against the template and conventions of the office it came from, not a generic format.
• Physician anchored. The reference is the section a pathologist finalized and signed.
• Separable failure modes. Scoring separates matching an office’s wording from preserving the dictated content and the order, so one cannot hide the other.

Cause of death stays the pathologist’s call. What we measure is narrower and more checkable: whether the system carries the dictated findings into that office’s style.

What MDI-Bench asks: learn a house style from two documents.

For every graded row, the system receives the dictation, the section name, that office’s blank template and one completed report from the same office. It returns the formatted section. The two documents are the entire specification of the house style.

The reference is the section as it entered the record. Click through the pipeline below.

What it is built from: real casework, hand labeled by the people who dictated it.

1,486 dictated sections from 230 decedent cases, across 4 report templates from 3 offices, dictated by 5 pathologists. The median dictation is 209 characters against a median reference of 296, and 66% of references are a single block. All text is de-identified.

The answer for every task is text that office actually produced. It is not an annotator’s reconstruction of what the office might have written.

Three scores, kept separate.

A single number hides the result. Ordering can be right while the wording is wrong, and a weighted blend would let the easy axis cover for the hard one. So MDI-Bench reports three axes, each bounded 0 to 1, by fixed rules. No language model is the judge, and no axis is weighted above another. The overall score is their unweighted mean.

Try each one. These demos use simplified stand-ins for the real scorers so you can see what each axis is looking at.

A score that does not apply to a section is left out of that axis, not counted as zero. Sequencing, for example, only applies when the signed section has more than one block, which is about a third of sections.

First results: three models, one plateau.

We ran three frontier systems through one harness with one prompt shape, 4,458 graded rows in all, and added a do-nothing floor: the dictation left unchanged. Gemini 3.5 Flash scores 0.661, Claude Sonnet 5 0.655 and GPT-5.6 Terra 0.647. The raw dictation scores 0.256.

The three models sit within 0.014 of each other. All three clear the floor by a wide margin, and none separates from the others. This is a measurement of the task, not a ranking of vendors.

Why the floor is the raw dictation

A score means little without a reference for doing nothing. If a pathologist dictates a section and no system touches it, what is left is the raw dictation. That is the cheapest possible output, and it is what a model has to beat to be worth using.

It is not an empty answer. Every dictated finding is already in it, and speech carries some structure, which is why it scores 0.48 on layout and 0.27 on wording. Reading each model against this floor shows what the model adds, rather than crediting it for work the dictation already did. The floor is lowest on sequencing, at 0.02, which is also the axis where the models add the most.

Where the models help, and where they do not

Order is largely handled. The raw dictation scores 0.02 on sequencing. The models reach 0.71 to 0.79. Putting findings in the order the office signs is something general-purpose models do well when shown a template.

Wording is the unsolved axis. On lexical fidelity the models sit between 0.51 and 0.56, against 0.27 for the dictation left unchanged. The failures are house phrasing: a fixed verb, a sentence the pathologist never said and still signs, a header that exists because the template says so.

Layout is partly there. The dictation already scores 0.48, because speech carries some structure. The models land between 0.64 and 0.69.

What it means.

The three systems plateau together. The gap is specific: getting the order right is largely handled, and writing in the office’s own words is not.

For now a draft still needs editing for wording even when its structure is right. The pathologist stays the author, and the draft has to be reviewable before anything enters the record.

What these results do not show.

• One office supplies 76% of the sections. A pooled score describes that office more than it describes the field. Weighting offices equally could change the picture.
• Sequencing is the thinnest axis. It applies to about a third of sections, so it is the least stable of the three.
• The axes measure agreement with the reference text. They do not measure whether a pathologist would accept the draft. A swapped laterality or a changed unit can be a small word change with a large consequence, and these axes do not weigh that.
• One condition, one time point. A generic harness, three models and a single de-identified corpus. Every number is in-sample, and nothing here estimates generalization to a new office.
• This is not a test of Locarda’s product. It measures general-purpose models given two documents. Our production system was not in this run.

What comes next.

MDI-Bench will grow to cover more of medicolegal documentation. New results will be published here as they are ready.