MIMIC-IV-Ext-SDBench: Sequential Diagnosis Benchmark Cases Derived from MIMIC-IV

@article{PhysioNet-mimic-iv-ext-sdbench-1.0.0,
  author = {Nori, Harsha and Daswani, Mayank and Kelly, Christopher and Lundberg, Scott and {Correia Ribeiro}, Marco Tulio and Wilson, Marc and Liu, Xiaoxuan},
  title = {{MIMIC-IV-Ext-SDBench: Sequential Diagnosis Benchmark Cases Derived from MIMIC-IV}},
  journal = {{PhysioNet}},
  year = {2026},
  month = oct,
  note = {Version 1.0.0},
  doi = {10.13026/xdmx-ty42},
  url = {https://doi.org/10.13026/xdmx-ty42}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

MIMIC-IV-Ext-SDBench contains 488 case records derived from MIMIC-IV emergency department encounters in which the patient was subsequently admitted as an inpatient. Each case is formatted primarily for use with the Sequential Diagnosis Benchmark, an interactive diagnostic-reasoning benchmark in which an agent starts from a brief vignette and must request further information iteratively from a gatekeeper agent that mediates access to the underlying case record. Once an agent emits a diagnosis, a judge agent grades it against the ground truth.

Each MIMIC-IV-Ext-SDBench case provides an initial diagnosis-free vignette, the de-identified discharge summary and the ground truth diagnosis sourced from MIMIC-IV-Note v2.2. The dataset is split into 163 train, 254 validation and 71 clinician-reviewed test cases.

Beyond its use as a dataset for SDBench specifically, it can be used as a curated dataset for other sequential and non-sequential diagnostic reasoning tasks, evaluations, and benchmarks, with the clinician-reviewed test set providing expert-checked ground truth diagnoses.

Background

Most diagnostic-AI evaluations rely on static vignettes that present all relevant findings upfront. By compressing diagnosis into one inference step, static benchmarks cannot measure whether a system asks the right questions, avoids indiscriminate testing, updates the differential diagnosis as evidence accrues, or recognizes when to conclude the workup. High performance on such tasks may therefore overstate diagnostic competence while masking inappropriate testing or premature closure. In clinical practice, physicians begin with limited information and choose questions and tests while weighing their expected diagnostic yield against cost, patient burden, and delay. As findings return, they revise the differential diagnosis and decide whether further workup is needed.

The Sequential Diagnosis Benchmark (SDBench) reframes diagnosis as an iterative encounter that scores both diagnostic accuracy and the cost of the workup. Participants start from a brief vignette and iteratively request history, examination findings, and diagnostic tests from a gatekeeper before deciding when to commit to a diagnosis. This dataset provides a selected cohort of routine emergency-department admissions that can be used for an SDBench evaluation, complementing the educational NEJM clinicopathological cases already used in SDBench with cases drawn from routine hospital documentation.

See the SDBench paper [1] and website [2] for the benchmark design, evaluation methodology, open source baseline harness and published results.

Methods

Cases were constructed from MIMIC-IV v3.1 [3], MIMIC-IV-ED v2.2 [4], and MIMIC-IV-Note v2.2 [5].

Dataset curation steps:

1. Discharge summaries are joined with their admission, ED, and ICU records.
2. ED-originated admissions are sampled, excluding direct-to-ICU admissions and limiting each patient to a single note.
3. Cases were screened for inclusion to ensure they have a clear presenting complaint, key clinical findings, and a definitive final diagnosis.
4. A brief, diagnosis-free initial vignette that includes the patient's de-id age and recorded sex is synthesised.
5. The names of tests performed during the admission are extracted, along with an estimate of the number of clinician visits that were made.
6. Determine the ground-truth diagnosis from the discharge summary.
7. The dataset is divided into train, validation and test splits.

Large language models were used to automate some of the data curation following the PhysioNet guidance on using MIMIC data with large language models [6]. Claude Opus 4.1 was used for steps 3 and 6, and GPT-5 mini for steps 4 and 5.

All cases in the test split were then manually reviewed by clinicians for inclusion and for correctness of the ground-truth diagnosis. 13 ground-truth diagnoses were corrected during clinician review.

Data Description

The dataset is provided as a single file, dataset.json, containing a JSON array of 488 case records. These records represent 488 distinct patients admitted to hospital following emergency department encounters. Each case pairs a brief, diagnosis-free initial presentation with the corresponding discharge summary and ground truth diagnosis. The discharge summary includes the documented history, clinical findings, investigations, and hospital course.

Each record is an object with the following fields:

{
  "case_id": 0,                     // integer: unique identifier within this dataset
  "mimic_note_id": "10000000-DS-1", // string: MIMIC-IV-Note note_id
  "final_diagnosis": "...",         // string: ground-truth diagnosis
  "case_title": "...",              // string: short title combining note_id and diagnosis
  "case_summary": "...",            // string: initial vignette presented to the agent
  "full_text": "...",               // string: the de-identified MIMIC discharge summary
  "estimated_clinician_visits": 3,  // integer
  "tests": [{"name": "..."}],       // array of {name} objects: hospital tests performed
  "case_sets": ["train"]            // array of strings: split membership for this case
}

The final_diagnosis or ground-truth diagnosis is the principal condition responsible for hospitalization. The full_text field includes a copy of case_summary prepended to the discharge summary. The tests field contains extracted test names for tests performed during the admission, though findings remain in the discharge summary. The estimated_clinician_visits field estimates subsequent rounds of clinician-directed test ordering after the initial presentation, rather than an administrative encounter count. The mimic_note_id permits linkage to the original source note.

Splits (read from the case_sets field):

• train (163 cases): Provided as additional development data. Not used in published SDBench evaluations.
• validation (254 cases): Included in published SDBench evaluations. Used for development of the MIMIC specific gatekeeper but not used for agent development.
• test (71 cases): Clinician reviewed. Included in published SDBench evaluations as fully held-out data.

Usage Notes

The dataset is primarily designed to be loaded directly by the open-source SDBench evaluation harness, which reads the dataset.json from a specified directory. When running SDBench with GPT-5, the baseline agent achieved 76.6% diagnostic accuracy on the 325 MIMIC-IV validation and test cases, while the Diagnostic Orchestrator (DxO) agent without budget constraints achieved 80.3%. Mean estimated workup costs were approximately US$4,240 and US$4,980 per case, respectively.

See the SDBench paper [1] and website [2] for the benchmark design, evaluation methodology, open source harness and published results.

When using this dataset, the PhysioNet guidance on using MIMIC data with large language models [6] should be followed. The SDBench harness routes LLM calls to zero-data-retention providers by default, but care must be taken if the harness is configured to use other providers.

The dataset contains selected hospital admissions with diagnosed conditions, so its case mix and disease prevalence do not represent all emergency presentations. When used with SDBench, the cases are presented through a text-based simulation that does not capture the full uncertainty of patient communication, changes over time or responses to treatment, or direct interpretation of images, waveforms, and pathology slides. Discharge summaries may omit findings, and the gatekeeper may synthesize responses that are not observed patient measurements. Cost estimates use standardized U.S. prices and do not fully capture local pricing, test availability, invasiveness, or time to results. The benchmark assesses diagnostic correctness but does not assess treatment safety, longitudinal management, or patient outcomes.

Beyond SDBench, the dataset may support clinical information extraction, including developing methods to identify symptoms and investigations in discharge summaries or evaluating extraction of the principal diagnosis against the supplied ground truth. It may also provide cases for other sequential and non-sequential diagnostic reasoning evaluations and benchmarks, supporting comparisons of diagnoses produced from different amounts of case information and analysis of diagnostic errors. The clinician-reviewed test set (71 cases) provides expert-checked ground truth diagnoses for these applications.

Release Notes

Version 1.0.0

Initial release

Ethics

Conflicts of Interest

References

1. Nori H, Daswani M, Kelly C, Lundberg S, Ribeiro MT, Wilson M, Liu X, Sounderajah V, Carlson JM, Lungren MP, Gross B, Hames P, Suleyman M, King D, Horvitz E. Sequential diagnosis with language models. Science. 2026. Available from: https://www.science.org/doi/10.1126/science.aeh7930
2. SDBench: Sequential Diagnosis Benchmark [Internet]. Microsoft; 2026. Available from: https://microsoft.github.io/sdbench
3. Johnson A, Bulgarelli L, Pollard T, Gow B, Moody B, Horng S, Celi L A, Mark R. MIMIC-IV (version 3.1). PhysioNet. 2024. RRID:SCR_007345. Available from: https://doi.org/10.13026/kpb9-mt58
4. Johnson A, Bulgarelli L, Pollard T, Celi L A, Mark R, Horng S. MIMIC-IV-ED (version 2.2). PhysioNet. 2023. RRID:SCR_007345. Available from: https://doi.org/10.13026/5ntk-km72
5. Johnson A, Pollard T, Horng S, Celi L A, Mark R. MIMIC-IV-Note: Deidentified free-text clinical notes (version 2.2). PhysioNet. 2023. RRID:SCR_007345. Available from: https://doi.org/10.13026/1n74-ne17
6. Use of MIMIC data with large language models and online services [Internet]. PhysioNet (MIT Laboratory for Computational Physiology); 2025 Sep 24 [cited 2026 Jun 17]. Available from: https://physionet.org/news/post/llm-responsible-use/