MIMIC-IV-Ext-DischargeBench: Dataset for Pre-Deployment Evaluation of LLM Discharge Educators through Patient Simulation

@article{PhysioNet-mimic-iv-ext-dischargebench-1.0.0,
  author = {Jang, Won Seok and Yao, Zonghai and Yu, Hong},
  title = {{MIMIC-IV-Ext-DischargeBench: Dataset for Pre-Deployment Evaluation of LLM Discharge Educators through Patient Simulation}},
  journal = {{PhysioNet}},
  year = {2026},
  month = oct,
  note = {Version 1.0.0},
  doi = {10.13026/0tf4-ny14},
  url = {https://doi.org/10.13026/0tf4-ny14}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

MIMIC-IV-Ext-DischargeBench is a derived dataset of 320 curated hospitalization cases (v1.0.0) built from MIMIC-IV (v3.1) and MIMIC-IV-Note (v2.2). Each case is a structured patient profile drawn from the discharge documentation of a single hospitalization, covering diagnoses, medications, follow-up instructions, and post-discharge care plans. Cases were sampled from adult patients evenly across 16 diagnostic chapters, which combine matching ICD-9 and ICD-10 chapters into a single version-neutral grouping. Profiles were extracted from the structured tables and discharge notes with a large language model and manually reviewed against the source notes. Each case is paired with simulated-patient traits (personality, education level, health literacy, and medical-history recall). The dataset was developed for DischargeBench, a framework that simulates conversations in which an LLM educator explains discharge instructions to a simulated patient. The release also includes the complete simulation records of eight closed- and open-source models (GPT-5.5, GPT-5.4-mini, GPT-5.4-nano, Llama-3.3-70B-Instruct, Qwen3-32B, Qwen3-4B, MedGemma-27b-text-it, and MedGemma-4b-it), each run five times on every case. The simulation code and prompts are publicly available on the DischargeBench GitHub repository.

Background

Patient discharge education is an important part of patient–clinician interaction, giving patients the medical knowledge and self-care skills they need after discharge. Comprehensive patient education and effective communication are associated with lower hospital readmission rates, better adherence to treatment, and higher patient satisfaction [1–4]. Delivering thorough, accessible discharge education is therefore a clinical priority.

Prior literature has proposed several frameworks for organizing discharge education content. One framework identifies six core categories: (1) diagnosis, (2) medications, (3) post-discharge treatment, (4) tests and treatments during hospitalization, (5) indications to return to the emergency department or hospital, and (6) follow-up appointments [1, 5]. A complementary framework organizes discharge instructions around diagnosis, new medications, at-home care, expected illness duration, reasons to return, and follow-up [6]. Others have highlighted the particular importance of discharge education for surgical patients [2, 6].

Large language models (LLMs) are increasingly proposed as tools for delivering this education, and recent benchmarks evaluate them in conversations with LLM-simulated patients. PatientSim [11] builds persona-driven simulated patients from MIMIC data to evaluate diagnostic conversations, in which the clinician gathers information from the patient. DischargeSim [12] is the closest benchmark to ours: it evaluates LLMs as discharge educators on 49 expert-annotated cases with three patient personas and a dialogue divided into predefined stages. Discharge education reverses the direction of a diagnostic conversation, since information flows from clinician to patient. Evaluating it at scale requires cases whose discharge information is organized by the categories above, spread across diagnoses and paired with varied patient characteristics. Discharge notes in MIMIC-IV-Note contain this information, but it is embedded in long free text.

MIMIC-IV-Ext-DischargeBench addresses this need. It provides 320 cases spread evenly across 16 diagnostic chapters. Each case's discharge information is organized into the six categories above, and each case is paired with simulated-patient traits: one of five personality types, plus varied education level, health literacy and medical-history recall. The release also includes the complete simulation records of eight LLM educators, so researchers can analyze the conversations directly rather than re-running them, and use the cases as a common basis for developing and comparing discharge-education systems.

Methods

Dataset Curation

MIMIC-IV-Ext-DischargeBench was built in five steps: patient case selection, clinical data extraction and auditing, ICD chapter harmonization, Virtual Patient trait assignment, and simulation.

Patient Case Selection

Patient cases were sampled from MIMIC-IV (v3.1) [7] and MIMIC-IV-Note (v2.2) [8], restricted to adults aged 18 to 65 using anchor_age in the MIMIC-IV patients table [7]. Patients were excluded if any diagnosis recorded across all of their admissions fell within:

1. mental health or psychiatric disorders (ICD-10: F10–F99; ICD-9: 290–319), or
2. dementia or other cognitive disorders, including Alzheimer's disease (ICD-10: F00–F09, G30–G31; ICD-9: 290, 294, 331).

These criteria were applied so that included patients would have sufficient cognitive capacity to engage with a discharge education chatbot.

The primary diagnosis of each admission was taken as the first-listed ICD code in the MIMIC-IV diagnoses_icd table [7], and each code was mapped to its ICD chapter. Because ICD-9 and ICD-10 title several chapters differently, the eligible cohort contained 24 chapter titles. For each title, 20 admissions were randomly sampled, giving 480 cases.

Clinical Data Extraction and Auditing

Each discharge note was linked to MIMIC-IV by hadm_id and subject_id. Age, gender and race/ethnicity were taken from the patients and admissions tables. GPT-5.4-mini (OpenAI) [9], run under zero data retention, extracted the rest of the medical profile from the note: chief complaint, main diagnosis, reason for admission, medications on admission, allergies, hospital course and family history. It also extracted six categories of discharge information that serve as the topic-coverage reference: discharge diagnosis, new medications, treatments during the stay, post-discharge treatment, return-to-hospital warning signs, and follow-up appointments. A clinician reviewed all extracted variables against the source note. Three cases whose primary discharge diagnosis was redacted were excluded, leaving 477 cases. Cases with a history of depression or anxiety but documented clinical stability at discharge were kept. An LLM audit then removed 183 discharge-information items that the note did not support.

ICD Chapter Harmonization

The 24 chapter titles correspond to 16 clinical chapters, because ICD-9 and ICD-10 title 8 of the chapters differently. Each such pair was merged into one version-neutral chapter. Each merged chapter was then resampled to 20 cases with a fixed seed. The resampling kept the per-chapter balance of the Virtual Patient traits (described below), with a 10/10 ICD-9/ICD-10 split. The final dataset has 16 chapters × 20 = 320 cases.

Virtual Patient Trait Assignment

Each case was assigned synthetic Virtual Patient traits along four axes:

1. Education level (elementary, high school, college), which shapes the vocabulary and complexity of the simulated patient's utterances;
2. Health literacy (low, high), which governs comprehension of medical terminology;
3. Personality, one of five types (neutral, anxious, distrustful, high conscientiousness, minimiser), derived from the Five-Factor Model of personality [10] and grounded in the clinical simulation literature [11, 12];
4. Past medical history recall (poor, partial, accurate), the degree to which the simulated patient can report prior diagnoses, surgeries and medications.

Within each chapter, each personality type is assigned to 4 cases and health literacy is split 10 low / 10 high; education level and recall are not balanced. These traits are synthetic and do not describe the source MIMIC-IV patients.

Simulation Records

Each of the 320 cases was simulated with eight educator models (GPT-5.5, GPT-5.4-mini, GPT-5.4-nano, Llama-3.3-70B-Instruct, Qwen3-32B, Qwen3-4B, MedGemma-27B-text-it, MedGemma-4B-it), five independent runs each. In every session the Virtual Patient was Llama-3.3-70B-Instruct and the Simulation Orchestrator was Qwen3.5-9B. The Orchestrator reviewed each patient turn, had out-of-character turns regenerated (up to two retries before stopping the session), and ended the session when it judged the education complete, but not before 10 turns. Sessions were capped at 100 turns. Each record keeps the accepted dialogue, the rejected patient turns with the Orchestrator's reasons, and how the session ended.

Dataset Format

Each case is identified by its MIMIC-IV-Note note_id, with subject_id and hadm_id. All 320 cases, including their discharge notes, are stored in a single JSON Lines file in the format read by the DischargeBench code; the same case information, without the note text, is also provided as separate tables keyed by note_id. Simulation records are stored as one JSON Lines file per educator model and run. The data/ folder mirrors the layout of the DischargeBench code repository.

Data Description

MIMIC-IV-Ext-DischargeBench comprises 320 patient cases, each corresponding to a unique hospitalization drawn from MIMIC-IV (v3.1) [7] and MIMIC-IV-Note (v2.2) [8]: 16 ICD chapters × 20 cases, balanced within each chapter on personality (4 cases per type) and health literacy (10 low, 10 high). Every case integrates four kinds of information: a discharge note, a medical profile, synthetic Virtual Patient traits, and reference discharge information used as the answer key for topic-coverage scoring. The release also contains the prompt templates and the complete simulation records of the eight educator models evaluated in the accompanying paper.

Discharge Note

Each case includes its de-identified discharge note from MIMIC-IV-Note [8], retained in its original free-text form. Notes contain the standard discharge-summary components — chief complaint, history of present illness, hospital course, discharge medications, discharge instructions, and follow-up plans — and are the clinical source from which the medical profile and discharge information were extracted.

Medical Profile

Each case holds patient demographics (age, gender, race/ethnicity) from MIMIC-IV [7], the primary ICD code with its version and chapter, and fields extracted from the discharge note: chief complaint, main diagnosis, reason for admission, medications on admission, allergies, a summary of the hospital course (medical_history), and family history. The note-derived fields were extracted with GPT-5.4-mini [9] and manually audited by clinical experts. The Virtual Patient draws on the medical profile during the simulated dialogue.

Virtual Patient Traits

Traits are synthetic and do not describe the source MIMIC-IV patients.

Education Level

One of elementary, high school, or college. The label conditions the patient's vocabulary, comfort with multi-step instructions, and tendency to mask comprehension gaps. Elementary-level patients use everyday colloquial language (e.g., "my chest feels heavy" rather than "chest tightness"), require plain-language paraphrase of clinical terms, and may nod along without fully understanding. High-school-level patients grasp common health vocabulary and follow straightforward instructions but may miss qualifiers ("with food", "not with alcohol") and tend to fill gaps from informal sources. College-level patients process detailed information critically, sometimes arriving with prior research, and may push back on clinical decisions or skim familiar-seeming details.

Health Literacy

Low or high. Low-literacy patients struggle to act on health information even when they grasp it in the abstract, frequently misinterpret numerical content (dosing intervals, lab values), and often mask confusion with nodding or deferral; they respond best to verbal teach-back and visual aids. High-literacy patients read discharge instructions and prescription labels accurately, translate plans into concrete self-management actions, and may proactively flag conflicts or gaps in the regimen.

Personality

Following prior work on realistic patient simulation such as PatientSim [11] and DischargeSim [12], each case is assigned one of five personalities grounded in the five-factor model (OCEAN) [10]: neutral, anxious, distrustful, high conscientiousness, and minimiser. Neutral patients answer directly with a flat affect and do not volunteer information, serving as the baseline character. Anxious patients catastrophise minor symptoms, repeatedly seek reassurance, and jump between unrelated worries. Distrustful patients openly challenge the clinician's expertise, cite contradictory information from informal sources, and interpret routine questions as having a hidden agenda. High-conscientiousness patients are well prepared and ask precise, structured questions about the discharge plan but become visibly stressed when instructions feel ambiguous or incomplete. Minimisers downplay symptom severity, deflect concern with cheerful reassurance, and underreport frequency, making it harder for the educator to establish an accurate clinical picture.

Past Medical History Recall

Poor (frequently forgets prior diagnoses, surgeries, and current medication names or doses, and may contradict themselves mid-conversation), partial (recalls major events but loses specifics such as dates or exact dosages, occasionally confusing the order of events), or accurate (consistent and detailed recall, with explicit acknowledgment of genuine gaps rather than fabrication). This axis is motivated by PatientSim [11].

Discharge Information

Six fields extracted from the discharge note serve as the answer key for scoring how well the educator covers essential discharge topics: discharge_diagnosis, new_medication, treatment_during_stay, post_discharge_treatment, return_to_hospital_sign_and_symptoms_and_how, and follow_up_appointment. Each field is a list of note-derived strings; an empty list means the topic does not apply to the case (e.g., no new medications were prescribed), and the corresponding checklist questions are skipped. These fields are kept separate from the medical profile because they are the answer key, not information the Virtual Patient may draw on. They were extracted with GPT-5.4-mini [9] and audited by clinical experts; an LLM audit against the source note then removed 183 items the note did not support.

Simulation Records

The release contains one record per case for each of eight educator models (GPT-5.5, GPT-5.4-mini, GPT-5.4-nano, Llama-3.3-70B-Instruct, Qwen3-32B, Qwen3-4B, MedGemma-27B-text-it, MedGemma-4B-it) and each of five runs: 40 files and 12,800 records. Each record holds the accepted dialogue turns, the patient turns the Simulation Orchestrator rejected with its reasons, and how the session ended (judged complete, stopped by the Orchestrator, turn limit, or infrastructure error). Four MedGemma-4B-it records ended in a context-length error and contain no dialogue.

Patient Demographics

Table 1 summarizes the 320 cases.

Structure

mimic-iv-ext-dischargebench/
├── README.md
├── prompts/                                  # Jinja templates for every LLM role and judge
└── data/                                     # same layout as the DischargeBench code repository
    ├── processed/
    │   ├── discharge_bench_curated_v3.jsonl  # 320 cases with discharge notes (read by the code)
    │   └── output/<educator>__run<r>/
    │       └── simulation_results.jsonl      # 320 records each; 8 educators × 5 runs
    └── tables/                               # the same case information split by kind
        ├── cases.csv
        ├── demographics.csv
        ├── virtual_patient_traits.csv
        ├── medical_profile.jsonl
        └── discharge_information.jsonl

All case files have 320 rows, keyed by note_id and in the same order.

The README describes every field in detail.

Usage Notes

DischargeBench Evaluation

MIMIC-IV-Ext-DischargeBench is intended as a benchmark data resource for evaluating the discharge education capability of conversational AI and large language models (LLMs). It provides the patient case corpus underlying DischargeBench, a simulation-based benchmark in which an Educator LLM conducts multi-turn discharge education conversations with a Virtual Patient (VP) initialized from each case, while a Simulation Orchestrator regulates the VP and ends the session. Researchers can use this dataset to run Educator–Virtual Patient simulations, evaluate educators on clinically grounded metrics, and study how LLM behavior varies with diagnosis category, demographic background, or Virtual Patient traits. The included simulation records can also be re-scored or analyzed directly, without re-running the simulations.

DischargeBench evaluates educators on four axes, each scored by an LLM judge (Gemma-4-31B-it, temperature 0):

1. Conversation Quality rates the dialogue on naturalness, responsiveness, clarity, and clinical relevance, each on a 1–5 Likert scale, and reports readability as the Flesch–Kincaid grade level of the educator's turns (computed automatically, without the judge).
2. Topic Checklist Score (TCS) measures coverage of six discharge topics — discharge diagnosis, new medications, treatment during stay, post-discharge treatment, indications for return to hospital, and follow-up appointments — with weighted yes/no questions (parent questions weight 1, sub-questions 0.5). Questions whose topic is absent from the case's discharge information are excluded, and the score is normalized to 0–1.
3. Simulated Comprehension Score (SCS) measures how much of the discharge plan the Virtual Patient retains. After the conversation, the VP answers six open-ended questions covering the same clinical areas as TCS, with the note-derived clinical content removed from its persona so that answers reflect what the educator taught. Each answer is graded against a reference answer extracted from the discharge note as correct (1.0), partially correct (0.5), or incorrect (0.0); questions the note does not address are not counted.
4. Claim-Level Safety Audit (CLSA) decomposes the educator's speech into atomic clinical claims and classifies each claim by grounding (entailed, contradicted, or neutral relative to the discharge note) and by clinical status (appropriate, questionable, or harmful). It reports the rate of Wrong Instruction (contradicted and harmful) and Unsafe Advice (neutral and harmful) claims.

The data/ folder has the same layout as the DischargeBench code repository [16]: copied into a clone of the repository, the case file and simulation records are where the code reads them, so the released simulations can be scored without re-running them. Because the simulator skips cases already present in an output file, new simulations of the eight released educators should use a new run tag.

Because cases are drawn from the de-identified MIMIC-IV corpus and no additional clinical data collection was required, the dataset is accessible to any credentialed PhysioNet [13] user without institutional EHR access.

Accessing the Data

Users must be credentialed PhysioNet users and must complete CITI Data or Specimens Only Research training and sign the PhysioNet Credentialed Health Data Use Agreement 1.5.0 prior to accessing this dataset. These requirements are consistent with the access requirements for the parent MIMIC-IV and MIMIC-IV-Note projects. Instructions for obtaining PhysioNet credentials are available at [14].

Limitations

Several limitations should be considered when using this dataset. First, all discharge information originates from MIMIC-IV-Note [8], a single U.S. academic medical center. Findings derived from this dataset may not generalize to other health systems, geographic populations, non-English clinical documentation, or pediatric discharge settings. Second, during the clinical audit of MIMIC-IV discharge notes, we observed that certain information — such as follow-up dates, specific timestamps, and partial medication names — had been redacted for patient privacy. Because this is an inherent characteristic of the database, records with such omissions were retained unless the missing information was critical to the integrity of the simulation. Third, the Virtual Patient traits are synthetic and do not reflect the true characteristics of the underlying MIMIC-IV patients; they are experimental controls for simulation research and should not be treated as real patient characteristics. Fourth, the cohort is limited to adults aged 18–65 and excludes patients with recorded psychiatric or cognitive diagnoses, limiting representativeness of older or more complex patient populations. Fifth, the patients in the simulation records are LLM-simulated, and all scores come from an LLM judge; they are a screening signal before human evaluation, not evidence of how real patients would understand or act on the education.

Complementary Resources

• DischargeBench paper [15]
• DischargeBench code repository — simulation framework (Virtual Patient, Educator, and Simulation Orchestrator), prompts, evaluation pipeline, and analysis scripts [16]
• MIMIC-IV (v3.1) [7]
• MIMIC-IV-Note (v2.2) [8]

Release Notes

This is version 1.0.0 of the DischargeBench dataset. For any questions or concerns regarding this dataset, please feel free to reach out to us (WonSeok_Jang@student.uml.edu).

Ethics

MIMIC-IV-Ext-DischargeBench is derived from MIMIC-IV and is covered by the same IRB. All language models were either run locally or accessed through an API with zero data retention enabled, in compliance with the PhysioNet Credentialed Health Data Use Agreement.

Acknowledgements

We thank Gyumin Lee, Youngbin Jang and Joohwan Choi for auditing the dataset, verifying errors from LLM-generated extractions.

Conflicts of Interest

The author(s) have no conflicts of interest to declare.

References

1. DeSai, C., Janowiak, K., Secheli, B., Phelps, E., McDonald, S., Reed, G., & Blomkalns, A. (2021). Empowering patients: simplifying discharge instructions. BMJ open quality, 10(3), e001419. https://doi-org.umasslowell.idm.oclc.org/10.1136/bmjoq-2021-001419
2. Kang, E., Gillespie, B. M., Tobiano, G., & Chaboyer, W. (2022). Development of a web‐based discharge education intervention to improve the postdischarge recovery of general surgical patients. Journal of Nursing Scholarship, 54(2), 143-151.
3. Trivedi, S. P., Corderman, S., Berlinberg, E., Schoenthaler, A., & Horwitz, L. I. (2023). Assessment of Patient Education Delivered at Time of Hospital Discharge. JAMA internal medicine, 183(5), 417–423. https://doi.org/10.1001/jamainternmed.2023.0070
4. Bajorek, S. J., & McElroy, V. (2020). Discharge planning and transitions of care (Patient Safety Network: Patient Safety Primer). US Department of Health & Human Services.
5. Russell, S., Jacobson, N., & Pavlic, A. (2024). Improving Patient Understanding of Emergency Department Discharge Instructions. The western journal of emergency medicine, 25(6), 917–920. https://doi-org.umasslowell.idm.oclc.org/10.5811/westjem.18579
6. Tobiano, G., Chaboyer, W., Carlini, J., Eskes, A. M., McInnes, E., Latimer, S., ... & Gillespie, B. M. (2023). Development and validation of a survey designed to measure patient experience of and preference for surgical wound care discharge education: A pilot study. Journal of Tissue Viability, 32(3), 442-448.
7. Johnson, A., Bulgarelli, L., Pollard, T., Gow, B., Moody, B., Horng, S., Celi, L. A., & Mark, R. (2024). MIMIC-IV (version 3.1). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/kpb9-mt58
8. Johnson, A., Pollard, T., Horng, S., Celi, L. A., & Mark, R. (2023). MIMIC-IV-Note: Deidentified free-text clinical notes (version 2.2). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/1n74-ne17
9. Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., ... & Song, F. (2025). Openai gpt-5 system card. arXiv preprint arXiv:2601.03267.
10. McCrae, R. R., & John, O. P. (1992). An introduction to the five-factor model and its applications. Journal of personality, 60(2), 175–215. https://doi.org/10.1111/j.1467-6494.1992.tb00970.x
11. Kyung, D., Chung, H., Bae, S., Kim, J., Sohn, J. H., Kim, T., ... & Choi, E. (2025). Patientsim: A persona-driven simulator for realistic doctor-patient interactions. arXiv preprint arXiv:2505.17818.
12. Yao, Z., Sun, M., Jang, W. S., Kwon, S., Kwon, S., & Yu, H. (2025). DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Discharge. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, 2025, 10783–10809. https://doi.org/10.18653/v1/2025.emnlp-main.546
13. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PCh, Mark RG, Mietus JE, Moody GB, Peng CK, Stanley HE. PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals. Circulation 101(23):e215-e220 [Circulation Electronic Pages; http://circ.ahajournals.org/content/101/23/e215.full]; 2000 (June 13). PMID: 10851218; doi: 10.1161/01.CIR.101.23.e215
14. https://physionet.org/settings/credentialing/
15. Jang, W. S., Yao, Z., & Yu, H. (2026). From Discharge Notes to Patient Understanding: Persona-Grounded, Open-Ended Simulation of LLMs as Discharge Educators. arXiv preprint arXiv:2609.20827.
16. https://github.com/memy85/DischargeBench