MIMIC-CXR-Ext-ReRef: A Radiologist-Validated Dataset of MIMIC-CXR Report Pairs Following Different Reporting Practices

@article{PhysioNet-mimic-cxr-ext-reref-1.0.0,
  author = {Jeong, Daniel and Li, Charles and Hosseiny, Hossein and Bhalla, Nitya and {Uyar Morency}, Fatma and Ravikumar, Pradeep and Lipton, Zachary and Oberst, Michael},
  title = {{MIMIC-CXR-Ext-ReRef: A Radiologist-Validated Dataset of MIMIC-CXR Report Pairs Following Different Reporting Practices}},
  journal = {{PhysioNet}},
  year = {2026},
  month = oct,
  note = {Version 1.0.0},
  doi = {10.13026/q8ta-eb51},
  url = {https://doi.org/10.13026/q8ta-eb51}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

Radiologists follow heterogeneous reporting practices: two radiologists examining the same image and identifying the same clinical findings might nevertheless compose superficially distinct reports, varying in terminology, shorthand, formatting, and level of detail. Automated evaluation metrics for radiology report generation (RRG) models typically assess machine-generated reports based on their concordance with a single human-written reference, and can be sensitive to the reporting practices of the reference. To study this sensitivity, we developed ReRef, a large language model (LLM)-based method that rewrites a reference report along an axis of a radiologist-informed taxonomy of reporting practice variation while preserving its clinical interpretation. MIMIC-CXR-Ext-ReRef contains radiologist annotations on 120 (original, alternative) reference report pairs derived from the official test set of MIMIC-CXR v2.0.0, where the original reports are radiologist-written and the alternative reports were generated by ReRef, with 10 pairs for each of 12 axis-aligned perturbations. Board-certified radiologists assessed (i) whether each original and alternative report reads as a realistic report written by a practicing radiologist in routine clinical care (Task 1: Realism) and (ii) whether the two reports in each pair are clinically equivalent, i.e., a treating physician reading either report would identify the same findings and reach the same clinical decisions (Task 2: Clinical Equivalence). The dataset includes 855 individual annotations with free-text reasoning, adjudicated labels for three-way disagreements, and consensus labels for every example. MIMIC-CXR-Ext-ReRef is intended to support research on the robustness of RRG evaluation metrics to reporting practice variation and on methods for generating clinically equivalent reference reports.

Background

Automated generation of radiology reports, the primary medium through which radiologists document their interpretation of medical images, has the potential to improve the efficiency of radiologist workflows, helping reduce physician burnout and improve patient care. Given the free-form nature of radiology reports, a major challenge to the development of radiology report generation (RRG) models lies in designing evaluation methods that reliably characterize the clinical accuracy and utility of model-generated reports. As manual review by a radiologist is not scalable, a common approach is to employ an automated evaluation metric (e.g., GREEN [1], RadCliQ-v1 [2], RadGraph-F1 [3]) that compares the model-generated report to a reference report written independently by a radiologist, and outputs a numerical score.

However, radiologists follow highly heterogeneous reporting practices, often communicating interpretations with similar clinical impact in varying language and levels of detail [4-6]. This raises concerns about whether current evaluation approaches are sensitive to these differences. For instance, a model-generated report that enumerates normal findings (e.g., "The lungs are clear", "There is no pleural effusion") may receive a higher score when evaluated against a similarly enumerated reference (e.g., "Lungs are clear", "No pleural effusion or pneumothorax evident") than against a more abstract and lexically different reference report (e.g., "No acute cardiothoracic abnormality").

In Jeong et al. [7], we investigated how variations in reporting practice can impact the conclusions about RRG model performance. We constructed a taxonomy that characterizes the salient ways in which radiology reporting practice differs, designed under the guidance of a board-certified radiologist, and developed ReRef, a method that uses a large language model (LLM) to rewrite reference reports along the axes of the taxonomy while preserving the overall clinical interpretation. By quantifying the sensitivity of automated metrics to such reporting practice perturbations, we found that many existing metrics fail to decouple clinical interpretation from conformity to reporting practices, sometimes to an extent large enough to alter the rankings of models.

To validate that ReRef produces alternative reports that are (i) realistic (i.e., plausibly written by a practicing radiologist) and (ii) clinically equivalent to the original report (i.e., no change in diagnosis and patient management), we conducted a radiologist annotation study. MIMIC-CXR-Ext-ReRef releases the subset of annotated report pairs derived from MIMIC-CXR [8,9], to support future research on the evaluation of RRG models and on methods for simulating reporting practice variation.

Methods

Source Data

All radiology reports in this dataset are derived from the official test set of MIMIC-CXR v2.0.0 [8,9]. For each study, the key sections of the radiology report (Indication, Comparison, Technique, Findings, and Impression) were extracted by prompting Claude Sonnet 4.5 [10] (under a setup compliant with the PhysioNet Credentialed Health Data Use Agreement for MIMIC-CXR) with instructions to extract each section verbatim from the original report, without adding any information that is not directly stated in the original report. The full section extraction prompt is provided in Appendix C of Jeong et al. [7]. The "original" reports in this dataset correspond to the extracted Findings and Impression sections of the radiologist-written MIMIC-CXR reports.

Taxonomy of Reporting Practice Variation

Each alternative report in this dataset was generated by rewriting an original report along a single axis of reporting practice variation. These axes are defined by a taxonomy of reporting practice variation that we developed in Jeong et al. [7], which specifies the target variations used by ReRef to generate the alternative reports (see the next subsection). Under the guidance of a board-certified radiologist, we categorize the variations in radiology reporting practice along four dimensions (Organization, Completeness, Granularity, and Language; in bold), where each comprises one or more specific axes (in italics) along which radiology reports often differ without substantial change in clinical interpretation [7]. We describe each dimension and axis below.

Organization captures how report content is arranged at the report level. Structure contrasts highly templated formats, e.g., use of explicit anatomical headings, bullet points, numbered lists, and subheadings for groups of findings (Structure (Structured)), with continuous free-text prose (Structure (Free-Text)). Section Assignment captures the variations in how radiologists allocate content to different sections of a radiology report (e.g., comparisons to prior studies and differential diagnoses may be discussed in the Findings, in the Impression, or in both, depending on the radiologist's preference).

Completeness captures how exhaustively a report describes the findings visible in a study. We focus on a specific instantiation of this dimension, Findings Scope, which contrasts an exhaustive style that enumerates all normal and abnormal findings (Findings Scope (Exhaustive)) against a minimal style that selectively omits normal findings and reports only a focused list of abnormalities most likely to be clinically relevant (Findings Scope (Minimal)).

Granularity captures the level of detail at which findings are reported. Anatomical Granularity contrasts reports that enumerate findings for each anatomical substructure (Anatomical Granularity (High)) with those that aggregate findings at the organ/system level (Anatomical Granularity (Low)). Quantitative Granularity contrasts reports that include explicit numerical measurements (e.g., "cardiothoracic ratio of 0.55") with those that provide qualitative descriptions (e.g., "mild cardiomegaly") (Quantitative Granularity (Low)).

Language captures the linguistic variation in how radiologists report their interpretations. Terminology captures substitution of near-synonymous phrasings (e.g., "cardiomegaly" vs. "enlarged cardiac silhouette"), reflecting differences in training background, subspecialty, or institutional convention. Hedging captures how diagnostic uncertainty is expressed at an approximately fixed underlying confidence level (e.g., "worrisome for" vs. "concerning for"; "consistent with" vs. "likely represents"). Negation contrasts explicit negation of specific findings (e.g., "no pneumothorax, no pleural effusion") (Negation (Explicit)) with implicit phrasings that convey the same information through positive statements about normality (e.g., "pleura appear normal") (Negation (Implicit)).

Generating Alternative Reference Reports with ReRef

To simulate the reporting practice variations defined by the taxonomy above, we developed the ReRef method, which takes as input a reference report (all sections) and an axis-aligned target variation (e.g., Terminology, Structure (Free-Text)), and outputs an alternative reference rewritten to follow the specified reporting practices [7]. ReRef is implemented with a Generator and a Verifier, which are each in charge of drafting the axis-aligned alternative reference, and validating whether the generated report (i) accurately simulates the target variation and (ii) is clinically equivalent to the original reference, i.e., the overall diagnosis of the patient and implied management plans do not change. The Verifier also checks whether the target variation is feasible for the input report, as certain variations are impossible to simulate without hallucinating new clinical information. If the Verifier rejects a generated report, the Generator is prompted again to sample a different report, and this process is repeated until either the Verifier accepts the generated report or a maximum of 5 iterations is reached.

Out of the 13 total possible axis-aligned variations according to our taxonomy, we excluded Quantitative Granularity (High) from ReRef entirely, as it would require adding new numerical measurements that are not present in the original reference. Thus, a total of 12 axis-aligned reference perturbations are considered.

We instantiated both the Generator and the Verifier with o3 [11], under a setup compliant with the PhysioNet Credentialed Health Data Use Agreement for MIMIC-CXR. The full prompts and inference setup are provided in Appendix A of Jeong et al. [7], and the source code is available in our GitHub repository [12].

Sampling Report Pairs for Annotation

After applying the 12 axis-aligned perturbations to the reports from the MIMIC-CXR test set, we randomly sampled 10 (original, alternative) report pairs per perturbation, resulting in 12 × 10 = 120 report pairs for annotation. Pairs were sampled uniformly at random from those for which the Verifier accepted the alternative report as clinically equivalent to the original and as a valid simulation of the target variation, excluding pairs for which the Verifier determined that the original report was already in the target form or lacked the information required for the perturbation. Because pairs were sampled independently for each perturbation, the 120 pairs cover 114 unique studies, with 6 studies appearing in two pairs under different perturbations. The MIMIC-CXR report pairs were annotated as part of a larger study in which 10 report pairs were sampled per (dataset, axis-aligned variation) combination from four chest X-ray (CXR) datasets, resulting in 4 × 12 × 10 = 480 total pairs [7]; only the MIMIC-CXR pairs are released here, excluding others with license restrictions.

Radiologist Annotation

We obtained our annotations from 13 board-certified radiologists, with specialties in cardiothoracic imaging, general radiology, abdomen/pelvis imaging, and neuroradiology, who routinely interpret CXR images. The radiologists were blinded to which report was the original and which was the alternative: in Task 1, each report was presented on its own with no indication of its source, and in Task 2, the order of the two reports in each pair was randomized. The full annotation guide provided to the radiologists is included in Appendix B of Jeong et al. [7].

Task 1: Realism

We randomly provided either the original radiologist-written report or its alternative version (from ReRef), and asked each annotator to provide one of three scores:

• Realistic: Reads as a report written by a practicing radiologist in routine clinical care.
• Minor Issues: Mostly realistic, but contains subtle phrasing or formatting choices that are slightly unusual (though clinically correct).
• Major Issues: Contains major errors, incoherent phrasing, or unnatural elements that a practicing radiologist would not write in a standard radiology report.

Free-text comments were required for Minor Issues and Major Issues. For each annotation example, we also provided the Indication, Comparison, and Technique sections of the report (which remains identical between the original and alternative reports) as context about the corresponding imaging study, but the annotators were instructed to base their rating only on the Findings and Impression sections of the report being shown. We obtained realism ratings for both the original and alternative reports to test whether the realism ratings for the alternative reports exhibit significant degradation from those of the original reports. The original and alternative reports from each pair were annotated separately, since the goal was to assess standalone realism.

Task 2: Clinical Equivalence

We provided a randomly sampled (original, alternative) report pair, presented as "Report A" and "Report B" in a randomized order, and asked each annotator to provide one of three possible ratings:

• Equivalent: A treating physician reading either report would identify the same findings, and reach the same clinical decisions.
• Minor Discrepancy: Differences between the reports are limited to style, structure, terminology, or level of descriptive detail, and do not affect what clinical actions would follow.
• Major Discrepancy: The reports differ in major ways that could lead a treating physician to different clinical decisions.

Free-text comments were required for Minor Discrepancy and Major Discrepancy. As in Task 1, we provided the Indication, Comparison, and Technique sections of the report as context about the corresponding imaging study.

Consensus Labels

For each example being annotated in Tasks 1 and 2, we first obtained annotations from two radiologists. If the two annotations disagreed, we queried a third radiologist to provide their own independent annotation, and the consensus label was determined by majority vote. If all three annotators disagreed (three-way disagreement), we queried a fourth (cardiothoracic imaging) radiologist to provide the final consensus annotation, after accounting for the ratings and rationales provided by the first three annotators.

Preparation of the Released Files

The released files were prepared from the annotation results by restricting to the MIMIC-CXR examples, identifying each example by its (i) MIMIC-CXR study_id and (ii) axis-aligned perturbation, and removing all annotator identifiers. Individual annotations of each example are distinguished only by a 0-indexed per-example indicator (the annotation_index field). The report and context texts are provided as shown to the annotators.

Data Description

The dataset contains radiologist annotations of 120 (original, alternative) report pairs from 114 unique studies in the official MIMIC-CXR test set, with 10 report pairs for each of the 12 axis-aligned perturbations. Each original report is the radiologist-written report from MIMIC-CXR, and each alternative report is a rewrite of the original report generated by ReRef. For Task 1 (Realism), the original and alternative reports were annotated separately, resulting in 573 annotations of 240 reports; for Task 2 (Clinical Equivalence), the two reports were annotated as a pair, resulting in 282 annotations of 120 report pairs. For each task, we provide all individual annotations, the final ratings from the fourth radiologist for examples with three-way disagreements, and the consensus rating for every example in separate files.

File Structure and Contents

The dataset contains a README, a license, and two folders (task1 and task2), each containing three comma-separated value (CSV) files.

• README.md: Data dictionary describing each column of the CSV files.
• LICENSE.txt: Copy of the PhysioNet Credentialed Health Data License 1.5.0.
• task1
  • task1_full.csv: Contains all 573 individual annotations of the 240 reports, one row per annotation, together with the context and report text shown to the annotators. Each report has two (147 reports) or three (93 reports) annotations.
  • task1_disagreed.csv: Contains the final ratings and rationales from the fourth radiologist for the 15 reports with three-way disagreements.
  • task1_consensus.csv: Contains the consensus realism rating for each of the 240 reports.
• task2
  • task2_full.csv: Contains all 282 individual annotations of the 120 report pairs, one row per annotation, together with the context and report text shown to the annotators. Each report pair has two (78 pairs) or three (42 pairs) annotations.
  • task2_disagreed.csv: Contains the final rating and rationale from the fourth radiologist for the single report pair with a three-way disagreement.
  • task2_consensus.csv: Contains the consensus clinical equivalence rating for each of the 120 report pairs.

Each row can be linked back to the corresponding study in MIMIC-CXR via study_id, and the axis-aligned perturbation is given by reref_variation (e.g., structure--free_text, hedging, anatomical_granularity--low). A Task 1 example (a single report) is uniquely identified by (study_id, reref_variation, report_type) where report_type indicates either original or alternative, and a Task 2 example (a report pair) is uniquely identified by (study_id, reref_variation). The full definition of each column in the CSV files is provided in README.md.

Usage Notes

Prior Use

The annotations in this dataset are a subset of those used in Jeong et al. [7] to validate that ReRef generates realistic alternative references that preserve the original clinical interpretation. The results reported in that work aggregate annotations over all four CXR datasets (480 report pairs), of which the 120 MIMIC-CXR pairs are released here.

Reuse Potential

The dataset can be used to (i) evaluate the robustness of automated RRG evaluation metrics to reporting practice variations in the reference, by comparing metric scores against clinically equivalent (original, alternative) references; (ii) evaluate LLM-based judges of report realism or clinical equivalence against radiologist judgment; and (iii) study inter-radiologist variability in judgments of report realism and clinical equivalence, together with the free-text reasoning provided by the annotators.

Linking to MIMIC-CXR

Using this dataset requires access to MIMIC-CXR v2.0.0 [9] (or MIMIC-CXR-JPG v2.0.0 [13] for the images). The study_id column can be used to retrieve the corresponding chest X-ray images, full original reports, and patient identifiers (subject_id) from the MIMIC-CXR metadata files (e.g., mimic-cxr-2.0.0-metadata.csv, mimic-cxr-2.0.0-split.csv). All studies are from the official MIMIC-CXR test split.

Annotation Indices

The annotation_index column only distinguishes the annotations within a single example. It does not identify annotators across examples, and it does not indicate the order in which the annotations were collected. Analyses that require annotator identities across examples (e.g., annotator-level agreement statistics) are therefore not supported.

Limitations

While we conducted radiologist validation to verify ReRef's capability in generating realistic and clinically equivalent alternative references, it is possible that the generated outputs exhibit subtle deviations in clinical interpretation. The released pairs were sampled from alternative reports accepted by the ReRef Verifier and are therefore not representative of all ReRef outputs, including those rejected by the Verifier. The dataset covers 10 report pairs per axis-aligned perturbation from a single dataset (MIMIC-CXR), which may limit conclusions about individual perturbations or other institutions' reporting practices. As ReRef is implemented based on a proprietary, closed-source LLM (o3), reproducing the alternative references exactly may be difficult; nonetheless, our repository [12] contains the complete generation and verification pipeline, and all of the prompting details are included in Jeong et al. [7]. The report sections were extracted from the original MIMIC-CXR reports with an LLM [10] and may differ slightly in formatting from the source reports. Finally, realism and clinical equivalence are assessed from the report text alone, without access to the images.

Code

The source code for ReRef and for the evaluations in Jeong et al. [7] is available in our GitHub repository [12].

Release Notes

Version 1.0.0: Initial release.

For any questions or inquiries regarding this dataset, please contact us at danielje@cs.cmu.edu.

Ethics

This dataset is derived from MIMIC-CXR v2.0.0 [8,9], a de-identified database of chest radiographs with free-text radiology reports. The ethics approval of this dataset follows from that of the parent MIMIC-CXR dataset, which was approved by the Institutional Review Board of the Beth Israel Deaconess Medical Center, with the requirement for individual patient consent waived [8]. All processing of MIMIC-CXR radiology reports with LLMs was conducted under a setup compliant with the PhysioNet Credentialed Health Data Use Agreement for MIMIC-CXR.

This dataset provides radiologist annotations that support research on the automated evaluation of radiology report generation models, with the potential to improve the clinical accuracy and reliability of such models. The released report text is derived from de-identified MIMIC-CXR radiology reports and contains no additional information about patients beyond that in the source dataset. The annotations were provided by radiologists who were compensated for their work through a commercial annotation service. The released files contain only their ratings and free-text comments on the reports, and no information that identifies the annotators.

Acknowledgements

We gratefully acknowledge NSF (IIS-2211955), UPMC, Highmark Health, Abridge, Ford Research, Mozilla, the PwC Center, Amazon AI, JP Morgan Chase, the Block Center, the Center for Machine Learning and Health, and the CMU Software Engineering Institute (SEI) via Department of Defense contract FA8702-15-D-0002, for their generous support of ACMI Lab's research.

Conflicts of Interest

The author(s) have no conflicts of interest to declare.

References

1. Ostmeier S, Xu J, Chen Z, Varma M, Blankemeier L, Bluethgen C, Michalson AE, Moseley M, Langlotz C, Chaudhari AS, Delbrouck JB. GREEN: generative radiology report evaluation and error notation. In: Findings of the Association for Computational Linguistics: EMNLP 2024; 2024 Nov; Miami, Florida, USA. Association for Computational Linguistics; 2024. p. 374-90. doi:10.18653/v1/2024.findings-emnlp.21
2. Yu F, Endo M, Krishnan R, Pan I, Tsai A, Reis EP, Fonseca EKUN, Lee HMH, Abad ZSH, Ng AY, Langlotz CP, Venugopal VK, Rajpurkar P. Evaluating progress in automatic chest X-ray radiology report generation. Patterns (N Y). 2023;4(9):100802. doi:10.1016/j.patter.2023.100802
3. Delbrouck JB, Chambon P, Bluethgen C, Tsai E, Almusa O, Langlotz C. Improving the factual correctness of radiology report generation with semantic rewards. In: Findings of the Association for Computational Linguistics: EMNLP 2022; 2022 Dec; Abu Dhabi, United Arab Emirates. Association for Computational Linguistics; 2022. p. 4348-60. doi:10.18653/v1/2022.findings-emnlp.319
4. Weiss DL, Langlotz CP. Structured reporting: patient care enhancement or productivity nightmare? Radiology. 2008;249(3):739-47. doi:10.1148/radiol.2493080988
5. Bosmans JML, Neri E, Ratib O, Kahn CE Jr. Structured reporting: a fusion reactor hungry for fuel. Insights Imaging. 2015;6(1):129-32. doi:10.1007/s13244-014-0368-7
6. Delbrouck JB, Xu J, Moll J, Thomas A, Chen Z, Ostmeier S, Azhar A, Li KZ, Johnston A, Bluethgen C, Reis EP, Muneer MS, Varma M, Langlotz C. Automated structured radiology report generation. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); 2025 Jul; Vienna, Austria. Association for Computational Linguistics; 2025. p. 26813-29. doi:10.18653/v1/2025.acl-long.1301
7. Jeong DP, Li CQ, Hosseiny H, Bhalla NM, Uyar Morency F, Ravikumar P, Lipton ZC, Oberst M. Reporting practice matters: the impact of reference choice on chest X-ray report evaluation [Preprint]. arXiv:2609.19093. 2026 Sep 16. doi:10.48550/arXiv.2609.19093
8. Johnson AEW, Pollard TJ, Berkowitz SJ, Greenbaum NR, Lungren MP, Deng CY, Mark RG, Horng S. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci Data. 2019;6(1):317. doi:10.1038/s41597-019-0322-0
9. Johnson A, Pollard T, Mark R, Berkowitz S, Horng S. MIMIC-CXR Database (version 2.0.0). PhysioNet. 2019. doi:10.13026/C2JT1Q
10. Anthropic. Introducing Claude Sonnet 4.5 [Internet]. 2025 Sep 29 [cited 2026 Oct 17]. Available from: https://www.anthropic.com/news/claude-sonnet-4-5
11. OpenAI. OpenAI o3 and o4-mini system card [Internet]. 2025 Apr 16 [cited 2026 Oct 17]. Available from: https://openai.com/index/o3-o4-mini-system-card/
12. Jeong DP. rrg-reref [Internet]. GitHub; 2026 [cited 2026 Oct 17]. Available from: https://github.com/taekb/rrg-reref
13. Johnson A, Lungren M, Peng Y, Lu Z, Mark R, Berkowitz S, Horng S. MIMIC-CXR-JPG - chest radiographs with structured labels (version 2.0.0). PhysioNet. 2019. doi:10.13026/8360-t248