Challenge Credentialed Access
CXR-LT: Multi-Label Long-Tailed Classification on Chest X-Rays
Published: June 12, 2023. Version: 1.0.0 <View latest version>
Announcing CXR-LT, a competition for long-tailed disease classification on chest X-rays (June 21, 2023, 9:47 a.m.)
We are pleased to announce CXR-LT, a competition on Multi-Label Long-Tailed Classification on Chest X-Rays. Many real-world problems, including diagnostic medical imaging exams, are “long-tailed”: there are a few common findings followed by more relatively rare conditions. This competition will provide a challenging large-scale multi-label long-tailed learning task on chest X-rays (CXRs), encouraging community engagement with this emerging interdisciplinary topic.
CXR-LT is organized as a shared task for the workshop on Computer Vision for Automated Medical Diagnosis (CVAMD) held in association with the International Conference on Computer Vision (ICCV) 2023. Participants will be invited to submit their solutions for publication presentation at CVAMD 2023 and publication in the ICCV 2023 workshop proceedings.
The challenge uses an expanded version of MIMIC-CXR-JPG v2.0.0, a large benchmark dataset for automated thorax disease classification. Each CXR study in the dataset was labeled with 12 newly added disease findings extracted from the associated radiology reports. The resulting long-tailed (LT) dataset contains 377,110 CXRs, each labeled with at least one of 26 clinical findings (including a "No Finding" class).
05/01/2023: Development Phase begins. Participants can begin making submissions and tracking results on the public leaderboard.
07/14/2023: Testing Phase begins. Unlabeled test data will be released to registered participants. The leaderboard will be kept private for this phase.
07/17/2023: Competition ends. Participants are invited to submit their solutions as 8-page papers to ICCV CVAMD 2023!
07/28/2023: ICCV CVAMD 2023 submission deadline. (Competition participants may receive an extension if needed.)
08/11/2023: ICCV CVAMD 2023 acceptance notification.
10/06/2023: ICCV CVAMD 2023 workshop.
This competition is supported in part by the Artificial Intelligence Journal (AIJ). For any questions, please contact cxr.lt.competition.2023@gmail.
When using this resource, please cite:
(show more options)
Holste, G., Wang, S., Jaiswal, A., Yang, Y., Lin, M., Peng, Y., & Wang, A. (2023). CXR-LT: Multi-Label Long-Tailed Classification on Chest X-Rays (version 1.0.0). PhysioNet. https://doi.org/10.13026/721s-vs37.
Please include the standard citation for PhysioNet:
(show more options)
Goldberger, A., Amaral, L., Glass, L., Hausdorff, J., Ivanov, P. C., Mark, R., ... & Stanley, H. E. (2000). PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation [Online]. 101 (23), pp. e215–e220.
Many real-world problems, including diagnostic medical imaging exams, are “long-tailed” – there are a few common findings followed by more relatively rare conditions. In chest radiography, diagnosis is both a long-tailed and multi-label problem, as patients often present with multiple disease findings simultaneously. This is distinct from most large-scale image classification benchmarks, where each image only belongs to one label and the distribution of labels is relatively balanced. While researchers have begun to study the problem of long-tailed learning in medical image recognition, few have studied its interplay with label co-occurrence. This competition will provide a challenging large-scale multi-label long-tailed learning task on chest X-rays (CXRs), encouraging community engagement with this emerging interdisciplinary topic.
Chest radiography, like many diagnostic medical exams, produces a long-tailed distribution of clinical findings; while a small subset of diseases are routinely observed, the vast majority of diseases are relatively rare . This poses a challenge for standard deep learning methods, which exhibit bias toward the most common classes at the expense of the important, but rare, “tail” classes . Many existing methods  have been proposed to tackle this specific type of imbalance, though only recently with attention to long-tailed medical image recognition problems [4-6]. Diagnosis on chest X-rays (CXRs) is also a multi-label problem, as patients often present with multiple disease findings simultaneously; however, only a select few studies incorporate knowledge of label co-occurrence into the learning process [7-9, 12].
Since most large-scale image classification benchmarks contain single-label images with a mostly balanced distribution of labels, many standard deep learning methods fail to accommodate the class imbalance and co-occurrence problems posed by the long-tailed, multi-label nature of tasks like disease diagnosis on CXRs .
To develop a benchmark for long-tailed, multi-label medical image classification, we expand upon the MIMIC-CXR-JPG [10,11] dataset by enlarging the set of target classes from 14 to 26 (see full details in “Data Description”), generating labels for 12 new disease findings by parsing radiology reports. This follows the procedure of Holste et al. , who added 5 new findings to MIMIC-CXR-JPG – Calcification of the Aorta, Subcutaneous Emphysema, Tortuous Aorta, Pneumomediastinum, and Pneumoperitoneum – to study long-tailed learning approaches for CXRs and Moukheiber et al. , who added 5 new classes – Chronic obstructive pulmonary disease, Emphysema, Interstitial lung disease, Calcification, Fibrosis – to study ensemble methods for few-shot learning on CXRs.
CXR-LT challenge task
Given a CXR, detect all clinical findings. If no findings are present, predict "No Finding" (with the exception that "No Finding" can co-occur with "Support Devices"). To do this, participants will train multi-label thorax disease classifiers on the provided labeled training data.
ICCV 2023 CVAMD workshop
This shared task is hosted in conjunction with the ICCV 2023  workshop, Computer Vision for Automated Medical Diagnosis (CVAMD) . Upon completion of the competition, we will invite participants to submit their solutions for potential presentation at CVAMD 2023 and publication in the ICCV 2023 workshop proceedings. We intend to accept 5-6 papers for publication and select 3 of the accepted papers for oral presentation at CVAMD in Paris.
This shared task will be conducted on CodaLab . Participants must have credentialed access to MIMIC-CXR-JPG v2.0.0 [10,11] (more details in the "Data Description" section) and register for the competition on CodaLab.
- 05/01/2023: Development Phase begins. Participants can begin making submissions and tracking results on the public leaderboard.
- 07/14/2023: Testing Phase begins. Unlabeled test data will be released to registered participants. The leaderboard will be kept private for this phase.
- 07/17/2023: Competition ends. Participants are invited to submit their solutions as 8-page papers to ICCV CVAMD 2023!
- 07/28/2023: ICCV CVAMD 2023 submission deadline. (Competition participants may receive an extension if needed.)
- 08/04/2023: ICCV CVAMD 2023 acceptance notification.
- 10/06/2023: ICCV CVAMD 2023 workshop.
This challenge will use an expanded version of MIMIC-CXR-JPG v2.0.0 [10,11], a large benchmark dataset for automated thorax disease classification. Following Holste et al. , each CXR study in the dataset was labeled with 12 newly added disease findings extracted from the associated radiology reports. The resulting long-tailed (LT) dataset contains 377,110 CXRs, each labeled with at least one of 26 clinical findings (including a "No Finding" class). In addition to the 13 clinical findings in the original MIMIC-CXR-JPG dataset, the following 12 new findings have been added:
- Calcification of the Aorta
- Pleural Thickening
- Subcutaneous Emphysema
- Tortuous Aorta
Training set image IDs, metadata, and labels can be found in
train.csv and development set image IDs and metadata can be found in
development.csv. Alternatively, these files are available to registered participants on CodaLab  under "Participate" -> "Files". Test set image IDs and metadata (
test.csv) will be released when the testing phase begins on 07/14/2023.
Participants will upload image-level predictions on the provided test sets for evaluation. Since this is a multi-label classification problem with severe imbalance, the primary evaluation metric will be mean Average Precision (mAP) (i.e., "macro-averaged" AP across the 26 classes). While Area Under the Receiver Operating Characteristic Curve (AUC) is a standard metric for related datasets, AUC can be heavily inflated in the presence of strong imbalance. Instead, mAP is more appropriate for the long-tailed, multi-label setting since it both (i) measures performance across decision thresholds and (ii) does not degrade under class imbalance. For thoroughness, mean AUC (mAUC) and mean F1 score (mF1) — using a threshold of 0.5 for each class — will be calculated and appear on the leaderboard, but not contribute to team rankings.
A sample submission .csv for the development phase can be found in
development_sample_submission.csv or on CodaLab  under "Participate" -> "Files". A sample submission for the test phase (
test_sample_submission.csv) will be released when the testing phase begins on 07/14/2023. Reproducible code is required for this challenge, so all final submissions must contain all necessary code for preprocessing, training, inference, etc. See instructions below for how to successfully submit on CodaLab.
Submission file structure
All CodaLab submissions are required to be in .zip format. For this competition, this compressed .zip file must contain (i) a predictions .csv file and (ii) a "code/" directory with all of your training and inference code. The required file structure is as follows:
xxx.csv # predictions .csv file code/ # code directory ├── yyy.py ├── zzz.py ├── ...
To create the final submission .zip file, you might then run
zip -r submission.zip xxx.csv code. Please note that the names of your individual submission files do not matter, though the code directory must be named "code".
Predictions .csv file requirements
Your predictions .csv file must contain image-level predictions of the probability that each of the 26 classes are present in a given image. Specifically,
- You must have a "dicom_id" column with the provided unique image IDs for the given evaluation set.
- Entries must be strings.
- You must have a column for each of the 26 class labels ("Atelectasis", "Calcification of the Aorta", etc.).
- Entries must be floats or integers in the interval [0, 1].
Version 1.0.0: This is the initial release for the CXR-LT 2023 challenge dataset containing 377,110 CXRs, each labeled with at least one of 26 clinical findings (including a "No Finding" class).
This shared task uses image data from MIMIC-CXR-JPG v2.0.0 and generates labels from free-text radiology reports in MIMIC-CXR, a de-identified dataset that we gained access to through a PhysioNet Credentialed Health Data Use Agreement (v1.5.0).
We thank our steering committee for their support of this project: Leo Anthony Celi, Zhiyong Lu, George Shih, and Ronald Summers.
Conflicts of Interest
The authors have no conflicts of interest to declare.
- Zhou SK, Greenspan H, Davatzikos C, Duncan JS, Van Ginneken B, Madabhushi A, Prince JL, Rueckert D, Summers RM. A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE. 2021 Feb 26;109(5):820-38.
- Holste G, Wang S, Jiang Z, Shen TC, Shih G, Summers RM, Peng Y, Wang Z. Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study. In Data Augmentation, Labelling, and Imperfections: Second MICCAI Workshop, DALI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings 2022 Sep 16 (pp. 22-32). Cham: Springer Nature Switzerland.
- Zhang Y, Kang B, Hooi B, Yan S, Feng J. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023 Apr 19.
- Zhang R, Haihong E, Yuan L, He J, Zhang H, Zhang S, Wang Y, Song M, Wang L. MBNM: multi-branch network based on memory features for long-tailed medical image recognition. Computer Methods and Programs in Biomedicine. 2021 Nov 1;212:106448.
- Ju L, Wang X, Wang L, Liu T, Zhao X, Drummond T, Mahapatra D, Ge Z. Relational subsets knowledge distillation for long-tailed retinal diseases recognition. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24 2021 (pp. 3-12). Springer International Publishing.
- Yang Z, Pan J, Yang Y, Shi X, Zhou HY, Zhang Z, Bian C. ProCo: Prototype-Aware Contrastive Learning for Long-Tailed Medical Image Classification. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VIII 2022 Sep 16 (pp. 173-182). Cham: Springer Nature Switzerland.
- Chen H, Miao S, Xu D, Hager GD, Harrison AP. Deep hierarchical multi-label classification of chest X-ray images. In International Conference on Medical Imaging with Deep Learning 2019 May 24 (pp. 109-120). PMLR.
- Wang G, Wang P, Cong J, Liu K, Wei B. BB-GCN: A Bi-modal Bridged Graph Convolutional Network for Multi-label Chest X-Ray Recognition. arXiv preprint arXiv:2302.11082. 2023 Feb 22.
- Chen B, Li J, Lu G, Yu H, Zhang D. Label co-occurrence learning with graph convolutional networks for multi-label chest x-ray image classification. IEEE Journal of Biomedical and Health Informatics. 2020 Jan 16;24(8):2292-302.
- Johnson AE, Pollard TJ, Greenbaum NR, Lungren MP, Deng CY, Peng Y, Lu Z, Mark RG, Berkowitz SJ, Horng S. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042. 2019 Jan 21.
- PhysioNet. MIMIC-CXR-JPG - chest radiographs with structured labels [Internet]. Available from: https://physionet.org/content/mimic-cxr-jpg/2.0.0/.
- Moukheiber D, Mahindre S, Moukheiber L, Moukheiber M, Wang S, Ma C, Shih G, Peng Y, Gao M. Few-Shot Learning Geometric Ensemble for Multi-label Classification of Chest X-Rays. In Data Augmentation, Labelling, and Imperfections: Second MICCAI Workshop, DALI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings 2022 Sep 16 (pp. 112-122). Cham: Springer Nature Switzerland.
- IEEE/CVF. International Conference on Computer Vision 2023 [Internet]. Available from: https://iccv2023.thecvf.com/.
- Yang Y, Wang F, Holste G, Peng Y. Computer Vision for Automated Medical Diagnosis [Internet]. Available from: https://cvamd2023.github.io/.
- CodaLab. CXR-LT: Multi-Label Long-Tailed Classification on Chest X-Rays [Internet]. Available from: https://codalab.lisn.upsaclay.fr/competitions/12599.
Only credentialed users who sign the DUA can access the files.
License (for files):
PhysioNet Credentialed Health Data License 1.5.0
Data Use Agreement:
PhysioNet Credentialed Health Data Use Agreement 1.5.0
CITI Data or Specimens Only Research
DOI (version 1.0.0):
DOI (latest version):