CheXmask-U: Uncertainty Estimation for Landmark-Based Anatomical Segmentation Masks in Chest X-ray Images

@article{PhysioNet-chexmask-u-1.0.0,
  author = {Cosarinsky, Matias and Gaggion, Nicolas and Ferrante, Enzo},
  title = {{CheXmask-U: Uncertainty Estimation for Landmark-Based Anatomical Segmentation Masks in Chest X-ray Images}},
  journal = {{PhysioNet}},
  year = {2026},
  month = sep,
  note = {Version 1.0.0},
  doi = {10.13026/6vn3-3j51},
  url = {https://doi.org/10.13026/6vn3-3j51}
}

Abstract

We present an extended version of the CheXmask dataset, augmenting the original anatomical segmentation masks of chest X-rays with node-wise uncertainty estimates. The dataset retains the comprehensive coverage of five public sources—ChestX-ray8, CheXpert, MIMIC-CXR-JPG, Padchest, and VinDr-CXR—while providing uncertainty estimates for each anatomical landmark. CheXmask-U enables both landmark-level and image-level uncertainty assessment, supporting uncertainty-aware segmentation research, robustness evaluation, calibration analysis, and out-of-distribution detection in medical imaging. The dataset includes preprocessed landmark-based segmentation masks for the left lung, right lung, and heart with corresponding node-wise uncertainty estimates, as well as the original individual quality control scores for each of the masks based on Reverse Classification Accuracy (RCA). The obtained uncertainty estimates were evaluated using manual landmark annotations, synthetic corruption experiments, and out-of-distribution analyses, demonstrating meaningful relationships between uncertainty and prediction reliability.

Background

Uncertainty estimation plays a key role in medical image segmentation by enabling the identification of unreliable predictions and supporting safe clinical deployment [1]. CheXmask-U extends the CheXmask [2] dataset by adding node-wise uncertainty estimates to anatomical landmark-based segmentation masks of chest X-rays. The original CheXmask dataset covers five public datasets—ChestX-ray8 [3], CheXpert [4], MIMIC-CXR-JPG [5], PadChest [6], and VinDr-CXR [7]—and provides segmentation masks for the left lung, right lung, and heart, together with image-level quality metrics based on Reverse Classification Accuracy (RCA) [8]. In contrast, CheXmask-U additionally provides fine-grained uncertainty estimates for each anatomical landmark, enabling localized reliability assessment of anatomical structures. The dataset leverages the HybridGNet [9] framework to produce probabilistic landmark predictions, supporting research in uncertainty-aware segmentation, robustness evaluation, calibration analysis, and out-of-distribution detection in chest X-rays.

Methods

HybridGNet Model and Training:

CheXmask-U uses the same HybridGNet weights employed to generate the original CheXmask dataset [2]. These weights were obtained by retraining HybridGNet on the complete 911-image Chest-Xray-Landmark dataset, with no held-out data split, comprising 246 JSRT images [11], 137 PadChest images [6], 138 Montgomery images [12], and 390 Shenzhen images [13]. All images contain lung landmark annotations, while 383 images additionally contain heart annotations. The training procedure follows the heterogeneous-label strategy described in [14]: each training batch contains images from a single source dataset, and the landmark regression loss is computed only for anatomical structures available in that dataset; unavailable landmark coordinates are excluded from backpropagation. The training objective combines mean squared error between predicted and ground-truth landmark coordinates with a KL-divergence regularization term. During training, random rotations, scaling, and color shifting are used for data augmentation. The accompanying implementation uses Adam optimizer with an initial learning rate of 10⁻⁴, a batch size of 4, and 1000 training epochs. The KL-divergence weight is linearly increased from 10⁻⁵ to 10⁻² during the first 200 epochs.

The trained model was subsequently applied to the five chest X-ray datasets included in CheXmask [2] to generate the anatomical landmark predictions, segmentation masks, and uncertainty estimates released in CheXmask-U.

Predictive Uncertainty and Segmentation:

Predictive uncertainty was estimated using Monte Carlo latent-space sampling. For each image, the latent space of HybridGNet was sampled N=50 times, and each latent sample was independently decoded into a complete landmark configuration, resulting in 50 stochastic landmark predictions per image. From these predictions, we compute:

• Landmark mean positions, representing the expected anatomical landmark locations across samples.

• Landmark std, representing node-wise predictive uncertainty for each anatomical landmark.

Pixel-level segmentation masks for the left lung, right lung, and heart were generated by filling the contours defined by the mean landmark positions. This approach produces a single mask per organ while retaining fine-grained landmark-level uncertainty information. Segmentation masks are provided both at the standardized 1024×1024 resolution and transformed back to the original image dimensions.

Additional implementation and training details can be found in [10].

Automated Quality Assessment:

Segmentation quality of the released masks is evaluated using Reverse Classification Accuracy (RCA), an established approach that provides an estimate of the Dice Similarity Coefficient (DSC) in the absence of ground truth annotations. The procedure involves training a reverse classifier using the predicted mask as the ground truth. This reverse classifier is then applied to a reference image set with known segmentations. The performance of the reverse model on the reference set is assumed to reflect the quality of the original predicted mask.

To accelerate evaluation, we use a deep learning-based atlas registration approach as part of RCA. This consists of two stages: rigid registration for global alignment, followed by deformable registration to correct local misalignments. The resulting transformation is applied to the predicted mask, and Dice scores are calculated based on the overlap between the transformed mask and the reference segmentations. For CheXmask-U, RCA is applied to masks generated from the mean landmark positions computed across stochastic predictions.

Validation:

The segmentation masks generated for CheXmask were previously validated against a manually annotated gold-standard set of 250 images, comprising 50 images from each of the five source datasets [2]. The uncertainty estimation framework was subsequently evaluated using manual landmark annotations, synthetic corruption experiments, and out-of-distribution analyses [10].

Uncertainty estimates were compared against landmark localization error on a separate set of 255 manually annotated chest X-rays, with two independent annotations, that were not used during model training. Higher predicted uncertainty was associated with larger landmark localization error, indicating that the uncertainty estimates reflect prediction reliability [10].

Additional experiments using synthetic image corruptions, including Gaussian noise and occlusion perturbations, demonstrated increasing uncertainty under degraded image conditions. The uncertainty measures were also evaluated for out-of-distribution detection and calibration analysis, demonstrating consistent relationships between uncertainty and prediction error.

The proposed approach was further benchmarked against established pixel-level uncertainty quantification methods, including U-Net with Monte Carlo Dropout, PHiSeg, and HybridGNet with Monte Carlo Dropout [10]. The variational HybridGNet framework demonstrated stronger uncertainty–error correlation while additionally providing fine-grained node-wise uncertainty information.

Data Description

Dataset size. CheXmask-U provides annotations for 657,566 chest X-ray images across five public datasets: 112,120 from ChestX-ray8, 187,825 from CheXpert, 243,334 from MIMIC-CXR-JPG, 96,287 from PadChest, and 18,000 from VinDr-CXR. These correspond to the same images included in the original CheXmask dataset [2].

Dataset structure. The dataset is structured as individual CSV files for each chest X-ray dataset. The dataset does not include original images or metadata. Instead, an image ID is provided in the first column of the CSV files, aligning with the ID column of the respective original dataset, making it straightforward to match rows in this CSV with the original dataset's instances. All images were preprocessed to a 1024×1024 resolution for consistent mask generation, and the resulting segmentation masks are provided both at this resolution and restored to the original image size.

The CSV files contain the following columns:

• Image ID: References to the original images according to the original metadata. Column names may vary across datasets.

• Dice RCA (Max): Maximum Dice Similarity Coefficient from Reverse Classification Accuracy (RCA), indicating segmentation quality.

• Dice RCA (Mean): Mean Dice Similarity Coefficient from RCA, providing an additional quality measure.

• Landmarks (Mean): Mean positions of anatomical landmarks, computed across multiple stochastic predictions generated from the HybridGNet latent space.

• Landmarks (Std): Standard deviation of landmark positions, representing node-wise uncertainty.

• Left Lung: Segmentation masks of the left lung, formatted in run-length encoding (RLE).

• Right Lung: Segmentation masks of the right lung, also in RLE format.

• Heart: Segmentation masks of the heart, in RLE format.

• Height: Height of the segmentation mask, necessary for decoding RLE.

• Width: Width of the segmentation mask, necessary for decoding RLE.

This structure provides information on image identifiers, segmentation quality, landmark-based uncertainty, and organ segmentation masks for each image. The landmark standard deviation values may be interpreted as localized measures of prediction uncertainty, where higher values indicate lower confidence in anatomical landmark localization. These uncertainty estimates can support downstream applications including uncertainty-aware segmentation benchmarking, quality-control filtering, robustness evaluation, confidence-aware pseudo-labeling, and out-of-distribution detection in chest X-ray analysis. Users should note that the uncertainty estimates reflect predictive uncertainty derived from stochastic latent-space sampling and should not be interpreted as direct measures of clinical uncertainty or diagnostic confidence.

Usage Notes

The dataset does not release the original chest X-ray images. To access the source images, users must refer to the original datasets (ChestX-ray8, CheXpert, MIMIC-CXR-JPG, PadChest, and VinDr-CXR) and comply with their respective access requirements, including any ethics courses or training. For downstream analysis, we recommend using segmentation masks with Dice RCA (Mean) ≥ 0.7 to ensure reliable masks. Additionally, the Landmarks (Std) column provides node-wise uncertainty; users may consider the average landmark standard deviation per image as an additional criterion for filtering uncertain predictions. Since the uncertainty estimates are derived from stochastic latent-space sampling, they should be interpreted as measures of model predictive uncertainty rather than direct indicators of clinical uncertainty or diagnostic confidence.

Release Notes

Version 1.0.0: Initial Release

Ethics

All publicly available datasets utilized in this study adhered to strict ethical standards and underwent thorough anonymization, with identifiable details removed. CheXmask-U does not release any part of the original image datasets; it provides only anonymized image identifiers to allow researchers to match the original images with the uncertainty-augmented segmentation annotations. The MIMIC-CXR-JPG dataset required additional ethics training and research courses for access. The study authors fulfilled all ethics courses and data use agreement requirements to ensure ethical data usage.

Conflicts of Interest

The authors have no conflicts of interest to declare.

References

1. Zou K, et al. A review of uncertainty estimation and its application in medical imaging. Meta-Radiology. 2023;1(1):100003.
2. Gaggion N, et al. CheXmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images. Scientific Data. 2024;11(1).
3. Wang X, et al. ChestX-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
4. Irvin J, et al. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI conference on artificial intelligence. 2019;33(01).
5. Johnson AE, et al. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint. 2019. arXiv:1901.07042.
6. Bustos A, et al. PadChest: A large chest x-ray image dataset with multi-label annotated reports. Med Image Anal. 2020;66:101797.
7. Nguyen HQ, et al. VinDr-CXR: An open dataset of chest X-rays with radiologist’s annotations. Sci Data. 2022;9(1):429.
8. Valindria VV, et al. Reverse classification accuracy: predicting segmentation performance in the absence of ground truth. IEEE Trans Med Imaging. 2017;36:1597–1606.
9. Gaggion N, Mansilla L, Mosquera C, Milone DH, Ferrante E. Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis. IEEE Trans Med Imaging. 2022. doi:10.1109/TMI.2022.3224660.
10. Cosarinsky M, Gaggion N, Echeveste R, Ferrante E. CheXmask-U: Quantifying uncertainty in landmark-based anatomical segmentation for X-ray images. Accepted for publication at Medical Imaging with Deep Learning (MIDL). 2026.
11. Shiraishi J, Katsuragawa S, Ikezoe J, Matsumoto T, Kobayashi T, Komatsu K, et al. Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules. AJR Am J Roentgenol. 2000;174(1):71–74.
12. Candemir S, Jaeger S, Palaniappan K, Musco JP, Singh RK, Xue Z, et al. Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration. IEEE Trans Med Imaging. 2014;33(2):577–590.
13. Jaeger S, Karargyris A, Candemir S, Folio L, Siegelman J, Callaghan F, et al. Automatic tuberculosis screening using chest radiographs. IEEE Trans Med Imaging. 2014;33(2):233–245.
14. Gaggion N, Vakalopoulou M, Milone DH, Ferrante E. Multi-center anatomical segmentation with heterogeneous labels via landmark-based models. Proceedings of the IEEE 20th International Symposium on Biomedical Imaging (ISBI). 2023;1–5.