Early Prediction of Sepsis from Clinical Data: the PhysioNet/Computing in Cardiology Challenge 2019

22 April 2019:
• The official phase of the 20th PhysioNet/Computing in Cardiology Challenge is now open.
• We have made several changes from the unofficial phase of the Challenge (see below). We invite comments and questions about these changes. We plan to accept submissions again on Thursday, 25 April at 12:01 am GMT.
• We will only use our new cloud submission system for the official phase of the Challenge. See the updated instructions (here) for details. The past submission system is no longer available.
• We ask participants to write causal algorithms that make predictions using current and past (but not future) information. See the submission instructions and sample prediction code ( here ) for details.
• Note that you will have 10 submissions in this official phase. We will score your results on a subset of the test data. At the end of the competition we will ask you to nominate your 'best' algorithm (it need not be the one that gave you the best score) and we will run it on the full test data to provide the final test score.
• Please do not submit all ten entries in the last week of the competition. Even though we can scale the computing, failures require manual intervention and feedback. We can't do this for 1000+ entries in the final week.
• Also - we will be offering up to $500 in Google Cloud credits (courtesy of the Google Cloud Team) to the best performing entries by June, so it's worth getting a good score before then! 11 April 2019: • We have now implemented a cloud submission system. Instructions for the cloud submission system available here. • The deadline of the unofficial phase of the competition has been extended to 11.59pm GMT on Sunday 14th April 2019. This will give you enough time to receive scores and, very importantly, submit abstracts to cinc.org before the 15th. Please see here for important hints on how to prepare a successful abstract, even if you don't receive a score. • If you don't receive a score - don't panic - just submit an abstract to cinc.org by the deadline on the 15th April with some results - cross validated on the training data. • The hiatus has now been moved, and starts on the 15th April and lasts until 12 midnight GMT on the 21st April. The Challenge re-opens at 12.01 am GMT on the 22 April. 8 April 2019: • The entry submission system is available here. • The deadline of the unofficial phase of the competition has been extended to 11.59pm GMT on Wednesday 10th April 2019. 1 April 2019: 6 February 2019: • The 20th PhysioNet/Computing Challenge in Cardiology is now open. • This year's challenge is co-sponsored by the Gordon and Betty Moore Foundation, MathWorks, and Google. • The PhysioNet Challenge data, examples, and scoring functions can be downloaded here. If you have any questions or comments regarding this Challenge, then please post them directly in our Community Discussion Forum. This will increase transparency (benefiting all the competitors) and ensure that all the Challenge organizers see your question. Introduction Sepsis is a life-threatening condition that occurs when the body's response to infection causes tissue damage, organ failure, or death (Singer et al., 2016). In the U.S., nearly 1.7 million people develop sepsis and 270,000 people die from sepsis each year; over one third of people who die in U.S. hospitals have sepsis (CDC). Internationally, an estimated 30 million people develop sepsis and 6 million people die from sepsis each year; an estimated 4.2 million newborns and children are affected (WHO). Sepsis costs U.S. hospitals more than any other health condition at$24 billion (13% of U.S. healthcare expenses) a year, and a majority of these costs are for sepsis patients that were not diagnosed at admission (Paoli et al., 2018). Sepsis costs are even greater globally with the developing world at most risk. Altogether, sepsis is a major public health issue responsible for significant morbidity, mortality, and healthcare expenses.

Early detection and antibiotic treatment of sepsis are critical for improving sepsis outcomes, where each hour of delayed treatment has been associated with roughly an 4-8% increase in mortality (Kumar et al., 2006; Seymour et al., 2017). To help address this problem, clinicians have proposed new definitions for sepsis (Singer et al., 2016), but the fundamental need to detect and treat sepsis early still remains, and basic questions about the limits of early detection remain unanswered. The PhysioNet/Computing in Cardiology Challenge 2019 provides an opportunity to address these questions.

Challenge Data

Data used in the competition is sourced from ICU patients in three separate hospital systems. Data from two hospital systems will be publicly available; however, one data set will be censored and used for scoring. The data for each patient will be contained within a single pipe-delimited text file. Each file will have the same header and each row will represent a single hour's worth of data. Available patient co-variates consist of Demographics, Vital Signs, and Laboratory values, which are defined in the tables below.

The following time points are defined for each patient:

${t}_{\mathrm{suspicion}}$
1. Clinical suspicion of infection identified as the earlier timestamp of IV antibiotics and blood cultures within a specified duration.
2. If antibiotics were given first, then the cultures must have been obtained within 24 hours. If cultures were obtained first, then antibiotic must have been subsequently ordered within 72 hours.
3. Antibiotics must have been administered for at least 72 consecutive hours to be considered.
${t}_{\mathrm{SOFA}}$
The occurrence of end organ damage as identified by a two-point deterioration in SOFA score within a 24-hour period.
${t}_{\mathrm{sepsis}}$
The onset time of sepsis is the earlier of ${t}_{\mathrm{suspicion}}$ and ${t}_{\mathrm{SOFA}}$ as long as ${t}_{\mathrm{SOFA}}$ occurs no more than 24 hours before or 12 hours after ${t}_{\mathrm{suspicion}}$; otherwise, the patient is not marked as a sepsis patient. Specifically, if ${t}_{\text{suspicion}}-24\le {t}_{\text{SOFA}}\le {t}_{\text{suspicion}}+12$, then ${t}_{\text{sepsis}}=\mathrm{min}\left({t}_{\text{suspicion}},{t}_{\text{SOFA}}\right)$.
Table 1: Columns in each training data file.
Vital signs (columns 1-8)
HRHeart rate (beats per minute)
O2SatPulse oximetry (%)
TempTemperature (Deg C)
SBPSystolic BP (mm Hg)
MAPMean arterial pressure (mm Hg)
DBPDiastolic BP (mm Hg)
RespRespiration rate (breaths per minute)
EtCO2End tidal carbon dioxide (mm Hg)
Laboratory values (columns 9-34)
BaseExcessMeasure of excess bicarbonate (mmol/L)
HCO3Bicarbonate (mmol/L)
FiO2Fraction of inspired oxygen (%)
pHN/A
PaCO2Partial pressure of carbon dioxide from arterial blood (mm Hg)
SaO2 Oxygen saturation from arterial blood (%)
ASTAspartate transaminase (IU/L)
BUNBlood urea nitrogen (mg/dL)
AlkalinephosAlkaline phosphatase (IU/L)
Calcium(mg/dL)
Chloride(mmol/L)
Creatinine(mg/dL)
Bilirubin_directBilirubin direct (mg/dL)
GlucoseSerum glucose (mg/dL)
LactateLactic acid (mg/dL)
Magnesium(mmol/dL)
Phosphate(mg/dL)
Potassium(mmol/L)
Bilirubin_totalTotal bilirubin (mg/dL)
TroponinITroponin I (ng/mL)
HctHematocrit (%)
HgbHemoglobin (g/dL)
PTTpartial thromboplastin time (seconds)
WBCLeukocyte count (count*10^3/µL)
Fibrinogen(mg/dL)
Platelets(count*10^3/µL)
Demographics (columns 35-40)
AgeYears (100 for patients 90 or above)
GenderFemale (0) or Male (1)
Unit1Administrative identifier for ICU unit (MICU)
Unit2Administrative identifier for ICU unit (SICU)
ICULOSICU length-of-stay (hours since ICU admit)
Outcome (column 41)
SepsisLabelFor sepsis patients, SepsisLabel is 1 if $t\ge {t}_{\text{sepsis}}-6$ and 0 if $t<{t}_{\text{sepsis}}-6$. For non-sepsis patients, SepsisLabel is 0.

Objective of the Challenge

The goal of this Challenge is the early detection of sepsis using physiological data. For the purpose of the Challenge, we define sepsis according to the Sepsis-3 guidelines, i.e., a two-point change in the patient's Sequential Organ Failure Assessment (SOFA) score and clinical suspicion of infection (as defined by the ordering of blood cultures or IV antibiotics) (Singer et al., 2016).

The early prediction of sepsis is potentially life-saving, and we challenge participants to predict sepsis 6 hours before the clinical prediction of sepsis. Conversely, the late prediction of sepsis is potentially life-threatening, and predicting sepsis in non-sepsis patients (or predicting sepsis very early in sepsis patients) consumes limited hospital resources. For the challenge, we designed a utility function that rewards early predictions and penalizes late predictions as well as false alarms.

We ask participants to design and implement a working, open-source algorithm that can, based only on the clinical data provided, automatically identify a patient's risk of sepsis and make a positive or negative prediction of sepsis for every time interval. The winners of the Challenge will be the team whose algorithm gives predictions with the highest utility score for the patients in the hidden test set.

Accessing the Data

• Click here to download the complete training database (42 MB), consisting of two parts: training set A (20,336 subjects) and B (20,000 subjects).

The Challenge data repository contains one file per subject (e.g., training/p00101.psv for the training data).

Each training data file provides a table with measurements over time. Each column of the table provides a sequence of measurements over time (e.g., heart rate over several hours), where the header of the column describes the measurement. Each row of the table provides a collection of measurements at the same time (e.g., heart rate and oxygen level at the same time). The table is formatted in the following way:

HR |O2Sat|Temp|...|HospAdmTime|ICULOS|SepsisLabel
NaN|  NaN| NaN|...|        -50|     1|          0
86|   98| NaN|...|        -50|     2|          0
75|  NaN| NaN|...|        -50|     3|          1
99|  100|35.5|...|        -50|     4|          1


There are 40 time-dependent variables HR, O2Sat, Temp ..., HospAdmTime, which are described here. The final column, SepsisLabel, indicates the onset of sepsis according to the Sepsis-3 definition, where 1 indicates sepsis and 0 indicates no sepsis. Entries of NaN (not a number) indicate that there was no recorded measurement of a variable at the time interval.

Note: spaces were added to this example to improve readability. They will not be present in the data files.

Entries should be submitted through the new cloud submission system for the official phase of the Challenge. Please see here for detailed instructions.

Given a table of clinical measurements (columns) over time (rows), your entry must report the risk of sepsis (a real number) with a binary sepsis prediction (0 or 1) at the k+6th hour using the first k hours of a patient’s clinical record (see the “Accessing the Data” section). We will repeat this process for k = 1, 2, …, N - 6, where N is the number of hours in a given recording, and for each patient in the test set. See the sample prediction and evaluation code (here) for details.

Scoring

Your final algorithm will be graded for its binary classification performance using a utility function that we created for the Challenge. This utility function rewards classifiers for early predictions of sepsis and penalizes them for late/missed predictions and for predictions of sepsis in non-sepsis patients.

We first define a score $U\left(s,t\right)$ for each prediction, i.e., for each patient $s$ and each time interval $t$ (each line in the data file):

The following figure illustrates the utility function for a sepsis patient (upper plot) with ${t}_{\mathrm{sepsis}}$ = 48 as an example and a non-sepsis patient (lower plot):

This utility function rewards or penalizes classifiers using their predictions on each patient:

• For patients that eventually have sepsis (i.e., with at least one SepsisLabel entry of 1), we reward classifiers that predict sepsis between 12 hours before and 3 hours after ${t}_{\mathrm{sepsis}}$, where the maximum reward is a parameter (1.0). We penalize classifiers that do not predict sepsis or predict sepsis more than 12 hours before ${t}_{\mathrm{sepsis}}$, where the maximum penalty for very early detection is a parameter (0.05) and the maximum penalty for late detection is also a parameter (-2.0).
• For patients that do not eventually have sepsis (i.e., all SepsisLabel entries of 0), we penalize classifiers that predict sepsis, where the maximum penalty for false alarms is a parameter (0.05; equal to the very early detection penalty). We neither reward nor penalize those that do not predict sepsis.

We then compute a score for a classifier by summing $U\left(s,t\right)$ over each prediction, i.e., over each patient $s$ and each time interval $t$ (each line in the data file):

$U total = ∑ s ∈ S ∑ t ∈ T ( s ) U ( s , t )$

To improve interpretability, we normalized the above classifier score so that the optimal classifier (highest possible score) receives a normalized score of 1 and that a completely inactive classifier (no positive predictions) receives a normalized score of 0:

$Unormalized=Utotal-Uno predictionsUoptimal-Uno predictions$

Each classifier receives a ${U}_{\text{normalized}}$ score, and the classifier with the highest ${U}_{\text{normalized}}$ score wins.

A Python implementation of the scoring metric is available here.

Sample Submission

A simple example algorithm is provided and may be used as a template for your own submission. MATLAB ( here), Python (here), and R ( here) implementations are available.

Entrants may have an overall total of up to 15 submitted entries over both the unofficial and official phases of the competition (see Table 2).

All deadlines occur at 11:59pm GMT (UTC) on the dates mentioned below. If you do not know the difference between GMT and your local time, find out what it is before the deadline!

Table 2: Challenge deadlines.
Start Entry limit End
Unofficial Phase 6 February 5 14 April
[Hiatus] 15 April 0 21 April
Official Phase 22 April 10 25 August

All official entries must be received no later than 11:59pm GMT on 25 August. In the interest of fairness to all participants, late entries will not be accepted or scored. Entries that cannot be scored (because of missing components, improper formatting, or excessive run time) are not counted against the entry limits.

To be eligible for the open-source award, you must do all of the following:

1. Entering an Abstract to CinC: Submit an acceptable abstract (about 300 words) on your work on the Challenge to Computing in Cardiology no later than 15 April. Include the overall score for your Phase I entry in your abstract. Please select “PhysioNet/CinC Challenge” as the topic of your abstract, so it can be identified easily by the abstract review committee. You will be notified if your abstract has been accepted by email from CinC during the first week in June.
2. Submit at least one open-source entry that can be scored during the second phase (between 12:01am GMT on 22 April and 11:59pm GMT on 25 August). Only your final entry will count for ranking.
3. Submit a full (4-page) paper on your work on the Challenge to CinC no later than the deadline of conference paper submission.
4. Attend CinC 2019 (8-11 September 2019) in Singapore and present your work there.

Please do not submit analysis of this year’s Challenge data to other Conferences or Journals until after CinC 2019 has taken place, so the competitors are able to discuss the results in a single forum. We expect a special issue from a journal to follow the conference and encourage all entrants (and those who missed the opportunity to compete or attend CinC 2019) to submit extended analysis and articles to that issue, taking into account the publications and discussions at CinC 2019.

Non open source licenses

Software with non open-source licenses will be scored but not ranked in the official competition. All scores will be made public. At the end of the competition, all software will be posted publicly, and therefore, automatically mirrored on several sites around the world. We have no control over these sites so cannot remove code even on request. Before the end of the competition your code is not publicly available and you can choose to withdraw from this event up until the closing date of the competition in August.

However, there is one caveat for all non open source code: The organizers reserve the right to retain and use a copy of the code for non-commercial use. This allows us to re-score if definitions change, and validate any claims made by competitors.

Submitting an Abstract to the Conference

Your abstract is intended as preliminary evidence that you will attend the conference and have meaningfully contributed to the unofficial phase. If you have not been successful in receiving a score from our scoring system, don't worry. Report cross validated scores on the available public data we have posted and your methods in detail. Do not describe the challenge. It's a waste of valuable words - the reviewers know what the Challenge is this year. We also know your results and your methods will change. Your final abstract (which will accompany the preprint you post to the conference in early September before the conference begins) will be very different. The point of the abstract is to decide if you have an original idea, promising/logical approaches and can express these coherently. It helps us allocate limited oral presentations and poster space to the best entries. [If you do get rejected, there may be a 'wild card' chance to enter later on in the Challenge, but if we able to do this, it will be for an exceptionally good entry, so please don't rely on this.]

Attending the Conference

If your abstract is accepted, then you must log in to the conference site and agree that you will attend. You must also submit a full article describing your results and mark it as a preprint (for others to read) by 4 September. (Don't forget that the competition deadline is noon GMT on 25 August - this deadline will not be extended.)

More details about the conference will follow.

Look out for future announcements via the community discussion forum.

After the Challenge

We hope to run a special issue with a closing date of (to be decided). We will therefore encourage competitors (and non-competitors) to submit updates and further reworks based on the Challenge after the award ceremony at the Computing in Cardiology Conference in Singapore in September.

Obtaining complimentary MATLAB licenses

MathWorks has generously decided to sponsor this Challenge by providing complimentary licenses to all teams that wish to use MATLAB. Users can apply for a license and learn more about MATLAB support by visiting the PhysioNet Challenge page from MathWorks. If you have questions or need technical support, then please contact MathWorks at studentcompetitions@mathworks.com.

Google has generously agreed to provide $50,000 in Google Cloud Platform (GCP) credits for this Challenge. We will award these to the 100 top performing teams, probably in May (at a date yet to be determined). This will hopefully provide an added incentive to submit more entries earlier on, and give teams the maximum opportunity to learn before spending money in the cloud. At the time of launching this Challenge, Google Cloud offers multiple services for free on a one-year trial basis and$300 in cloud credits. See here for more information. Additionally, if teams are based at an educational institution in selected countries, they can access free GCP training online here.