# AnesLLM Dataset — Design and Creation
This document describes how the dataset was designed and built, end to end. The pipeline runs
in two stages: (1) extraction of the decision windows from the raw VitalDB signals, and
(2) enrichment and re-derivation of the *acceptable actions* and the final splits.
## 1. Source: VitalDB
[VitalDB](https://vitaldb.net) is a public dataset of high-resolution intraoperative biosignals
from **Seoul National University Hospital** (South Korea), comprising 6,388 surgical cases with
continuous waveforms (ECG, arterial pressure, plethysmography, capnography, BIS…) and clinical
records (demographics, labs, drugs, ventilator settings).
## 2. Step 1 — Extraction of decision windows
### 2.1 Window definition
Each decision is anchored to a point in time; everything *before* that point is the model's
input, and the action taken *at* that point is the label.
- **Vitals window**: the 10 minutes preceding the decision (sampled as a time series with
offsets from the decision point).
- **Action/event history**: the 20 minutes preceding the decision.
- **Decision point**: `t = 0`, the moment the anesthesiologist acts.
This 10-minute / 20-minute split was chosen as a compromise between having enough context to
characterize the hemodynamic state and keeping the window short enough that the state is still
the one being acted upon.
### 2.2 Action vocabulary and `no_action`
The raw signal of interest is the **infusion rate of the two drug families**:
- hypnotics (propofol; the volatile hypnotic sevoflurane is tracked but excluded from the
ground-truth action — see §2.3),
- opioids (remifentanil).
A ground-truth action is the *direction of change* of a drug infusion at the decision point:
- `increase_hypnotic` / `reduce_hypnotic` (hypnotic rate up/down),
- `increase_opioid` / `reduce_opioid` (opioid rate up/down),
- `no_action` — **no infusion-rate change was performed** at that moment.
The `no_action` rule is what turns a dense, mostly-stationary signal into a classification
problem: the large majority of intraoperative time is "do nothing", and the model must learn
when (and how) to intervene.
### 2.3 `result_real` (ground truth)
`result_real` is the fine-grained action the clinician actually performed, mapped to the five
canonical classes above. Two simplifications were applied:
- **`vasopressor`** is kept as a *possible option* (a recommendation system should be able to
propose it) but removed as a ground-truth class, because vasopressor management is a distinct
problem with different pharmacology.
- **Ventilation actions** (FiO₂, PEEP, tidal volume, respiratory rate) are removed from both
options and ground truth, scoping the dataset to hypnotic/opioid titration.
- **Volatile-agent changes** (sevoflurane, a volatile hypnotic) are removed the same way: they
were detected as a separate `increase_volatile` / `reduce_volatile` category and excluded
from both options and ground truth, so the `increase_hypnotic` / `reduce_hypnotic` action
reflects propofol changes only.
After filtering, the dataset contains **84,813 windows** from **3,168 cases**.
## 3. Step 2 — Enrichment
Two enrichments were added on top of the extracted windows:
- **Preoperative laboratories** (`lab_*`): last preoperative value per (case, test) and its age,
fetched from the VitalDB labs endpoint (34 tests).
- **Signal filling** (`vitals_json`): windows with missing vital signs were filled from the
VitalDB track API (HR, MAP, SBP, DBP, SpO₂, EtCO₂), keeping the real staleness of each value.
PPV/SVV were later **removed** because of heavy artifacts.
## 4. Acceptable actions — cluster analysis
The core modeling question: given a window, which set of actions is *acceptable*? Seven
candidate strategies were evaluated, each producing a top-2 `(result_1, result_aux)` with
confidence ratios:
| strategy | method |
|---|---|
| `knn` | k-NN, euclidean, K=50 (baseline) |
| `random_forest` | out-of-fold probabilities (3 folds) |
| `lightgbm` | out-of-fold probabilities (3 folds) |
| `logistic` | softmax logistic regression (3 folds) |
| `centroid` | softmax(−distance to class centroid) |
| `dwknn` | distance-weighted k-NN |
| `umap_knn` | UMAP (10 dims) + k-NN K=50 |
The neighborhood/pool for these models used **6 classes** (the 5 final classes plus
`vasopressor`), so the model can propose vasopressors without them being a target in the final
dataset.
## 5. External validation — three reviewers
The ground truth comes from the clinicians of the source hospital (Korea). To obtain
**external coherence** and avoid circular validation, three annotators **external to the Korean
hospital** reviewed a subset of **298 windows**. Each reviewer independently judged, for every
policy action proposed on a window, whether that action was **appropriate** (`appropriate`,
yes/no, blind to any algorithm). The `valid_actions` field of the review exports is the tool's internal reference answer (not a reviewer judgment) and is not used as an annotation.
## 6. Strategy selection: agreement with the real action
The strategy was selected on a single criterion: **agreement with the real action**
(`result_real`).
| strategy | top1 = real | real ∈ top2 |
|---|---|---|
| **lightgbm** | **55.8%** | **80.4%** |
| logistic | 53.7% | 78.7% |
| random_forest | 53.5% | 78.2% |
| dwknn | 51.2% | 75.6% |
| knn | 50.9% | 75.3% |
| umap_knn | 46.4% | 71.2% |
| centroid | 38.6% | 64.2% |
### External review (reported for transparency, not a selection criterion)
Three external reviewers judged a subset of 298 windows with their actual `appropriate` judgments
(`dataset/reports/human_concordance_redo.csv`). Under this metric `centroid` (50.9%) and
`logistic` (45.2%) score **higher** than LightGBM (43.0%), so the review does **not** single out
LightGBM as the best strategy:
| strategy | redo % of max (`appropriate`) | top1 coverage | top1 approved (majority) |
|---|---|---|---|
| centroid | 50.9% | 75.5% | 43.1% |
| logistic | 45.2% | 85.9% | 28.9% |
| lightgbm | 43.0% | 94.6% | 26.6% |
| dwknn | 40.8% | 88.3% | 26.2% |
| knn | 40.5% | 88.9% | 26.4% |
| random_forest | 35.9% | 92.3% | 25.1% |
| umap_knn | 33.9% | 78.9% | 27.7% |
**LightGBM was selected**: it has the best agreement with the real action. Its top-2 became
`result_1` / `result_aux`.
> The original `human_concordance.csv` (~99%) was scored against `valid_actions` (the tool's
> internal reference), not the reviewers' judgment; the table above is the corrected version
> using `appropriate`.
## 7. Difficulty split (easy / medium / hard)
Difficulty is defined **per case** (windows of a case stay together). The confidence of a window
is `result_1.ratio + result_aux.ratio`; the case score is the mean over its windows. With
percentiles computed at case level (p25 = 0.7772, p75 = 0.8318):
- `easy` — mean ≥ p75 (high model confidence),
- `hard` — mean < p25 (low confidence),
- `medium` — otherwise.
This yields a clean 25 / 50 / 25 split: **792 easy, 1,584 medium, 792 hard**.
## 8. Final splits (dev / train / test)
### 8.1 `dev` — engineered edge cases
25 cases selected for having the **highest null fraction** (heaviest missing data), to stress-test
systems that handle nulls poorly. The dev set **deliberately keeps bad physiological values**
(impossible hemodynamics from VitalDB artifacts); they are **not removed**, only **reported**.
The inventory of bad-value cases (with the offending fields and the reason) is maintained in
`audit/bad_cases.csv`:
- `dev` contains **1** bad case (`case3807`),
- `train` contains **864** bad cases — the reasons are the offending fields themselves
(`map_current`, `MAP_current`, `DBP_current` with negative/absurd blood pressure, plus
`lab_cr` and `mac` outliers), as listed in `audit/bad_cases.csv`.
### 8.2 `train` — the bulk
All remaining cases (2,994 after the test filtering below), used for training.
### 8.3 `test` — balanced and maximum quality
- **Representative**: stratified by difficulty (25 / 50 / 25).
- **No impossible values**: any case containing a physiologically impossible value was **moved
out of test into train**, guaranteeing that the evaluation set has the highest data quality.
Final test set: **149 cases, 0 windows with impossible values**.
## 9. Portability (transferability)
Despite the curation above, the test set remains **portable**: a drift analysis (KS test + PSI)
between `train` and `test` found no meaningful distribution shift — the `result_real` class
distribution is nearly identical (total-variation distance 0.011) and no feature drifts except
`lab_k` (PSI 0.14) and `lab_plt` (PSI 0.11). Models trained on `train` therefore transfer
fairly to `test`.
## 10. Privacy
VitalDB is de-identified at source, but the patient fields still carry two direct identifiers
(`pt_caseid`, `pt_subjectid`). A k-anonymity evaluation at window level shows that including
vital signs makes ~75% of windows unique (k = 1). For any public release, the direct
identifiers must be removed and the quasi-identifiers (demographics, clinical context, vitals)
generalized or suppressed.
## 11. Reproducibility
The final dataset and its splits are deterministic: the split assignment is recorded in
`dataset/split_manifest.csv`, and every case belongs to exactly one split. The full data
dictionary is in `dataset/reports/data_dictionary.csv`.