# AnesLLM — Evaluation Guide This guide explains how to load the dataset and evaluate a model against it. ## 1. Loading the data The data lives in `dataset/data/{dev,train,test}/case.jsonl`: one JSON object per line, one line per decision window, all windows of a case in the same file. ```python import json from pathlib import Path def load(split): for f in sorted((Path("dataset/data") / split).glob("case*.jsonl")): for line in open(f, encoding="utf-8"): yield json.loads(line) for rec in load("test"): x = rec["input"] # dict with 280 pre-decision features out = rec["output"] # dict with result_1 / result_aux / result_real ``` ## 2. Record contract ```json { "window_id": "case14_w127", "case_id": "14", "input": { "...280 fields, everything observable BEFORE the decision..." }, "output": { "result_1": { "action": "reduce_hypnotic", "ratio": 0.48 }, "result_aux": { "action": "reduce_opioid", "ratio": 0.45 }, "result_real": "reduce_hypnotic" } } ``` - **`input`** is the model input. Nulls are encoded as JSON `null`. The nested `*_json` fields are JSON objects (e.g. `vitals_json`, `staleness_json`, `actions_history_json`). - **`output.result_real`** is the ground-truth action (no ratio) — the action the clinician actually performed. - **`output.result_1`** and **`output.result_aux`** are the primary and alternative consensus actions: top-2 of a LightGBM model trained to predict the clinician's action (case-grouped out-of-fold predictions); `ratio` is the model probability. ## 3. Action vocabulary Ground truth (`result_real`) is always one of: - `increase_hypnotic`, `reduce_hypnotic`, `increase_opioid`, `reduce_opioid`, `no_action` The consensus alternatives (`result_1`, `result_aux`) may additionally be `vasopressor` (a valid *option*, but never a ground-truth label). Ventilation actions never appear. ## 4. Metrics Two standard metrics capture how well a model reproduces the clinician: - **top-1 accuracy** — the model's single best action equals `result_real`: `acc = mean(argmax(model(x)) == result_real)`. - **top-2 recall** — `result_real` is among the model's two best actions. Reference numbers — agreement between the generated alternatives and the real action, on the full set: | metric | value | |---|---| | top-1 accuracy | 55.8% | | top-2 recall | 80.4% | Because the true action is not always unique (several actions can be acceptable), the dataset also provides the alternative action and supports external-validity evaluation (see `docs/dataset_design.md`, §5–6). ## 5. Difficulty Each case has a difficulty label in `dataset/reports/split_manifest.csv` (`case_id, difficulty, split, n_windows, null_score`): - `easy` — high consensus confidence (≥ p75 of the per-case confidence), - `hard` — low consensus confidence (< p25), - `medium` — otherwise. Use these to report performance by difficulty (a model should do better on `easy` than on `hard`). ## 6. Splits - **`train`** — train your model here. - **`test`** — balanced (stratified by difficulty) and curated to have **no physiologically impossible values**; the reference for final evaluation. - **`dev`** — 25 edge cases with heavy missing data and deliberately retained bad values; use it to harden your data-loading and null handling, not for performance reporting. ## 7. Practical notes - **Nulls**: coverage varies a lot per variable (see `dataset/reports/coverage_by_variable.csv`). The `dev` split is the stress test for null handling. - **Vitals**: current values are also available as top-level `*_current` fields; the raw time series are in `input.vitals_json` (signals: MAP, HR, BIS, SBP, DBP, SpO2, EtCO2). - **No leakage**: `input` contains no post-decision information (audited — see `docs/audit_report.md`). - **Identifiers**: drop `pt_caseid` and `pt_subjectid` before training or release.