Anatomy of an Oncology Imaging Dataset

Model quality is bounded by dataset quality, and most published failures of generalisation are dataset failures described after the fact. This is what to interrogate before trusting a result.

Labels

• What is the reference standard — histology, outcome, or expert annotation?
• How many annotators, and what was their agreement? A model cannot be shown to beat a label it was trained on.
• Were annotators blinded to the information the model is meant to predict?

Splits

• Split at patient level. Splitting by image, slice or lesion lets one patient appear in both training and test, and inflates every reported figure.
• Was the test set touched during development? A test set consulted repeatedly has become a validation set.
• Are institutions separated across splits, or mixed?

Leakage risks

• Scanner or site signatures that correlate with outcome, so the model learns the hospital.
• Text burned into images, laterality markers, or acquisition metadata left in.
• Preprocessing statistics computed over the whole dataset before splitting.
• Duplicate or near-duplicate studies across splits.

Reporting essentials

• Population, inclusion criteria and case mix, with prevalence stated.
• Acquisition parameters and vendor distribution.
• Subgroup performance, not only aggregate.
• What the model does with inputs outside its competence.