Model Validation Levels Explained

“Validated” is not one claim. It is four, in ascending order of what they let you conclude, and papers frequently report the weakest while implying the strongest.

The four levels

1. Internal validation. Held-out data from the same source, or cross-validation. Establishes the model learned something beyond noise. Tells you nothing about transfer.
2. Temporal validation. Later patients from the same institution. Adds evidence that the model survives drift in practice over time.
3. External validation. Different institutions, scanners and populations. The first level that supports a claim about use elsewhere. Performance almost always drops here, and the size of the drop is the most informative number in the paper.
4. Prospective validation. Applied in real workflow, on consecutive patients, with outcomes recorded. The only level that measures the clinician-and-model system rather than the model.

What each level cannot tell you

• Internal results say nothing about other scanners, other populations, or other case mixes.
• Strong discrimination says nothing about calibration, which is what matters when a threshold drives a decision.
• None of the first three levels captures automation bias, workflow fit, or whether any decision changed.

Ask which level a claim rests on before asking how good the number is. A modest figure from prospective use is worth more than an excellent one from internal validation.