Prospective validation of an uncertainty-aware machine learning model for secondary progressive multiple sclerosis classification

Background: Relapsing-remitting multiple sclerosis (RRMS) often progresses to secondary-progressive MS (SPMS), but diagnosis is typically retrospective, delaying potential interventions. Existing rule-based and machine learning approaches often lack prospective validation and provide no uncertainty estimates, limiting clinical adoption. Objective: To prospectively evaluate a conformal prediction (CP)-enabled random forest model for reproducing clinician-recorded disease course labels in the Swedish Multiple Sclerosis Registry (SMSReg) and compare its classifications with the Lorscheider criteria. Methods: The CP model was trained and calibrated on SMSReg-2022 (10,787 individuals, 85,461 hospital visits) and then validated on SMSReg-2025 (15,719 individuals, 45,077 hospital visits; not included in the SMSReg-2022 training/calibration/testing datasets). Performance was assessed at the hospital visit and individual levels and compared with the Lorscheider criteria. Results: The model maintained good calibration on the prospective dataset. At the per-hospital visit-level, the model accuracies were 92% (RRMS) and 95% (SPMS). At the individual level, sensitivity was 98%, specificity 89%, and F1-score 0.75. With clinician-recorded labels as reference, the model showed higher sensitivity than the Lorscheider criteria (98% vs. 71%), more transition assignments (90% vs. 57%), and higher F1-score (0.76 vs. 0.69). Conclusion: The CP-enabled model prospectively reproduced clinician-recorded disease course labels with calibrated uncertainty, supporting harmonization and decision support.

5 Key Takeaways

1. The CP-enabled model demonstrated higher sensitivity than the Lorscheider criteria (98% vs. 71%) in identifying SPMS transitions using clinician-recorded labels as reference.
2. The model assigned more transition events than the Lorscheider criteria (90% vs. 57%) and achieved a higher F1-score (0.76 vs. 0.69) when compared to clinician-recorded labels.
3. The model maintained calibrated uncertainty on the prospective validation dataset, supporting its potential for clinical decision support in MS disease course classification.

OpenRounds transcript synthesis · Verify against the source