Artificial intelligence-based diagnostic decision support for primary care of older adults using collective intelligence and imitation learning.

Published in Journal of the American Medical Informatics Association : JAMIA. Abstract: Older adults face disproportionate risk of diagnostic errors in primary care, yet effective diagnostic decision support systems (DDSSs) remain lacking. Existing artificial intelligence (AI)-based DDSSs rely on gold-standard labels infeasible to obtain, have narrow diagnostic scope, and fail to reflect real-world uncertainty. We developed and validated a deep learning DDSS using imitation learning and collective clinician intelligence to generate diagnostic and order recommendations. We trained a multi-label neural network on structured and unstructured electronic health record data from primary care encounters of adults ≥65 years old to imitate clinician decisions. Models were evaluated using discrimination, calibration, threshold-based, and composite metrics on a temporally held-out test set. Clinical validity was assessed using a randomized, blinded equivalence study comparing model-generated recommendations with observed clinician decisions. The study included 707 598 primary care encounters, generating recommendations across 669 diagnoses and 1000 orders. The general model demonstrated excellent performance (micro c-statistic 0.995 [95% CI 0.995-0.996], macro c-statistic 0.896 [0.888-0.897], micro F1-score 0.904 [0.903-0.905], macro F1-score 0.384 [0.381-0.385], integrated calibration index (ICI) 0.0025 [0.0024-0.0026]). Order prediction was lower (micro c-statistic 0.812 [0.811-0.814], macro c-statistic 0.786 [0.777-0.786], micro F1-score 0.305 [0.303-0.307], macro F1-sc...