Multimodal Artificial Intelligence for Cause-of-Death Assignment From Verbal Autopsy Records: A National Benchmark Study in South Africa

BACKGROUND Timely cause-of-death (COD) information guides health policy, yet many deaths in low- and middle-income countries are not medically certified. In Africa, fewer than 10% of deaths are. Verbal autopsy (VA) yields a structured symptom questionnaire and a free-text narrative. Mortality surveillance increasingly records the narrative alongside the questionnaire, but the algorithms used to produce VA-based statistics were designed for structured responses only, so narratives inform physician COD assignment but do not contribute to algorithmic COD assignment. We evaluated whether supervised models could use both modalities to improve COD assignment. METHODS We analyzed 4,999 VA records for deaths at ages 12 years or older from a national sample of registered deaths in South Africa (RSA), using physician-assigned CODs as the reference standard. We trained unimodal and multimodal pipelines on the labelled records: pretrained language models (PLMs) fine-tuned on narratives alone, machine learning (ML) models on structured responses alone, and the two combined at the data, feature, and decision levels. We also prompted a locally deployed generative large language model (LLM) on the same unimodal and multimodal inputs, without parameter updates. All were compared with InSilicoVA, the operational comparator representing current automated practice. We evaluated individual-level assignment, population-level agreement, calibration, and generalization across provinces in RSA, reporting these metrics rather than ranking pipelines on any one of them. RESULTS Every supervised pipeline improved on InSilicoVA by 8.5-15.9 percentage points in top-1 accuracy and 3.5-7.1 points in cause-specific mortality fraction (CSMF) accuracy. The narrative and structured pipelines performed similarly in aggregate but differed by cause: 63.9% of deaths received identical predictions, 11.6% were...