A Reproducible Decomposed Large Language Model Pipeline for Structured Abstraction of Multimodal Neuroimaging Reports: Development and Independent Validation in 1,669 Emergency Department Encounters for Dizziness and Vertigo

ObjectiveTo develop and independently validate a decomposed, modality-specific large language model (LLM) pipeline that abstracts structured variables from the free-text neuroimaging reports of a multimodal stroke workup, and to validate the resulting structured dataset for reuse in downstream research. Materials and MethodsA frozen pipeline (temperature 0) extracting 22 variables across CT head, CT angiography, CT perfusion, and MRI brain was applied to 1,669 ED encounters for dizziness or vertigo; 11 variables were pre-specified as primary (Tier 1). Reproducibility used three same-session runs and a nine-week re-run; three frontier models (Claude Sonnet 4.6, GPT-5.4, Gemini 3.5 Flash) measured cross-model agreement (Cohens/Fleiss {kappa}). Accuracy was assessed in an independent 200-encounter cohort (150 random, 50 enriched for rare findings); 100 were double-read by two blinded reviewers with discordances adjudicated. ResultsTest-retest reproducibility was high (Tier 1 Fleiss {kappa}=0.993; 99.8% exact) and stable nine weeks later ({kappa}=0.992); inter-model agreement was {kappa}=0.962. Against the adjudicated reference, pooled Tier 1 accuracy was 0.971 to 0.988 across models (Cohens {kappa} 0.959 to 0.982), comparable to inter-reviewer agreement ({kappa}=0.932). Accuracy was 0.980 to 0.983 in random and 0.955 to 0.977 in enriched encounters (agreement on model-flagged findings, not sensitivity). DiscussionA decomposed LLM pipeline produced reproducible, model-portable, stable, and accurate multimodal abstraction. Because schema design and validation are up-front costs, such pipelines scale to larger research cohorts. ConclusionThe quantified per-field error rates support reuse of this dataset with field-specific review, and such pipelines offer a practical path to large-scale, reproducible research datasets.