Prospective Shadow-Mode Evaluation of an Artificial Intelligence Tool for Intracranial Aneurysm Detection on CT Angiography: Incremental Yield and Operational Impact.

Published in Journal of the American College of Radiology : JACR. Abstract: To evaluate the operational performance of an FDA-cleared artificial intelligence (AI) algorithm for brain CT angiography (CTA) aneurysm detection and the incremental value of combining AI with radiologists. Prospective shadow-mode study of consecutive brain CTAs (November 7, 2023, to December 19, 2023) was performed with radiologists blinded to AI. Aidoc's (Tel Aviv, Israel) AI algorithm processed CTAs, and natural language processing extracted radiology report results. Discordances underwent neuroradiologist adjudication with AI unblinding. Performance metrics included AI:radiologist incremental detection ratio, relative enhanced detection rate (rEDR), gain-to-pain ratio (GPR), and number-needed-to-examine (NNE). Performance metrics are estimated under a hybrid reference standard wherein only discordances are adjudicated. Among 3,856 CTAs, examination positive rate was 5.1% (195 of 3,856). Radiologist-AI concordance was 96.3% (3,714 of 3,856). Sensitivity of AI alone (0.846, 0.787-0.894) exceeded radiologist alone (0.718, 0.649-0.779), with similar specificity (radiologist: 0.985, 0.981-0.989; AI: 0.987, 0.983-0.991). AI surfaced additional aneurysms not identified by radiologists, yielding an rEDR of 39% (55 AI-only true-positives per 140 radiologist true-positives) and projecting a higher combined sensitivity for radiologists with AI. AI:radiologist incremental detection ratio was 1.83 (55 of 30), favorable for AI. Operationally, GPR was favorable at 1.20 (55 of 46), and ...