Artificial Intelligence-Driven Extraction of Patient-Reported Pain Outcomes in Osteoarthritis Using Large Language Models.

Published in Arthritis care & research. Abstract: Patient-reported outcome measures (PROMs) are central to assessing symptom burden and treatment response in Osteoarthritis (OA) but are often embedded in unstructured clinical notes. We evaluated whether prompt-engineered large language models (LLMs) could accurately extract or infer pain scores on a 0-10 scale from routine OA clinical documentation. We retrospectively analyzed 163 paired internal medicine and orthopedic clinic notes from patients in a prospective OA registry. A trained abstractor extracted explicitly documented pain scores from notes. For notes lacking explicit pain scores, physicians estimated pain severity from narrative documentation. The Llama 3.2-90B model, using Chain-of-Thought and Few-Shot prompting, inferred pain scores. Registry-collected pain scores served as a reference standard. Agreement was assessed using Kendall's Tau-b. Agreement between LLM-extracted and abstracted scores was strong for IM notes (n =132, τ = 0.76, p < 0.0001) and excellent for orthopedic notes (n = 65, τ = 0.95, p < 0.0001). The LLM identified 15 EMR-generated default zero scores in IM notes; after excluding them, agreement was perfect (τ = 1.0, p < 0.0001). For notes without explicit pain scores (31 IM and 98 orthopedic notes), LLM-based pain inference showed weak correlation with registry scores (IM: τ = 0.17; orthopedic: τ = 0.11), but performed comparably to physician estimation. LLM-physician agreement ranged from 54.2%-88.0% within ±1 point and 66.7%-100% within ±2 po...