896 - Evaluation of Patient Education Materials from Large-Language Models on Complex Regional Pain Syndrome: Are Patients Learning?

Friday, February 23, 2024

5:00 PM – 6:30 PM EDT

Presenting Author(s)

AG

Anuj Gupta, BS

Medical Student
Texas A&M School of Medicine
Allen, Texas, United States

Co-Author(s)

WH

William Hlavinka, BS

Medical Student
Texas A&M School of Medicine
Dallas, Texas, United States
TS

Tarun R. Sontam, BS

Medical Student
Texas A&M School of Medicine
Frisco, Texas, United States
MA

Mohammed Abdullah, MD

Physician
The University of Pennsylvania
Philadelphia, Pennsylvania, United States

Objectives: The internet is a primary resource for patients seeking information on Complex Regional Pain Syndrome (CRPS). With 80% of users searching for health data online, its influence on medical decisions is evident, as shown in a 2006 survey where 55% reported increased access to knowledge. This study evaluates the accuracy and readability of CRPS information from ChatGPT using standardized scoring to analyze AI-generated content.

Design: Google Web Search was performed using the term entered as “Complex Regional Pain Syndrome,” and the first ten FAQ’s were recorded. The questions were categorized into question topics: fact, policy, and value using the Rothwell Classification System. The website sources are categorized using previously established categories from the literature. The top ten questions were recorded and compared to those from Google. ChatGPT was then asked to answer each of the ten Google FAQs individually and provide sources. Answer content was analyzed using readability analysis.

Results: Most Frequently Asked Questions on Google vs ChatGPT: Of the ten most frequently asked questions on Google and ChatGPT, 40% were the same between the two search engines. A comparison between the answer content revealed that the ChatGPT answers (289 ± 80 words) were significantly longer than Google’s answers (41 ± 7 words) (p-value < 0.0001). The Gunning Fog index showed Google's grade level to be 12.61 ± 3.77 and ChatGPT to be at 14.31 ± 1.73 (p = 0.02).

Conclusions: Our study showed a 40% overlap in the top CRPS questions between ChatGPT and Google. Both had similar question categorizations, but ChatGPT's answers were longer and more academically sourced than Google's. Google's answers were more readable per the Flesh Reading Ease Score and Gunning Fog Index. However, both platforms' information exceeded the recommended reading level by NIH and AMA, potentially hindering patient comprehension.