1540 - Application of Chat-Bot Artificial Intelligence in Education of Traumatic Brain Injury: A Comparative Readability Analysis of ChatGPT and MSKTC Fact Sheets

Friday, February 23, 2024

5:00 PM – 6:30 PM EDT

Presenting Author(s)

ML

Matthew J. Lee, BA

Medical Student
Hackensack Meridian School of Medicine
Nutley, New Jersey, United States

Co-Author(s)

AC

Angelo Cadiente, BS

Medical Student
Hackensack Meridian School of Medicine
Nutley, New Jersey, United States
JC

Jamie Chen, BS

Medical Student
Hackensack Meridian School of Medicine
Nutley, New Jersey, United States
YZ

Yi Zhou, MD

Resident Physician
Hackensack Meridian JFK Johnson Rehabilitation Institute
Edison, New Jersey, United States
BG

Brian Greenwald, MD

Associate Medical Director
Hackensack Meridian JFK Johnson Rehabilitation Institute
Edison, New Jersey, United States

Objectives: Accessibility to comprehensible information is crucial for patients and caregivers after a traumatic brain injury (TBI). While fact sheets offer standardized information, the rise of chat-bot artificial intelligence (AI) presents an alternative source of information. This study compares readability metrics of responses from ChatGPT-3.5, ChatGPT-4 and the most-viewed TBI fact sheets from the Model Systems Knowledge Translation Center (MSKTC).

Design: Five TBI fact sheets were chosen from the MSKTC based on total views from 12/1/22-5/31/2023. Subheadings from the fact sheets were formatted into questions to simulate real-world queries. These questions were posed to both ChatGPT-3.5 and ChatGPT-4. Six scales — Flesch Kincaid Reading Ease, Flesch Kincaid Grade Level, Gunning Fog Score, Smog Index, Coleman Liau Index, and Automated Readability Index — were used to determine readability for the AI responses and fact sheets.

Results: Analysis of 38 prompts demonstrated that the MSKTC fact sheets significantly outperformed ChatGPT-3.5 and ChatGPT-4 responses across all readability metrics (p < 0.0001). For Flesch Kincaid Reading Ease scores, the standardized mean difference (SMD) was 2.21 (p < 0.0001) and 2.03 (p < 0.0001) against ChatGPT-3.5 and ChatGPT-4, respectively. The Flesch Kincaid Grade Level showed SMDs of -1.78 (p < 0.0001) and -1.48 (p < 0.0001) against ChatGPT3.5 and ChatGPT4, respectively. Similar significant differences were observed across other scales. Notably, ChatGPT-4 exhibited modest yet significant improvements in readability metrics compared to ChatGPT-3.5, with SMDs ranging from 0.56-0.75.

Conclusions: While ChatGPT offers potential as an alternative information source, MSKTC fact sheets remain superior in readability. ChatGPT-4, while significantly outperforms ChatGPT-3.5, still does not match the clarity of specialized TBI fact sheets. Further studies need to be undertaken to assess the quality of responses despite decreased readability.