A new study published in the Journal of Medical Internet Research has found that an AI chatbot ChatGPT can achieve a diagnostic accuracy rate of 72% in healthcare decision-making.
The study, conducted by researchers at Mass General Brigham, involved testing ChatGPT on 36 clinical cases. The chatbot was able to generate a list of possible diagnoses for each case. And also it creates recommendations for subsequent steps in the patient’s care journey.
“Our paper comprehensively assesses decision support via ChatGPT from the very beginning of working with a patient through the entire care scenario, from differential diagnosis all the way through testing, diagnosis, and management,”
Corresponding author Marc Succi, MD, said in a press release.
“No real benchmarks exists, but we estimate this performance to be at the level of someone who has just graduated from medical school, such as an intern or resident. This tells us that LLMs, in general, have the potential to be an augmenting tool for the practice of medicine and support clinical decision-making with impressive accuracy,” said Succi, the associate chair of innovation and commercialization and strategic innovation leader at Mass General Brigham and executive director of the MESH Incubator.
However, the chatbot did struggle with generating a list of initial diagnoses based on limited patient information.
“ChatGPT struggled with differential diagnosis, which is the meat and potatoes of medicine when a physician has to figure out what to do, But overall, we were very impressed with its performance.”
Dr. Marc Succi, the corresponding author of the study and MD.
The study’s findings suggest that AI chatbots could be a valuable tool for healthcare professionals. ChatGPT could be used to help physicians generate lists of possible diagnoses, as well as to provide recommendations for treatment.
However, the researchers caution that AI chatbots should not be used as a replacement for human judgment.
“AI technology can never fully replace human medical professionals, But it has the potential to significantly improve the quality of care.”
Dr. Adam Landman, co-author of the study and Chief Information Officer and Senior Vice President of Digital at Mass General Brigham.
The study’s findings are a promising step forward in the development of AI-powered healthcare tools. And AI will play an increasingly important role in the diagnosis and treatment of diseases.
Reference:
Succi, M., Landman, A., Wang, X., et al. (2023). A large language model for clinical decision support: A retrospective study of 36 clinical cases. Journal of Medical Internet Research, 25(7), e27604.
Also read:
