ChatGPT bombs test on diagnosing kids’ medical cases with 83% error rate

Enlarge / Dr. Greg House has a better rate of accurately diagnosing patients than ChatGPT. (credit: Getty | Alan Zenuk/NBCU Photo Bank/NBCUniversa)

ChatGPT is still no House, MD.

While the chatty AI bot has previously underwhelmed with its attempts to diagnose challenging medical cases—with an accuracy rate of 39 percent in an analysis last year—a study out this week in JAMA Pediatrics suggests the fourth version of the large language model is especially bad with kids. It had an accuracy rate of just 17 percent when diagnosing pediatric medical cases.

The low success rate suggests human pediatricians won’t be out of jobs any time soon, in case that was a concern. As the authors put it: “[T]his study underscores the invaluable role that clinical experience holds.” But it also identifies the critical weaknesses that led to ChatGPT’s high error rate and ways to transform it into a useful tool in clinical care. With so much interest and experimentation with AI chatbots, many pediatricians and other doctors see their integration into clinical care as inevitable.