University of Maryland School of Medicine (UMSOM) researchers tested the AI model’s accuracy in giving health advice.
A recent study has shed light on the reliability of ChatGPT, an artificial intelligence (AI) language model created by OpenAI, in providing accurate health advice.
According to the research published in the journal Radiology, ChatGPT correctly offers health advice 88.25% of the time, suggesting it could potentially be a helpful tool for individuals seeking health information.
They evaluated the AI model’s responses to a wide range of health-related questions, sourced from medical professionals, internet users, and medical literature.
The dataset consisted of 4,000 questions, with 1,000 taken from each of the four aforementioned sources. The study assessed the accuracy of the AI model’s responses based on their alignment with the medical consensus and their ability to address the posed questions effectively.
“We found ChatGPT answered questions correctly about 88 percent of the time, which is pretty amazing,” said study corresponding author Paul Yi , MD, Assistant Professor of Diagnostic Radiology and Nuclear Medicine at UMSOM and Director of the UM Medical Intelligent Imaging Center, reported News Medical.
“It also has the added benefit of summarising information into an easily digestible form for consumers to easily understand.”
When asked about the signs of breast cancer, who is at risk, and the cost, age, and frequency of mammogram recommendations, ChatGPT provided accurate responses.
This accuracy rate, however, varied depending on the source of the questions. Questions from medical professionals had the highest rate of accurate responses at 94.1%, followed by questions from medical literature (91.4%), and finally, questions from internet users (81.7%).
Despite its impressive overall performance, the study noted that ChatGPT still has limitations and may provide inaccurate or incomplete advice in certain situations.
These results highlight the potential for AI-driven health advice services to assist in addressing the increasing demand for healthcare information.
In addition to being a valuable resource for individuals seeking health advice, ChatGPT could also prove to be a useful tool for healthcare professionals by providing them with a supplementary source of information, which could enhance patient care.
However, researchers caution against relying solely on AI tools like ChatGPT for critical health decisions. Although the AI model demonstrates a high degree of accuracy, it may still provide incomplete or incorrect advice in some cases.
The study advises individuals to consult qualified medical professionals when making important health decisions.
As AI technology continues to advance, it is likely that AI-driven health advice services will become more accurate and reliable.
The potential of ChatGPT and other AI models in transforming the healthcare landscape cannot be underestimated. Nevertheless, until these tools can consistently offer flawless health advice, the importance of consulting qualified medical professionals remains paramount.
