Dear HIFA colleagues,
Extracts below. Full text: https://www.bbc.co.uk/news/articles/cpd8l088x2xo
==
Using artificial intelligence (AI) chatbots to help seek medical advice can be "dangerous", a new study has found...
In the study, researchers asked nearly 1,300 participants to identify potential health conditions and a recommended course of action using different scenarios.
Some participants used large language model (LLM) AI softwares to receive a potential diagnosis and next steps, whereas others used more traditional methods - such as seeing a GP.
Researchers then evaluated the results, and found that the AI often provided a "mix of good and bad information" which users struggled to distinguish...
==
COMMENT (NPW): I have looked at the original study and the findings are a little more nuanced than the BBC reports. The paper does not include the word 'danger' or 'dangerous', despite this word being quoted in the headline. I have not had a chance to read the full text in depth, but the study appears to have compared how patients use Large Language Models [LLMs] versus a control group of patients that are invited to use whatever tools they like (such as a search engine). 'Despite LLMs alone having high proficiency in the task, the combination of LLMs and human users was no better than the control group in assessing clinical acuity and worse at identifying relevant conditions.'
I have emailed the corresponding auther Dr Mahdi for comment and will pass this on if I get a response.
HIFA profile: Neil Pakenham-Walsh is coordinator of HIFA (Healthcare Information For All), a global health community that brings all stakeholders together around the shared goal of universal access to reliable healthcare information. HIFA has 20,000 members in 180 countries, interacting in four languages and representing all parts of the global evidence ecosystem. HIFA is administered by Global Healthcare Information Network, a UK-based nonprofit in official relations with the World Health Organization. Email: neil@hifa.org