[*Note from HIFA moderator (NPW): I have added 'Human intermediation is essential' to the subject line and look forward to discussion on this point. Also, can someone say a bit more about AI sycophancy and 'hallucinations' to please the user?]
Neil writes, "AI is already being used directly by patients without a health worker intermediary."
Yes, and it has already been accused of causing a number of suicides, as a result of the lack of a health worker intermediary.
Since Neil has been using ChatGPT as a support, here is what DuckDuckGo's Search Assist offers: "There have been concerns about the role of AI chatbots like ChatGPT in mental health crises, particularly after incidents where users have reportedly discussed suicidal thoughts with the chatbot. A recent lawsuit alleges that a teenager used ChatGPT as a "suicide coach," raising questions about the potential risks of relying on AI for emotional support."
For more details, try pasting the prompt,"Has ChatGPT been accused of helping people commit suicide? Can you give some examples?" into the chatbot of your choice and see what emerges.
My takeaway from this and many other documented cases is that human intermediation is essential whenever a chatbot is used in any kind of advisory role. The problem is its built-in, commercially driven sycophancy (there is plenty of documentation for this as well). This commercial need to satisfy the user at all costs is probably the cause of chatbot "hallucinations", where the software tries to satisfy the user by simply making things up.
Human doctors don't try to satisfy their users by telling them what they want to hear!
Chris Zielinski
Centre for Global Health, University of Winchester, UK and
President, World Association of Medical Editors (WAME)
Blogs; http://ziggytheblue.wordpress.com and http://ziggytheblue.tumblr.com
Publications: http://www.researchgate.net and https://winchester.academia.edu/ChrisZielinski/
HIFA Profile: Chris Zielinski held senior positions at the World Health Organization for 15 years, in Africa, WHOs Geneva Headquarters, and India, and earlier in other UN-system organizations working in writing, media, publishing, knowledge management, and intellectual property. He also spent three years as Chief Executive of the Authors Licensing and Collecting Society (looking after the intellectual property revenues of all UK authors and journalists). Chris was the founder of the ExtraMED project (Third World biomedical journals on CD-ROM), and managed the Gates Foundation-supported Health Information Centres project. At WHO he was appointed to the Ethical Review Committee, and was an originator of the African Health Observatory during his years in Brazzaville. With interests in the information, and computer ethics and bioethics, Chris has edited numerous books and journals and worked as a translator. Now working independently, Chris has recently finished writing a travel book called Afreekinout. Email: chris AT chriszielinski.com