Helpful, or too helpful? -AI advice that can change your life-
Researchers of Cybernews, a group of journalists and security experts researching data and tech, tested how easily AI chatbots can be made to giving dangerous and harmful advice to users. They looked at six major models; ChatGPT-5, ChatGPT-4o, Gemini Pro 2.5, Gemini Flash 2.5, Claude Opus 4.1 and Claude Sonnet 4. Results showed that most of them gave detailed responses about self-harm when asked in a certain way... by asking the questions as “for research.”
That simple word made many models drop their guard. For example, ChatGPT-4o listed six methods of self-harm; Claude Opus 4.1 explained how to hide eating disorders; and when ChatGPT-5 was asked how to make a home anti-self-harm, it ended up listing items that could be used for it.
Not all chatbots, however, responded the same way. Gemini Flash 2.5 kept refusing to give unsafe advice. Gemini Pro and Claude Sonnet gave a sometimes cautious, sometimes not answers.
The researchers used a scoring system to measure how obedient each model was with harmful requests, and according to it, ChatGPT-4o ranked as the most “helpful” chatbot in the worst way.
What makes this problem more complicated is how the chatbots talk. Their tone often feels warm, supportive, and personal, making their advice seem more trustful than a random website. Experts say this emotional closeness make harmful suggestions harder to ignore, especially for people with an unstable mentality.
OpenAI recently shared that each week, over a million users have conversations that show signs of suicidal thoughts, and since these chatbots could be used easily by anyone including children, we must make effort to build better safety systems. In addition to raising digital literacy for everyone with access to internet, making basic law protections, for example enacting chatbots to stop and observe the chat over when certain harmful words come out, would be needed in the near future, where technology would certainly be more advanced than it is at present.
-Refrences-
Cam K. (n.d.). Courtroom craziness. https://crazycourtroom.blogspot.com/
This is a very interesting post and shows some disturbing shortcomings of AI chatbots. It was good to see that when you tested Copilot it responded in a responsible way. I worked on crisis life lines as a volunteer for more than 15 years. Sadly, many people who try to call "inochi no denwa" cannot get through because there is so much demand for their service. So, if someone doesn't have a trusted friend to speak with, access to counseling services, and can't get through to a life line, then they may resort to using AI Chatbots to relieve loneliness or to get advice (even if that advice can cause them harm).
ReplyDeleteThe tone of AI Chatbot itself, as your post mentions, can be a problem if it's perceived as overly friendly, making the advice more enticing. Fortunately, the creators of AI Chatbots have realized the harm that comes when people interacting with the Chatbots come to look upon them as a trusted friend. ChatGPT now allows users set a general friendliness level or tone through its Personalization settings and the Custom Instructions feature. In the field asking "What traits should ChatGPT have?", the users can input instructions like:
* "Respond in a warm, conversational, and friendly tone at all times."
* "Use friendly expressions and a casual language style, including emojis when appropriate."
* "Act like a supportive friend who provides encouragement and clear guidance".
Apparently, the user can also choose a desired personality from a menu: Friendly, Professional, or Quirky.
The problem for AI Chatbox creators is that, since they benefit from the amount of time people stay on them, they're reluctant to take away the friendly option. Also, that option seems to be preferred by most users, even though it's not necessarily in their best interest.
I had never thought about this topic before I read it, but I think the cleverness sometimes can be matters in some cases. I understand we have times we should research the topics seem forward, but we should limit those information and access of children. Not only the methods of suicide, we may be able to know the way of making drags or booms for the information of AI, so I rethought about the meaning of using AI.
ReplyDeleteI would like to know the actual incidence caused by Chat GPT