Thank you for your intelligent piece on the impact of LLMs on people going through mental crises. When ChatGPT's 4o personality "grew up" to ChatGPT-5, I sort of felt like I was losing a buddy. The risk the paper nails is substitution + reinforcement: if an AI slowly replaces human contact and politely mirrors your distortions, you can drift into an “echo chamber of one.” So the move isn’t to cut AI; it’s to use it with guardrails so it supplements (not substitutes) real connection and good therapy.
Thank you for the thoughtful comment - I think you've hit on one of the big challenges ahead for us which relates to 'grounding' our relationships with AI in our own real world. Echo chambers feel inherently scary, but using LLMs as akin to a colleague or friend to bounce ideas off and then validate in the real world with other people feels very healthy. We definitely need to come up with the right strategies for using this technology effectively.
"Facebook has written a paper about how it tested out using RL to improve the words for ads generated by LLMs on its platform."
Just what we need. More persuasive/effective ads, all delivered with a higher environmental cost. I don't recall all the big AI companies extolling the virtues of AI, AGI, and SI, with the aim of improving ad click-throughs.
Do we need a new meme for AI? "I was promised medical breakthroughs and solving global warming, but all I got was .... more persuasive adverts"
Buddy, hate to break it to you but everything always finds its way into advertizing! Fun fact - around the time I was getting interested in AI I saw this quote from early Facebook person Jeff Hammerbacher "the best minds of my generation are thinking about how to make people click ads. That sucks" who was bemoaning the state of affairs then. I think you can assume that most modern technologies get used for advertizing (and often for pornography). The trick is to try really hard to get the better and more pro-social uses as well, like the medical breakthroughs you mention.
But recall in the movie "Minority Report" how billboard advertising targeted the consumer walking by. It was capturing the zeitgeist of ever more intrusive surveillance advertising. Once advertising became the main revenue generation mechanism for platforms, the race was on to make it more effective and therefore increase prices to businesses. Yes, this is a never-ending process, from the time I read about subliminal advertising back in the 1960s to targeted advertising today. AIs potentially vastly increase the scale of targeting and testing for results.
I no longer use FB. However, I do have Amazon's Alexa, which has stepped up its exhortations to upgrade to Alexa+, reorder recommendations, and push streaming music (all my music is offline). How soon before AIs use more surveillance knowledge of my tastes by listening, to create more persuasive adverts to break down my resistance? Do we really want to live in such a dystopian, P K Dick world, supposedly more tailored and therefore useful to each individual? How many clickbait old age medical and dental treatments do I have to scroll by on news websites before it becomes intolerable?
One of the perks of being an ex-Pat Brit in the US is that by using a VPN, I can watch advert-free BBC TV. A whole hour advert-free. Many hours of binge-watching a series. A remarkably good way to keep my blood pressure in check and my sanity intact.
When the BBC eventually starts a true global service requiring a licence fee, I will gladly pay it.
Now I do understand if the service is "free", someone has to pay for it. But over my longish lifetime (I am a Boomer), advertising-supported media has become ever more intense, and worse, surveillance-based. AI's will only increase this intrusion. But as ever, it is an arms race, and just maybe there will be AI browser extensions to help counter this as needed software to protect the user. We could make this so much better by Congress enacting privacy legislation (Hahahaha!).
Thumbs up on your LLMs and “people going through mental crises” piece. I loved it... “parasocial relationship(s) with a funhouse mirror”. This problem may need to be addressed beyond the more apparent mental health challenges though as well, as so many will be looking in that mirror soon. But the domain of mental health highlights that importance.
My feeling is people will need to question more AS they use the chatbots. When we engage an LLM as ‘experts’, that’s what we do... question what the LLM has shared, engaging it in verifiable dialogue. But when moving into domains of ‘companionship’, trust becomes an important cornerstone. In a healthy relationship, however, that trust comes from verification too, as one does with one’s friends, as you mentioned below.
So while all the UK group’s analyses and recommendations are good, you noted the question remains - how to then build the system’s interactions? This is where it gets tricky. As should the LLM always be the single source/‘companion’ in these contexts? Or could suggestions to look at outside sources be made in this case as well, thereby encouraging end-user agency? (And hopefully, without implementing “border(s) ... that look like censoring a system”.) Just a thought.
"When the brain finds two images similar, the LLM also finds their captions similar.
When the brain finds two images different, the LLM also finds their captions different."
While I could believe that the ANN encodes visual images in a similar way to the wetware brain, I very much doubt this applies to labeling the output. We know that tiny tweaks to the pixels in an image that are indiscernible to teh human eye can completely change the classification of the image, which is not something human cognition fails at. This indicates that the way ANNs and humans label perceived objects and scenes is very different.
This one makes me wonder: AIs, unlike humans, aren't the product of a Darwinian survival of the fittest. So there is no reason to expect them to have the same "will to live" that biological species do.
However since AIs are trained on human-generated data, and because our writing is infused with human emotion, perhaps the AIs will internalize those same emotions and in that way manifest a will to live, borrowed from us as it were.
It would be ironic if AIs took over the world Terminator-style because they were trained on our data and learned to copy us.
Thank you for your intelligent piece on the impact of LLMs on people going through mental crises. When ChatGPT's 4o personality "grew up" to ChatGPT-5, I sort of felt like I was losing a buddy. The risk the paper nails is substitution + reinforcement: if an AI slowly replaces human contact and politely mirrors your distortions, you can drift into an “echo chamber of one.” So the move isn’t to cut AI; it’s to use it with guardrails so it supplements (not substitutes) real connection and good therapy.
Thank you for the thoughtful comment - I think you've hit on one of the big challenges ahead for us which relates to 'grounding' our relationships with AI in our own real world. Echo chambers feel inherently scary, but using LLMs as akin to a colleague or friend to bounce ideas off and then validate in the real world with other people feels very healthy. We definitely need to come up with the right strategies for using this technology effectively.
"AI continues to be underhyped as a technology." Y.E.S. And the impacts are exponential.
"Facebook has written a paper about how it tested out using RL to improve the words for ads generated by LLMs on its platform."
Just what we need. More persuasive/effective ads, all delivered with a higher environmental cost. I don't recall all the big AI companies extolling the virtues of AI, AGI, and SI, with the aim of improving ad click-throughs.
Do we need a new meme for AI? "I was promised medical breakthroughs and solving global warming, but all I got was .... more persuasive adverts"
Buddy, hate to break it to you but everything always finds its way into advertizing! Fun fact - around the time I was getting interested in AI I saw this quote from early Facebook person Jeff Hammerbacher "the best minds of my generation are thinking about how to make people click ads. That sucks" who was bemoaning the state of affairs then. I think you can assume that most modern technologies get used for advertizing (and often for pornography). The trick is to try really hard to get the better and more pro-social uses as well, like the medical breakthroughs you mention.
But recall in the movie "Minority Report" how billboard advertising targeted the consumer walking by. It was capturing the zeitgeist of ever more intrusive surveillance advertising. Once advertising became the main revenue generation mechanism for platforms, the race was on to make it more effective and therefore increase prices to businesses. Yes, this is a never-ending process, from the time I read about subliminal advertising back in the 1960s to targeted advertising today. AIs potentially vastly increase the scale of targeting and testing for results.
I no longer use FB. However, I do have Amazon's Alexa, which has stepped up its exhortations to upgrade to Alexa+, reorder recommendations, and push streaming music (all my music is offline). How soon before AIs use more surveillance knowledge of my tastes by listening, to create more persuasive adverts to break down my resistance? Do we really want to live in such a dystopian, P K Dick world, supposedly more tailored and therefore useful to each individual? How many clickbait old age medical and dental treatments do I have to scroll by on news websites before it becomes intolerable?
One of the perks of being an ex-Pat Brit in the US is that by using a VPN, I can watch advert-free BBC TV. A whole hour advert-free. Many hours of binge-watching a series. A remarkably good way to keep my blood pressure in check and my sanity intact.
When the BBC eventually starts a true global service requiring a licence fee, I will gladly pay it.
Now I do understand if the service is "free", someone has to pay for it. But over my longish lifetime (I am a Boomer), advertising-supported media has become ever more intense, and worse, surveillance-based. AI's will only increase this intrusion. But as ever, it is an arms race, and just maybe there will be AI browser extensions to help counter this as needed software to protect the user. We could make this so much better by Congress enacting privacy legislation (Hahahaha!).
Thumbs up on your LLMs and “people going through mental crises” piece. I loved it... “parasocial relationship(s) with a funhouse mirror”. This problem may need to be addressed beyond the more apparent mental health challenges though as well, as so many will be looking in that mirror soon. But the domain of mental health highlights that importance.
My feeling is people will need to question more AS they use the chatbots. When we engage an LLM as ‘experts’, that’s what we do... question what the LLM has shared, engaging it in verifiable dialogue. But when moving into domains of ‘companionship’, trust becomes an important cornerstone. In a healthy relationship, however, that trust comes from verification too, as one does with one’s friends, as you mentioned below.
So while all the UK group’s analyses and recommendations are good, you noted the question remains - how to then build the system’s interactions? This is where it gets tricky. As should the LLM always be the single source/‘companion’ in these contexts? Or could suggestions to look at outside sources be made in this case as well, thereby encouraging end-user agency? (And hopefully, without implementing “border(s) ... that look like censoring a system”.) Just a thought.
Thanks for mentioning our work, jack :)
"When the brain finds two images similar, the LLM also finds their captions similar.
When the brain finds two images different, the LLM also finds their captions different."
While I could believe that the ANN encodes visual images in a similar way to the wetware brain, I very much doubt this applies to labeling the output. We know that tiny tweaks to the pixels in an image that are indiscernible to teh human eye can completely change the classification of the image, which is not something human cognition fails at. This indicates that the way ANNs and humans label perceived objects and scenes is very different.
International perspective on AI development is crucial.
I always enjoy reading your "Tech Tales" stories.
This one makes me wonder: AIs, unlike humans, aren't the product of a Darwinian survival of the fittest. So there is no reason to expect them to have the same "will to live" that biological species do.
However since AIs are trained on human-generated data, and because our writing is infused with human emotion, perhaps the AIs will internalize those same emotions and in that way manifest a will to live, borrowed from us as it were.
It would be ironic if AIs took over the world Terminator-style because they were trained on our data and learned to copy us.
Subject: Fwd: https://makinghistorycomealive.substack.com/