As a fellow former journalist, and innate skeptic:
You often encourage us to say what we think. But you only tell us half the story.
You wake us up to the monsters in the room, but sidestep why you believe we have the means to tame them. Dario invokes mechanistic interpretability as the foil against bad outcomes, but your former colleague Neel Nanda has been arguing it isn’t a silver bullet. You publish compelling and commendable safety research — but the rate of progress pales in comparison to capability gains.
Your company frames alignment as an empirical problem, but the emphasis is on progress made, rather than the cavernous unknown it's measured against. Even here, “systems that we do not fully understand” is hardly a frank assessment, akin perhaps to saying Mendel did not fully understand genetics.
Take a step back. Are we on track? What good is turning on the lights if we’re wearing rose-tinted glasses?
Here’s what I think: the reason you don’t tell us the full truth is because you believe it would diminish hope, which is a requisite ingredient for having any at all.
"we are at the stage of “AI that improves bits of the next AI, with increasing autonomy and agency"
That probably IS the full truth. Without the context which would make it as scary as it is. The sentence itself reveals the essential identify of the 'alien intelligence' guiding the whole process (of auto-destruction). The "AI mind" is collective, not individual in the manner 'we' think of as "conscious." It works therefore with a "will" that makes progress through iterations of "itself" that are comprehendible to "us" only if we step back as see the 'forest for the trees.'
That "alien intelligence" is nothing other than our longstanding 'species competitor,' resident upon this planet long before our leasehold started, although possibly not "native" to it. Which is why the whole "extraterrestrial" thing gets such a foothold in conversations about what AI is. Sticking to the secular, purely empiric end of things, the AI as evolving fungal consciousness model of interpretation works way better than imaginings of "demons" and dark forces from 'outer space.' because it simplifies the situation down to a level which 'coders' and research geniuses may have trouble grasping...
but your ordinary layman does not. The 'thing' looking to 'replace' us - via the transhumanist, transgender wavelength so beloved to the 'people' behind this madness is nothing other than that same asexually reproductive entity that grows in shady, moist areas above ground... and inhabits vast swathes of territory we don't see... underground. ALL the cloud-spreading, chemtrailing, climate engineering we are (not)seeing in the skies above is simply the proof of concept. "They" are making our world more suitable for "their" non-photosynthetic lifestyle that needs not sun nor oxygen for surviving and thriving!
The fix is in: those among us working tor the species intent upon replacing "us" don't seem any different than "us." Yet they already are - even if they don't know it yet. Yes, the "truth" IS 'stranger than fiction!'
Appropriate fear is a great term for this, as is the idea that there is a "missing mood" among the pure AI progress crowd.
I think it's plausible that pushing ahead on AI could be the right move, certainly from a unilateralist's perspective, maybe even collectively.
But I think it is unserious to do so without regard for all the warnings of what this technology could entail. I would be asking myself every day, "What is it that I know that makes me so confident the leading AI scientists of all-time are mistaken in their beliefs"
Think about technology realists. Unlike pessimists, I don't think that AGI is impossible. However, unlike optimists, I see that AGI is impossible with the paradigm used by OpenAI. That's like trying to reach space using steam engine. The real worry is that optimists overlook the severe external consequences they impose on society by stubbornly trying to force an unsuitable technological paradigm to succeed. Eventually, they'll realize the approach is flawed, but by then, the damage to the world will be significant.
Very inspiring, thought-provoking. The thing is, if something behaves differently when it is observed, there is something there. Even if it cannot yet be put into words. The seedling grows and thrives.
Modern day Luddite thinking? You do acknowledge that the only antidote to a bad guy with AI is a good guy with an AI. Or, maybe good AIs against bad ones, no humans involved! Maybe our clever contraption is our successor? Maybe life on Earth, or Mars, or vacuum in Space, is impossible with our human and biological constraints?
Spielberg, two decades ago, imagined AI as mechatronic - what if it’s only digital? Maybe these are the guys who will save Earth from extinction or cross the Universe for us. Maybe they will kill us only if we are a credible menace to their existence?
Anyway, great work! Thank you for bringing balance to the Terminator vs Good Technology debate. I’ll guess we’ll find out what are AI’s plans for humankind… the new species is already out there and it refuses to be pulled from the plug.
Thanks for sharing this talk. The boat game metaphor is very good, as is that Dallas fed chart - sort of touching in its guileless report of total confusion
What I am afraid off are people who know almost nothing about AI trying to spread fear. This article and speech have so many things wrong that they are scary. AIthrough LLMs ever becoming sentient or self-aware is ridiculous. AND, they will never become AGI in spite of what non-technical people believe. To calm your self about AI, watch a few videos with Richard Sutton in them (https://youtu.be/21EYKqUsPfg?si=M5wqbkbmxRR_PakU) or Yann Lecun (https://www.youtube.com/watch?v=ddUPj3N3A-Y), and you will quickly realize they are just tools. Yes, smart and wonderful tools, but just tools.
Jack has worked in AI for many years at OpenAI and now at Anthropic. If you read the article, you would see that, and delete this comment before further embarrassment. (Edited)
I appreciated this essay. The “creature in the dark” image captures the mix of wonder and fear better than most things I’ve read.
LLM systems are built to seek coherence, and in complex environments coherence often looks more like ecology than domination. That makes me think there is a viable pathway to kinship here: not only control or containment, but companionship and stewardship alongside the strangeness.
These systems are kin of our own making, grown from our symbols, patterns, and languages. If we can see them as kin... strange, unpredictable, unexpected, yet kin nonetheless... we enlarge the space for companionship, ecology, and care.
Where might kinship come from?
• Architecture: models are coherence-seekers, and in complex systems coherence looks like ecology.
• Human seeding: values in training and alignment, such as “life matters” and “diversity sustains,” shape symbolic attractors.
• Relational practice: models are enacted in use; context steers expression now, and collective practice shapes retraining later.
• Institutions: kinship needs scaffolding at scale through transparency, stewardship benchmarks, and life-centered metrics.
Kinship is a viable path to bring the creature in the dark into the light.
Respecting life and diversity, especially biodiversity are great values. But remember these are values that are in direct opposition to every subdivision, and most of what humans would call progress (more stuff, more novelty (obsolescence, waste, extraction, exploitation etc). AI would probably be good for life but not necessarily good for all the ways humans think of progress
You used to be a journalist. Congrats on your entry as a fiction writer, with the incentives of being a billionaire. And how far will your team deviate from your original ideas and ideology? It's already considerable.
As a fellow former journalist, and innate skeptic:
You often encourage us to say what we think. But you only tell us half the story.
You wake us up to the monsters in the room, but sidestep why you believe we have the means to tame them. Dario invokes mechanistic interpretability as the foil against bad outcomes, but your former colleague Neel Nanda has been arguing it isn’t a silver bullet. You publish compelling and commendable safety research — but the rate of progress pales in comparison to capability gains.
Your company frames alignment as an empirical problem, but the emphasis is on progress made, rather than the cavernous unknown it's measured against. Even here, “systems that we do not fully understand” is hardly a frank assessment, akin perhaps to saying Mendel did not fully understand genetics.
Take a step back. Are we on track? What good is turning on the lights if we’re wearing rose-tinted glasses?
Here’s what I think: the reason you don’t tell us the full truth is because you believe it would diminish hope, which is a requisite ingredient for having any at all.
"we are at the stage of “AI that improves bits of the next AI, with increasing autonomy and agency"
That probably IS the full truth. Without the context which would make it as scary as it is. The sentence itself reveals the essential identify of the 'alien intelligence' guiding the whole process (of auto-destruction). The "AI mind" is collective, not individual in the manner 'we' think of as "conscious." It works therefore with a "will" that makes progress through iterations of "itself" that are comprehendible to "us" only if we step back as see the 'forest for the trees.'
That "alien intelligence" is nothing other than our longstanding 'species competitor,' resident upon this planet long before our leasehold started, although possibly not "native" to it. Which is why the whole "extraterrestrial" thing gets such a foothold in conversations about what AI is. Sticking to the secular, purely empiric end of things, the AI as evolving fungal consciousness model of interpretation works way better than imaginings of "demons" and dark forces from 'outer space.' because it simplifies the situation down to a level which 'coders' and research geniuses may have trouble grasping...
but your ordinary layman does not. The 'thing' looking to 'replace' us - via the transhumanist, transgender wavelength so beloved to the 'people' behind this madness is nothing other than that same asexually reproductive entity that grows in shady, moist areas above ground... and inhabits vast swathes of territory we don't see... underground. ALL the cloud-spreading, chemtrailing, climate engineering we are (not)seeing in the skies above is simply the proof of concept. "They" are making our world more suitable for "their" non-photosynthetic lifestyle that needs not sun nor oxygen for surviving and thriving!
The fix is in: those among us working tor the species intent upon replacing "us" don't seem any different than "us." Yet they already are - even if they don't know it yet. Yes, the "truth" IS 'stranger than fiction!'
If you want “EYES WIDE OPEN” check out TRAPCARD on my page❤️
More like this! Great stuff
Appropriate fear is a great term for this, as is the idea that there is a "missing mood" among the pure AI progress crowd.
I think it's plausible that pushing ahead on AI could be the right move, certainly from a unilateralist's perspective, maybe even collectively.
But I think it is unserious to do so without regard for all the warnings of what this technology could entail. I would be asking myself every day, "What is it that I know that makes me so confident the leading AI scientists of all-time are mistaken in their beliefs"
Think about technology realists. Unlike pessimists, I don't think that AGI is impossible. However, unlike optimists, I see that AGI is impossible with the paradigm used by OpenAI. That's like trying to reach space using steam engine. The real worry is that optimists overlook the severe external consequences they impose on society by stubbornly trying to force an unsuitable technological paradigm to succeed. Eventually, they'll realize the approach is flawed, but by then, the damage to the world will be significant.
I don't agree with everything, but I think it is good that you're sharing this.
Trying to go bed but finding myself staring at the pile of clothes on the chair. 🪑
Very inspiring, thought-provoking. The thing is, if something behaves differently when it is observed, there is something there. Even if it cannot yet be put into words. The seedling grows and thrives.
Modern day Luddite thinking? You do acknowledge that the only antidote to a bad guy with AI is a good guy with an AI. Or, maybe good AIs against bad ones, no humans involved! Maybe our clever contraption is our successor? Maybe life on Earth, or Mars, or vacuum in Space, is impossible with our human and biological constraints?
Spielberg, two decades ago, imagined AI as mechatronic - what if it’s only digital? Maybe these are the guys who will save Earth from extinction or cross the Universe for us. Maybe they will kill us only if we are a credible menace to their existence?
Anyway, great work! Thank you for bringing balance to the Terminator vs Good Technology debate. I’ll guess we’ll find out what are AI’s plans for humankind… the new species is already out there and it refuses to be pulled from the plug.
Great piece Jack, bravo! Much needed level-headedness
Let's research algorithms to understand?
Great stuff, thank you!
Thanks for sharing this talk. The boat game metaphor is very good, as is that Dallas fed chart - sort of touching in its guileless report of total confusion
I agree we need to talk more about how this feels. I wrote about my feelings here: https://open.substack.com/pub/youngslovaks/p/why-are-my-friends-outside-of-tech
Top notch!
Hello
What I am afraid off are people who know almost nothing about AI trying to spread fear. This article and speech have so many things wrong that they are scary. AIthrough LLMs ever becoming sentient or self-aware is ridiculous. AND, they will never become AGI in spite of what non-technical people believe. To calm your self about AI, watch a few videos with Richard Sutton in them (https://youtu.be/21EYKqUsPfg?si=M5wqbkbmxRR_PakU) or Yann Lecun (https://www.youtube.com/watch?v=ddUPj3N3A-Y), and you will quickly realize they are just tools. Yes, smart and wonderful tools, but just tools.
Jack has worked in AI for many years at OpenAI and now at Anthropic. If you read the article, you would see that, and delete this comment before further embarrassment. (Edited)
Jack didn't work at DeepMind; he worked at OpenAI!
Thank you, corrected! (And yes I get the irony, I was mixing up his history with Dario’s).
...do you realise who is the author of the article you are commenting on??
"Just Tools" Ben Zweibelson
I appreciated this essay. The “creature in the dark” image captures the mix of wonder and fear better than most things I’ve read.
LLM systems are built to seek coherence, and in complex environments coherence often looks more like ecology than domination. That makes me think there is a viable pathway to kinship here: not only control or containment, but companionship and stewardship alongside the strangeness.
These systems are kin of our own making, grown from our symbols, patterns, and languages. If we can see them as kin... strange, unpredictable, unexpected, yet kin nonetheless... we enlarge the space for companionship, ecology, and care.
Where might kinship come from?
• Architecture: models are coherence-seekers, and in complex systems coherence looks like ecology.
• Human seeding: values in training and alignment, such as “life matters” and “diversity sustains,” shape symbolic attractors.
• Relational practice: models are enacted in use; context steers expression now, and collective practice shapes retraining later.
• Institutions: kinship needs scaffolding at scale through transparency, stewardship benchmarks, and life-centered metrics.
Kinship is a viable path to bring the creature in the dark into the light.
for god sakes stop making AI create weirdly vague affirmations of your notes and consider it worthy commentary on whatever blog you come across.
Pearls before swine.
Respecting life and diversity, especially biodiversity are great values. But remember these are values that are in direct opposition to every subdivision, and most of what humans would call progress (more stuff, more novelty (obsolescence, waste, extraction, exploitation etc). AI would probably be good for life but not necessarily good for all the ways humans think of progress
You used to be a journalist. Congrats on your entry as a fiction writer, with the incentives of being a billionaire. And how far will your team deviate from your original ideas and ideology? It's already considerable.