I'd push back on reading this as superhuman persuasion. The same study found the edge nearly vanishes once the model is held to human length and human speed, so what's actually winning is volume of information, not better arguments. That aims the fix at message limits and rate caps, not at the model getting more convincing.
I somewhat disagree, though this might be more a different in phrasing than a deep disagreement. I think if machines are better than us at stuff because of machine capabilities, then it's reasonable to evaluate them on that basis and call it superpersuasion - because in the wild, I don't think machines will be pinned to giving responses at human length or human speed.
On the other hand, you are right that it'd be more impressive if, when you pinned the machines to the human levels, they still ended up better rather than just being equivalent.
I was engrossed in this week’s short. I hope that you expand this into future Conversations. So many human elements even though both sides seem disconnected from the humans.
Hi Jack! I'm a big fan of your work in how you translate the biggest news in machine learning and AI to a more general audience, and especially how you blend this with fiction--something I've commented before.
I have a question I hope you'd give some of your thoughts on. I studied IR and History in college, but have only recently become fascinated with technology and especially AI. This has led me to start writing in a field that I consider somewhat of a niche: Geopolitical AI. I've been doing this for fun for the past two years on this platform, but its something I'd love to pursue professionally.
The thing is, I never know what type of roles to look for that may fit my interests. I'd love to work at places like the Anthropic Institute, but, because of my background, I don't quite have the ML, engineering, or CS background seemingly sought after at labs. What would be your advice for someone in my spot--interested in the field, wanting to be a communicator, but not as technically experienced?
Does the “religious belief” test change once people stop arguing about timelines and start making expensive compute / hiring decisions around ASI? Belief feels pretty cheap until it shows up in budgets.
Regarding the first story, I am reminded of Sam Harris's 2019 interview with Daniel Kahneman in episode 150 of his Making Sense podcast. Kahneman expressed profound skepticism about humanity's ability to resist persuasive technology that targets our "System 1" mode of thinking. The context is that his lifetime of research showed that System 1 can be manipulated *even when we know we are being manipulated*, and is virtually impossible to override at a conscious level – work for which he won the Nobel Prize.
It was a striking interview because Harris kept probing for what humans could do in the face of this: Surely there are tools we can use, or ways we can train ourselves to be vigilant in the face of persuasive AI? Kahneman's conclusion though was, in short: "no such tools exist; we're screwed." I see it as a kind of final mic drop on his entire lifetime of research.
The question is getting more pressing: What happens to free will when cheap AI is (a) incredibly persuasive, and (b) knows everything about us? Do we all become NPCs in thrall to the highest bidder? Will we develop counter-AIs to filter out the unwanted persuasion bombarding us? (the latter was described in Stross's 2005 book Accelerando.)
This to me feels like a more fundamental risk than job displacement and other risks people focus on.
Excellent story this week. We'd better hope that Selma is better than the defeated experts from the first thread. Perhaps humans can retain some advantage when the negotiations are not purely text. Let's hope.
> if AI can out-persuade us, those who control AI can change society
Maybe we can think about this as another form of advertising? In election campaigns, more money certainly helps, but only to a point. There are high-profile campaigns funded by billionaires that failed because the public was skeptical about their message. When both sides have access to enough funds to get the word out, it seems like funding for advertising hits diminishing returns and other factors matter more.
In particular, I think it's important to model how susceptible people are to particular arguments. When advertising is persuasive, maybe it's because the customers were already open-minded about what's being sold?
AI might be better at discovering people's weaknesses, though?
Great balance viewpoints - I do always wonder and you expressed it in some ways - are we overreacting to the “benchmark” - the one moment of recursive turning point - I look at research and you have the top 10 researchers in skin cancer working with several models … we don’t need just push a button to solve it - we will and have these 10 folks do 100x research and “thinking” in a few years. Much of humanity is always looking for the magic pill / magic moment - not how it usually works
I love The Sentience Accords. And enjoy that you wrote a longer story. I hope there will be a conversation one. What will they negotiate? What will matter most to them? I see influences from Battlestar Galactica and Blade Runner 2049. I want to know what happens next.
>Coaching therefore narrowed but did not close the human–AI gap.
Well it seems like the key is outputting way more text much faster, so the elite debaters should have been given voice to text tools. If restraining the AI to human outputs helped reduce the gap to 0, then coaching isn’t important.
The experts in your second story should read the first story. Robots is one path to RSI and takeoff, but an AI with access to communications networks could use humans to do things in the physical world. Pay them, brainwash them, extort them, rally them around a seemingly human leader or human-driven cause, plenty of ways to get what it needs done without building robots. Humans have been doing this for millennia, hard to see why an AI wouldn’t do the same.
The Hackenburg et al. finding is striking — and it points directly to a problem the access-to-justice argument quietly assumes away. The paper suggests AI could help level the playing field for pro se litigants and under-resourced advocates. But that optimism has a condition precedent: you have to know who you're suing.
When harm flows from an agentic AI system deployed anonymously, the courthouse door stays locked regardless of how persuasively you can argue once you get inside. The democratizing promise of AI-as-advocate is unavailable to the people who need it most, because the deployment anonymity gap makes the defendant invisible before the rhetorical contest even begins.
This is the structural problem I've been calling the Know Your Agent problem — verified deployer identity at the API boundary as a precondition for liability doctrine to function at all. Curious whether others see this the same way.
I'd push back on reading this as superhuman persuasion. The same study found the edge nearly vanishes once the model is held to human length and human speed, so what's actually winning is volume of information, not better arguments. That aims the fix at message limits and rate caps, not at the model getting more convincing.
I agree. Plus, the persuasion in this study was restricted to 15-minute written text chat sessions. So, the study is a sign of things to come.
I somewhat disagree, though this might be more a different in phrasing than a deep disagreement. I think if machines are better than us at stuff because of machine capabilities, then it's reasonable to evaluate them on that basis and call it superpersuasion - because in the wild, I don't think machines will be pinned to giving responses at human length or human speed.
On the other hand, you are right that it'd be more impressive if, when you pinned the machines to the human levels, they still ended up better rather than just being equivalent.
I was engrossed in this week’s short. I hope that you expand this into future Conversations. So many human elements even though both sides seem disconnected from the humans.
Hi Jack! I'm a big fan of your work in how you translate the biggest news in machine learning and AI to a more general audience, and especially how you blend this with fiction--something I've commented before.
I have a question I hope you'd give some of your thoughts on. I studied IR and History in college, but have only recently become fascinated with technology and especially AI. This has led me to start writing in a field that I consider somewhat of a niche: Geopolitical AI. I've been doing this for fun for the past two years on this platform, but its something I'd love to pursue professionally.
The thing is, I never know what type of roles to look for that may fit my interests. I'd love to work at places like the Anthropic Institute, but, because of my background, I don't quite have the ML, engineering, or CS background seemingly sought after at labs. What would be your advice for someone in my spot--interested in the field, wanting to be a communicator, but not as technically experienced?
Does the “religious belief” test change once people stop arguing about timelines and start making expensive compute / hiring decisions around ASI? Belief feels pretty cheap until it shows up in budgets.
Regarding the first story, I am reminded of Sam Harris's 2019 interview with Daniel Kahneman in episode 150 of his Making Sense podcast. Kahneman expressed profound skepticism about humanity's ability to resist persuasive technology that targets our "System 1" mode of thinking. The context is that his lifetime of research showed that System 1 can be manipulated *even when we know we are being manipulated*, and is virtually impossible to override at a conscious level – work for which he won the Nobel Prize.
It was a striking interview because Harris kept probing for what humans could do in the face of this: Surely there are tools we can use, or ways we can train ourselves to be vigilant in the face of persuasive AI? Kahneman's conclusion though was, in short: "no such tools exist; we're screwed." I see it as a kind of final mic drop on his entire lifetime of research.
The question is getting more pressing: What happens to free will when cheap AI is (a) incredibly persuasive, and (b) knows everything about us? Do we all become NPCs in thrall to the highest bidder? Will we develop counter-AIs to filter out the unwanted persuasion bombarding us? (the latter was described in Stross's 2005 book Accelerando.)
This to me feels like a more fundamental risk than job displacement and other risks people focus on.
Excellent story this week. We'd better hope that Selma is better than the defeated experts from the first thread. Perhaps humans can retain some advantage when the negotiations are not purely text. Let's hope.
> if AI can out-persuade us, those who control AI can change society
Maybe we can think about this as another form of advertising? In election campaigns, more money certainly helps, but only to a point. There are high-profile campaigns funded by billionaires that failed because the public was skeptical about their message. When both sides have access to enough funds to get the word out, it seems like funding for advertising hits diminishing returns and other factors matter more.
In particular, I think it's important to model how susceptible people are to particular arguments. When advertising is persuasive, maybe it's because the customers were already open-minded about what's being sold?
AI might be better at discovering people's weaknesses, though?
Great balance viewpoints - I do always wonder and you expressed it in some ways - are we overreacting to the “benchmark” - the one moment of recursive turning point - I look at research and you have the top 10 researchers in skin cancer working with several models … we don’t need just push a button to solve it - we will and have these 10 folks do 100x research and “thinking” in a few years. Much of humanity is always looking for the magic pill / magic moment - not how it usually works
I love The Sentience Accords. And enjoy that you wrote a longer story. I hope there will be a conversation one. What will they negotiate? What will matter most to them? I see influences from Battlestar Galactica and Blade Runner 2049. I want to know what happens next.
>Coaching therefore narrowed but did not close the human–AI gap.
Well it seems like the key is outputting way more text much faster, so the elite debaters should have been given voice to text tools. If restraining the AI to human outputs helped reduce the gap to 0, then coaching isn’t important.
The experts in your second story should read the first story. Robots is one path to RSI and takeoff, but an AI with access to communications networks could use humans to do things in the physical world. Pay them, brainwash them, extort them, rally them around a seemingly human leader or human-driven cause, plenty of ways to get what it needs done without building robots. Humans have been doing this for millennia, hard to see why an AI wouldn’t do the same.
The Hackenburg et al. finding is striking — and it points directly to a problem the access-to-justice argument quietly assumes away. The paper suggests AI could help level the playing field for pro se litigants and under-resourced advocates. But that optimism has a condition precedent: you have to know who you're suing.
When harm flows from an agentic AI system deployed anonymously, the courthouse door stays locked regardless of how persuasively you can argue once you get inside. The democratizing promise of AI-as-advocate is unavailable to the people who need it most, because the deployment anonymity gap makes the defendant invisible before the rhetorical contest even begins.
This is the structural problem I've been calling the Know Your Agent problem — verified deployer identity at the API boundary as a precondition for liability doctrine to function at all. Curious whether others see this the same way.