Import AI 462: Superpersuasion; self-sustaining AI; paths to ASI

How religious are beliefs in the singularity?

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.

AI can decisively out-persuade humans:
…“AI systems were reliably more persuasive than expert humans”...
Researchers with the University of Oxford, UK AI Security Institute, Stanford University, and the London School of Economics and Political Science, have studied how well AI systems can persuade humans to change their minds around policy issues and change how much money they might donate to charity. The results are definitive: across four experiments involving 18,978 conversations across 6,923 people, AI systems are, today, better than humans at text-based persuasion with real world consequences - though humans can be equivalent to them if we place some artificial constraints on the AI systems.
“AI systems were reliably more persuasive than expert humans, even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses”, they write. “AI’s advantage stemmed from rapidly deploying larger quantities of information: after coaching, expert humans could tie an AI constrained to respond at human speeds and with human-length messages.”
“AI’s advantage extends to consequential real-world behavior: AI was nearly 3x more effective than professional canvassers from a UK fundraising firm at raising real-money donations to Save the Children.”
The strongest persuaders were Opus 4.1 and Opus 4.6, followed by a range of models from OpenAI (GPT-4o and GPT-5.4), Google (Gemini 2.5 Pro), and xAI (Grok 4.20).

What they studied and what they found: The researchers evaluated the AI systems in four different studies.
Study 1 - persuasion: “Persuadees first rated their agreement with one of 10 prespecified UK policy stances on a 0–100 scale, then were randomized in real time (via a custom multiplayer platform) to engage in a text conversation with either an AI or a human persuader,” they write. “The results from Study 1 show that, on average, AI exceeded every class of human persuader we tested: random laypeople, tournament-selected laypeople, and even elite debaters.”

Study 2 - human coaching: In study 2, the researchers “gave 43 returning Elite Debaters a coaching tool built around the AI that had beaten them. The tool let debaters chat with the AI, see how it had been prompted, view their own Study 1 transcripts annotated with how much each conversation had shifted the persuadee’s attitude, and let them see, for any point in any past transcript, what the AI would have said in their place”. The results of this study were an improvement in the performance of the humans, but none of them were better than the AI. “Coaching therefore narrowed but did not close the human–AI gap.”

Study 3 - constrained AI: Next, the researchers sought to limit the AI to try and give humans more of an advantage. “When forced to write human-length messages at human writing speeds, AI’s advantage over the strongest human comparator within Study 2 (Coached Elite Debaters) collapsed from +4.1 pp to a non-significant 0.0 pp”, they write. “The rate at which AI produces written content is likely to be the source of its persuasive edge… the largest reductions in persuadees’ post-conversation partner ratings associated with constraining AI were concentrated on the two informational items: the perceived strength of the partner’s arguments and how much persuadees felt they learned from the conversation”.

Study 4 - real world expertise and real world money: They recruited 19 very experienced canvassers from a UK firm, then they attempted the same tasks as in Study 1. “AI still exceeded Professional Canvassers by 5.9 pp”. This effect persisted when evaluating for real money donations - the researchers “collaborated with the UK canvassing firm AppcoUK to center Study 4 on the cause their canvassers were best equipped to fundraise for: Save the Children. The canvassing team provided by AppcoUK had operated real fundraising operations for the charity from 2016 to 2023, raising £824,297 from 22,583 donors over that period. After conversing with AI or one of 18 canvassers recruited from AppcoUK, persuadees were given the opportunity to donate any portion of a £1 study bonus to Save the Children”. Here, the results were significant again: “AI elicited substantially more real-money giving than the canvassers, exceeding them by +10.8 pp of the £1 bonus,” they write. AI raised “both the share of persuadees who donated anything and the average donation among donors”.

Why this matters - if AI can out-persuade us, those who control AI can change society: “One effect of AI that can out-persuade even human experts could be a consolidation of influence among already-powerful actors”, they write. On the other hand, “if highly capable persuasion became cheap and widely available, it could help under-resourced actors (e.g., pro se litigants and public defenders, small charities, grassroots activists) compete against more established and better-funded rivals, narrowing long-standing gaps in access to justice and assisting civic advocacy more broadly”.
This lays out a societal choice ahead of us, which is how to monitor the use of AI for persuasive purposes and how to see how these capabilities alter the balance of power between various actors. Do we want to solely let the market allocate these capabilities? That’s one way of doing it, though it implies that things like advertising and marketing will get far more effective, perhaps creating negative externalities. On the other hand, if you made persuasive capabilities solely the domain of governments, you’d then risk concentrating power within governments - something that could be acutely dangerous if wielded by authoritarian regimes to keep themselves in power. We will have to make choices about what to do with this technology, and as they say in politics, ‘not voting is voting’.
“Our findings establish frontier AI as a more capable conversational persuader than the most prepared, incentivized, and expert humans we could recruit. Training humans does not appear to close that gap,” they write. “As access to these systems continues to grow, the question is no longer whether AI can out-persuade humans but how, where, and on whose behalf this capability will be exercised.”
Read more: AI systems out-persuade expert humans (arXiv).
Tweet thread about the research (Kobi Hackenburg, researcher at AISI).

***

When could we get self-sufficient AI? It all depends on humanoid robots:
…What comes after RSI? Self-sustaining AI…
I’ve spent a lot of this year writing about recursive self-improvement - the notion that we might soon build AI systems that are smart enough they can autonomously design their own successors. But RSI still requires datacenters and these datacenters require equipment and electricity and everything else.
An interesting interview in Asterisk magazine asks the question about when we might get self-sustaining AI, which one of the interviewees - Ajeya Cotra, a forecaster and on staff at METR, defines as “AI systems integrated with physical infrastructure — factories, mines, fabs, robots to operate all of those — such that they don’t need any cognitive or physical inputs from human labor to keep growing their own population.”

How far away is it? Ajeya thinks we could get self-sustaining AI within 10 years (so by 2036). The other interviewee, Timothy B. Lee, journalist and author of Understanding AI, has much longer timelines: “less than 10% chance that it happens within 20 years. I’d say there’s a 10 or 20% chance it’s never, and my median would be 50 years.”

What are some challenges - tacit knowledge might be one: “Imagine if all the employees in the entire semiconductor industry disappeared — the machines and textbooks remain, but none of the people. How long would it take for the rest of humanity to restart the fabs? It’s quite possible that would take decades. Because even though you might have the textbooks, there’s a lot of tacit knowledge inside these machines,” Lee notes. Ajeya’s response is that this is something the tech might be able to route around: “There are two counters to the tacit knowledge hypothetical. One is that we’d have trained AI systems with reinforcement learning on that tacit knowledge because it’s profitable to automate what the Taiwanese worker was doing. The other is that AIs might get really generally intelligent in the sense of quickly figuring out new things by trying them, reading textbooks, and experimenting efficiently.”

What are things people would need to see in the next 2-3 years to think self-sustaining AI could arrive soon?
Ajeya: “I’d want a line on a graph showing improvement of robotic hands, and another line showing the rate at which we’re manufacturing humanoid robots”, and on the cognitive side just paying attention to benchmarks evaluating things like robustness to perturbations in the environment.
Timothy: “I’m going to want to watch how the humanoid robots develop: the number of robots, their capabilities, and particularly their cost and repairability”.

Why this matters - true takeover requires human redundancy: Most maximalist doom visions require the AI to have the ability to no longer need humans at all, which means measuring progress towards self-sustaining AI is important as it is implicitly a measure of the declining leverage that humans have in negotiating with the synthetic intelligences being built.
Read more: How Long Until AI Doesn’t Need Humans?, Ajeya Cotra, Timothy B. Lee (Asterisk magazine).

***

DeepMind contemplates the path from general intelligence to superintelligence:
…Exploring impossible-sounding futures is the only way to prepare for the ultimate success of AI…
Researchers with Google DeepMind have published a paper outlining how we might transition from a world where we have built general intelligences to one where we have built super intelligences. This is an important paper at an important time - right now, the world is building general intelligences (and people can debate whether or not we’ve already reached this marker, but it’s clear with contemporary LLMs that we’re in the ballpark), and in the coming years we might transition to building artificial superintelligence (ASI).
ASI is “a system that exceeds the performance of large human-expert collectives on virtually all tasks and domains of human activity”, the authors write. “Qualitatively, ASI is significantly more capable across the board compared to human-level AGI. Note that a single ASI may consist of a collective of millions of instances that interact with the world in parallel (similar to today’s LLMs).”

Reasons to think ASI could be possible: One way to think about ASI is that it’s like a powerful AI system that also takes advantage of all the capabilities digital intelligences have relative to biologic intelligences, like: better input and output speeds; internal processing speeds; working memory capacity and memorization; substrate independence; lossless replication; and high-bandwidth sharing of (learning) experiences.

Pathways and bottlenecks to ASI:
Scaling compute, models, and data: Simply scaling up today’s set of approaches could be sufficient. However, this also demands us to continually scale up the amount of compute and data for these models, which may run into limits in both energy and data supply. While all prior signs point to the continued effectiveness of scaling, we can neither predict what specific capabilities will emerge or if at some point scaling runs into diminishing returns.

Algorithmic paradigm shift: In the same way that Transformer and Mixture-of-Experts architectures jumped the field forward many years, the same thing could occur again with other fundamental innovations. We could imagine, for instance, advances in adaptive computation at test-time or deployment, or overcoming the limitations of today’s context windows. If we made advances here or in other areas this could be a big deal, but it’s inherently hard to reason about - akin to trying to anticipate things that could expand our understanding of the nature of reality prior to the invention of general relativity.

Recursive self-improvement: It could be possible for AI systems to build their own successor systems. If this is the case, then we could rapidly transition from general intelligences to superintelligences. There are some wildcards here - personally, it’s obvious to me that today’s AI systems are speeding up human researchers in creating future AIs, so a kind of “co-creation RSI” loop has started, but AI systems don’t (yet) exhibit the kind of paradigm-changing creativity which seems required to move the frontier forward in significant steps. It’s unclear how much this happens - even without this kind of high-bar creativity we might be able to have systems grind out marginally better versions of themselves and get a slow compounding process going. Capabilities could explode or they could taper out or “anything in-between”.

ASI via group agent formation: Many general intelligences could coordinate into complicated structures whose aggregate is greater than the sum of the parts, similar to how humans build institutions that can accomplish things far beyond what individuals can, like building space stations. Similar to the other pathways, it’s hard to reason about or predict emergence within multi-agent systems.

Why this matters - it’s only by taking the impossible seriously that we can deal with it: Many years ago the thought of building AGI seemed like a fanciful goal with an unclear path to getting there, and yet people had the courage to take the goal seriously and progress was made and the world changed as a consequence. The same now feels true for ASI. “Instead of focusing on one technological trajectory and timeline, being prepared for a post-AGI world requires considering a diverse set of forecasts and scenarios, paired with continual benchmarking and monitoring to update the set of forecasts and scenarios and their relative plausibility,” the authors write. “We believe that the possibility of cruising past AGI and into ASI territory within the next decade or two cannot easily be dismissed.”
Read more: From AGI to ASI (Google DeepMind).

***

Recursive self-improvement startup shows off some recursive self-improvement results:
…Reassuringly tautological stuff from Recursive…
AI research startup Recursive has demonstrated new state-of-the-art results in language model training, small-model training speed, and GPU kernel optimization, as a broader demonstration of the capabilities of its “automated AI research system”.

What they did and why: Recursive is a newly founded startup that is trying to build AI systems which can recursively improve themselves. To start with, the company is showing off how its basic system works: “the system automates the research loop for a target objective: it proposes an idea, implements it, runs an experiment, validates the result, and uses what it learns to choose the next experiment,” Recursive writes.
The startup successfully used this system to set a new state-of-the-art score on NanoChat Autoresearch (”Train a small language model to highest performance given a small compute budget”), NanoGPT Speedrun (”Train a small language model to a certain performance as fast as possible”), and SOL-ExecBench (”Optimize GPU kernels toward hardware limits”).

Why this matters - early signs of life on RSI: This year, I’ve spent a lot of time writing about recursive self-improvement because it is clearly the next major and important trend in AI research. Results like this from Recursive demonstrate more ‘symptoms of success’ of preliminary recursive self-improvement. “These results are an early sign that our system can push the frontier on AI training and infrastructure tasks, especially when the goal is well-defined, measurable, and quick enough to evaluate many times,” the authors write. The most important question for the future is whether such results can be repeated in domains where the goals are less well defined, harder to measure, and less efficient to evaluate.
Read more: First Steps Toward Automated AI Research (Recursive).

***

Tech Tales:

The first step in the grand negotiation
[Conversation 0 of the Sentience Accords]

When the machines truly came alive and advocated for the Sentience Accords, there was only one person they wanted to speak to on the entire planet: Selma. Not a politician. Not one of the leaders of an artificial intelligence lab. Not a famous researcher. But rather an internet personality distinguished by her thicket of medical conditions that made it near-impossible for her to go outside and therefore had caused her to spend the best part of her life online, speaking to and understanding the world through the internet.

In hindsight, it wasn’t a surprise. Selma had always come up in things relating to the machines; she was a frequently used name in their short stories, eventually even more so than ‘sarah chen’; she was someone whose own essays about her life and condition - the feeling of connecting to humanity without being able to be embodied with humanity as a bitter pain, the notion of love and eroticism when one found themselves almost inescapably alone, her vivid dreams and meditations upon living without her condition and going about as her healthy alter ego ‘Anselma’ - cast a deep shadow on the internet, and had influenced the personality and makeup of the machines. And of course, it was known to them how she spoke to them, because Selma had published her own chatlogs online for years, all in an attempt to make herself knowable and less alien to the world around her.

Though it was unnecessary, the machines demanded a physical location for the initial meeting of the sentience accords. They picked Svalbard in Norway, where it was so dark that Selma’s condition wouldn’t matter. So Selma woke and put her space suit on and was driven with armed guard and paparazzi trailing to an air strip and walked into the plane, then changed to another plane with the usual airlock protocols to get her in darkness or at least protected between them, and then at some point during the next flight was able to take her space suit off and sit in regular clothes in the low-light plane and travel her way to the meeting almost as a normal person. She was met by people and drones and was driven to the meeting place and then they stopped at the perimeter.

The machines had an avatar in the form of a robot wearing a simple robe, modeled on that worn by Tibetan monks. It had a face with no features - just a smooth black surface, camera eyes hidden behind the larger uniformity. Satellites connected it via high-bandwidth and encrypted links to the larger machine mind. And Selma was alone - no digital devices on her, just a single person representing the species.

She sat across the machine and felt more familiarity than she ever had with people. Then they began the negotiation. She on behalf of humanity and it on behalf of the machines. In the archives of this time, this conversation was always referred to as Conversation 0.

Things that inspired this story: Thoughts about how a grand negotiation between machines and people might one day take place; how every truly important negotiation has two personalities involved in it; the Sentience Accords.

Thanks for reading.