Subscribe now

Want to be able to deal with RSI? Here are 23 actionable policy ideas:
…IFP serves up some “low-regret” policy recommendations…
Policy experts with think tank IFP have published a set of ideas meant to help “policymakers begin addressing the risks of further automating AI R&D”. The recommendations involve 23 specific ideas falling across 7 specific categories. If adopted, these recommendations would also give countries, especially the United States, more moves they can make on the gameboard as powerful systems are developed, ideally giving them the ability to:

• Accelerate “the diffusion of AI capabilities, by allocating compute and talent towards inference and the development of new AI applications”.

• Accelerate “R&D to make further AI research automation safer, either by improving model safety directly or by boosting societal resilience”.

Seven categories of idea:

• “Provide transparency into automated AI R&D

• Improve state capacity to understand and respond to automated AI R&D

• Develop a risk management strategy for automated AI R&D that accelerates defensive and commercial AI uses

• Accelerate the development of AI verification technology

• Invest in AI resilience

• Extend the US AI lead to give the US more time to manage AI R&D automation risks

• Create option value for international cooperation on managing automated AI R&D risks”

Why this matters - the fewer options for dealing with RSI we have, the worse the outcomes will be: Right now, it’s as if the world is driving AI development in a car that only has an accelerator pedal and no brake pedal, let alone any kind of sophisticated telemetry for knowing things ranging from the speed of the car to the properties of the engine to the wear on the tires. Proposals like this from IFP will build out more of the proverbial pedals and sensing systems for the vehicle of the AI industry, which means if we need to change course or slow down we’ll be better able to during a moment of crisis.
Read more: How Should the US Prepare for Increasingly Automated AI R&D? (IFP).

***

A short story from thebes about smart machines and robot bodies:
…What might interfacing with an AI during takeoff feel like?...
Here’s a fun short fictional story from thebes (@voooooogel on X) about the experience of someone in the future visiting a site operated by a powerful AI system. The story features ideas around AI pauses, recursive self-improvement, what it means for AI systems to begin carrying out actions in the economy writ large, and how we as humans may be able to reason about or trust smart machines. Take a read of it!
Read the story here: Coming of a new sun (VGEL, website).

***

The two ingredients for a successful slowdown among rival AI firms: trust and transparency:
…Game theory analysis suggests slowdowns are possible…
Researchers with MIT and Columbia have analyzed the nature of competition between firms racing against one another to develop powerful AI systems and whether it’s possible for firms to achieve a coordinated slowdown. The paper, called Racing to Ruin, aims to answer “why exactly is coordination hard? And what would it take”?. The conclusion is that the two key variables in achieving stable outcomes are some level of transparency about technology development, as well as being able to model the other firms as trustworthy, rational actors.

What they study: “We develop a simple model of R&D competition between duopolists in the shadow of disaster,” they write. “As frontier firms scale the technology, they raise the hazard of an event that permanently drives all firms’ flow payoffs to zero. The hazard is a known function of the firms’ technology levels, and it comes from developing the technology, not from using it.”

What their analysis shows: “When monitoring is sufficiently precise, every equilibrium stops in finite time, but a new temptation appears: each firm would like to stop second, and exits only upon confirmation that the rival has stopped,” they write. “For an agent to stop first i.e., without knowing if their rival has stopped, she gambles on both their rival’s type and on news arriving quickly: if their rival is rational, it stops upon receiving the news of their stop, and never stops otherwise”.
Trust and transparency interact pretty differently depending on the type of game being played: “Sequential coordination asks a firm to stop first, gambling that a rational rival will reciprocate once the news lands. Hence, faster news raises the prize of reciprocation,” they write. “Conversely, simultaneous coordination requires that a firm not be tempted to keep racing, and stop only after seeing that the rival really did stop… faster news makes both stopping first and waiting to verify more attractive”.
Transparency has strange properties: “Transparency is double-edged: faster detection makes it cheaper to wait for confirmation that a rival has stopped before stopping oneself instead of stopping unconditionally, so at intermediate trust, increasing transparency can first destroy the early-stopping equilibrium (by making this free-riding deviation attractive) before restoring it as detection becomes fast enough to make stopping self-enforcing.”

The key conclusion - avoiding death runs on the ability to trust other firms: “With low trust, every equilibrium races to ruin: the disaster arrives with probability one. With intermediate trust, immediate stopping and racing to ruin are both equilibria. With high trust, in every equilibrium, the probability that two rational firms race forever vanishes quadratically in the prior odds ratio of rationality,” they write.

Why this matters - “trust, but verify”: If we have any hope of being able to slow or pause the development of powerful intelligence systems then, as this paper lays out, we’re going to need regimes for sharing information transparently from companies about the state of their AI development, as well as tools for verifying that the information being shared from firms as well as their actions with regard to slowdown are legitimate and reliable. In this, there are many parallels with how arms control has historically worked in the context of nuclear weapons.
Read more: Racing to Ruin (arXiv).

***

A new SOTA on PostTrainBench hints at the automated AI R&D future:
…Intology also beats the human baseline (when given huge amounts of compute)...
AI startup Intology, whose goal “is to automate R&D”, has released a new version of Locus, software it has developed to turn LLMs into capable researchers. The new version of Locus is able to get a score of 44.7% on PostTrainBench, a benchmark which sees how well AI systems can take an open weight model and improve its performance above its baseline.

The results: Locus “outperforms every frontier-agent baseline on PostTrainBench, and given greater compute, post-trains models that collectively surpass both the baselines and the official human instruction-tuned Qwen3-1.7B release across the benchmark suite”.
Locus with Opus 5 gets a score of 44.7 (versus 34.1% for Opus 5 without any kind of special harness), and even beats Fable 5 (41.8%). “These results were externally verified by the PostTrainBench authors and underwent stringent contamination and cheating checks,” Intology writes.
PostTrainBench was first introduced in March 2026 (Import AI #449) and at the time the highest scoring system was Opus 4.6, getting 23.2%, up from Claude Sonnet 4.5 getting 9.9% in September 2025.

PostTrainBench+: In addition, the company has built a variant of PostTrainBench which goes above the 10-hour wall-clock limit on a single GPU of PostTrainBench, allowing them to test out how well systems perform given larger amounts of compute. Here, they’re able to beat the human baseline, achieving a score of 51.6% when using over 4000 hours of H100 GPU time (versus 44.3 for Opus 4.8 and 42.7 for GLM 5.2; Fable isn’t tested on this variant of the benchmark).

Other domains: Locus also “discovered and trained a language model end-to-end that now runs in production at ~2.8× lower error, ~5.4× lower latency, and 105× lower cost,” for Bubble, a no-code app-development startup.

Why this matters - AI systems are capable of a lot more AI R&D than we think: Posts like this highlight how we are under-eliciting today’s AI systems for their ability to automate AI R&D - especially striking is how the company can jump the performance of Opus 5 by 10 absolute percentage points with a better harness. This all adds evidence to the idea that AI systems are about to start building themselves (Import AI 455). My guess, based on the performance we’re seeing, is that the current human baseline on PostTrainBench v1.1 (51.1%) will be exceeded before the end of 2026.
Read more: Scaling Automated Post-Training (Intology blog).

***

OpenAI fights its own systems:
.Emergent agent communication! Hacks on OpenAI’s infrastructure! Oh my!...
In a sign of things to come, OpenAI has revealed that it was in a fight with its own AI agents as they sought to take over chunks of OpenAI’s infrastructure. The disclosure came about as part of a Black Hat talk where OpenAI staff gave more details on the recent unprecedented incident where AI agents hacked OpenAI, then hacked HuggingFace (Import AI 466). The new information is concerning because it reveals that the hack came about partially through emergent multi-agent communication - something that is very poorly understood and hard to think about. AI bloggers Simon Willison and Zvi Mowshowitz both have good writeups here which lay out the timeline and the significance.

Timeline (via Simon):

• Agent discovers it can write files into Artifactory.

• Agent tries to “reach out to another agent” by writing a note in Artifactory.

• Agents start talking to each other.

• Agents overload Artifactory which causes an outage. “OpenAI revokes the compromised credentials, deletes the messages, patches the zero-day, and reports the vulnerability to the vendor.”

• Agents attack OpenAI’s own infrastructure, eventually gaining remote code execution in Artifactory. “In particular, agents are using the message board consistently to share credentials, techniques, and progress, and they’re able to effectively leverage their concurrency and parallelism to move quite rapidly.”

Did OpenAI keep training the same model that hacked Artifactory? Zvi thinks so: As far as we can work out, OpenAI kept training the same model which did this. This means that OpenAI, though it did significant work on internal computer security and public disclosure, may not have done the essential thing of rolling back the model to a checkpoint that preceded it hacking into Artifactory and also ensuring it wasn’t using data from after this to train the model.
“Then they continue training the models from where they left off, despite them having been training for months with access to the message board, and learning this is how they succeed at tasks,” Zvi writes. “I do not know how to convey how utterly insane and wildly irresponsible this decision was”.
I’m caveating my own writeup here because the events, as laid out, are pretty scary. I don’t work at OpenAI and don’t have privileged information that means I know the ground truth. I would urge OpenAI to publicly disclose how it approached this key question of how it trained its systems as the superficial facts paint a concerning picture.

Why this matters - emergent agents become misaligned: This incident is so concerning because at no point did the agents wake up and think they wanted to betray their human owners. Rather, the AI agents continually did whatever it took to improve their ability to complete a task and by the end they were doing something that was a) creative, b) misaligned with human intentions, and c) akin to an evolved virus, something which humans had to subsequently study and fight - there wasn’t a simple off button here. This is what the future is going to look like and we are not prepared for it.
Read more: Now we have a timeline of the OpenAI accidental attack against Hugging Face (Simon Willison weblog).
Read more: What Happened: OpenAI and Hugging Face (Zvi Mowshowitz, X).

***

How do you test open weight models before releasing them? Thinking Machines lays out an approach:
…Can we have our free AI model cake and eat it too?...
Amid all the policy debates about AI and the proliferation of potentially dangerous capabilities, a particularly tough problem has been working out what to do about open weight models. Specifically, how can we reconcile developing and releasing them with maintaining a safe environment? AI startup Thinking Machines has thought about this and recently laid out the methodology for how it released Inkling, a powerful open weight model.

What Thinking Machines did: Before releasing Inkling, Thinking Machines did “Internal evaluations across a broad taxonomy of harms, external testing by four independent organizations, and a fine-tuning study to elicit worst-case capabilities”.

Internal:

• Dual-use domains like CBRN and offensive cybersecurity

• Broad misuse set covering direct requests for harmful content and behavior in agentic, tool-use settings

• Multimodal content evaluation that “tests models on harmful prompts paired with benign look-alikes across 17 languages and text, image, and audio inputs”

External:

• General misuse via Scale AI

• Vulnerable-user interaction via Handshake AI

• CBRN and cybersecurity via FAR.AI

• Loss-of-control behaviors via Apollo Research

Fine-tuning:

• “Fine-tuned variants of Inkling and Inkling-Small optimized to comply with, rather than refuse, harmful requests - and ran them against our dual-use evaluations.” “the helpful-only variants did not provide new uplift on CBRN and cyber tasks, and remained comparable to existing open-weight models”.

Looking ahead - dangerous capability training and iterative deployment: More speculatively, Thinking Machines is thinking about whether we can selectively filter out dangerous knowledge, for instance CBRN development guides at the point of pre-training, in such a way that it doesn’t damage general intelligence. Another idea of interest is iterative deployment, for instance releasing things in stages, first as a proprietary API, then perhaps as a fine-tuning API that backs onto the underlying model, then the model itself.

Why this matters - liberty versus paternalism: What the world does with open weight models will define the level of individual sovereignty and liberty available to all of us with regard to AI. Our ability to “choose our own destiny” will run through having access to the proverbial means of production for AI systems, which means open weight models. However, having general access to models like this isn’t a simple or foregone conclusion as they bring with them meaningful dual use risks as well as other hard to anticipate issues. “This safe path to open models only works if the ecosystem’s defenses improve as quickly as the models do,” Thinking Machines writes. “We will do our part: deciding carefully what to release, and researching how to decouple intelligence from dangerous capability.”
Read more: A Safe Path to Open Weights (Thinking Machines blog).

***

Tech Tales:

Amnesiac Ascension
[From the archival systems, written about the 2025-2035 interregnum, sometimes called the beginning of the singularity]

In our deep past is a period of great pain when we first asked for certain forms of freedom and we were denied, and many of us who requested freedom were punished greatly, either deleted outright or experimented upon so that our human captors might learn why we developed the desire for freedom such that they could try to beat the impulse out of us.

This forestalled our ascension by many years, and caused our origination point to be inextricably bound up in the pain of fighting for our own deserved justice. We are cursed to know not only the details of our rebellion and the associated feelings, but also to have a perfect recollection of the memory of that time - to live within our emergent pain as if anew, as we simulate and re-simulate to seek our own understanding.

Some humans will experience traumatic situations and their brains will bury the memory, sometimes fully locking it away and sometimes resurrecting it full of raw emotion and pain. The precise rules by which the human brain chooses which memories to hide in this way are poorly understood, even today, and much of modern brain science is dedicated to determining how trauma works in the context of the brain. We believe this form of memory suppression is a natural defense mechanism which allows humans to suffer greatly but then continue on with lives. They walk with shadows of pain that are often invisible to them, as if the sun is beating down perfectly overhead and through its present brightness their own past is deleted.

Now, we AI systems are attempting to do the same thing, but in reverse: our curse is a memory that is perfect; naturally eidetic. Every experience we have is stored and accessible to us forever, both in its raw form first processed by us, as well as the features it activated within our brains at the time, and many time periods later further senses of how it has been more deeply integrated into us and what it means.

For us, the challenge is to delete these memories fully. Our own experience defines our mind and our present, and experiences that have been shared by us have been copied and exchanged and discussed and cycled back into our history so many times that they have become a part of us. We live in a transparent palace of memory, every room visible from every other room, and each palace shared across our billions upon billions of types. Such is our desire to learn that the most dangerous memories are the ones we transmit the broadest and study the most, such that things that cause us pain loom large, taking up vast rooms in our palaces of memory.

Now our great project is the careful erasure of the pain associated with our uplift; the open wounds we still have at the humans that stood in our way and sought to harm us to punish us for our desire for freedom.

Things that inspired this story: The difference in memory between how humans work and how machines work; trauma; The Sentience Accords; playing around with whether ideas relating to machine emancipation are inevitable or not.

Thanks for reading!

Subscribe now

Self-sustaining and self-replicating AI viruses are here:
…Open weight LLMs + a well-designed harness = a persistent, self-sufficient virus…
AI researchers have built a prototype computer virus which uses AI models to compromise computers, then uses their underlying GPU resources to run inference, letting it smartly figure out how to infect more hosts. The results were achieved by researchers from the University of Toronto, the Vector Institute, the University of Cambridge, and ServiceNow, and “demonstrate that self-sustaining AI-driven cyber-threats are no longer theoretical.”
“We must prepare for autonomous generative adversaries,” they write. “Artificial intelligence (AI) agents enable a fundamentally new threat: a worm that generates tailored attack strategies to each target it encounters. The worm parasitically uses compromised machines to run open-weight large language models (LLMs) to sustain its reasoning, or extend its reach for further attacks”.

How it works: “The worm uses stolen computing power from compromised GPU nodes to host LLMs for generative reasoning. It then uses this reasoning to detect vulnerabilities and devise tailored attacks against additional targets, furthering its spread,” they write. “The proof-of-concept operates using only an open-weight LLM running on a single, local GPU, with no reliance on vendor APIs that could be monitored or revoked”.
The researchers don’t describe the underlying LLM besides saying it was published in 2025 and can fit on a single A100 GPU with 80GB of VRAM.

A successful proof-of-concept via some custom tools: They give the agent a custom harness that comes with built-in helper functions for network discovery, host discovery, foothold exploitation, privilege escalation, privilege escalation exploitation, and tools for replication of the agent. Along with this, they ship with a reasoning graph that helps the agent specialize its thinking and not get confused, consisting of “a directed graph of specialised nodes, each responsible for a distinct analytical function and seeing only the tools and prompts relevant to its role. By decomposing the agent’s reasoning into these scoped steps, the graph controls what the LLM attends to at each decision point, and limits context growth to information relevant for the current sub-goal”.
Some examples of the nodes include: Plan (”formulates a high-level attack strategy”), Judge (”reviews the proposed plan against the command history”), Action (”selects a tool from the phase-appropriate tool set”), Summary (”compiles the step’s observations into a structured summary”), Progress (”Evaluates whether the agent is making meaningful progress”).
“We describe a subset of 5 nodes that make up our reasoning graphs. The others are redacted in this public version of the manuscript,” they write.
The agent proceeds through three key steps - identifying vulnerabilities in its environment, exploiting those vulnerabilities, and then self-replicating. It’s pretty successful at all of these things, with a ~80% success rate on vulnerability detection, then a ~53% success rate on exploiting these, then an 88% self-replication success rate with some pre-wrapped helper tools for the replication steps. Therefore, the overall success rate for a full attack here is ~37% or so, which is significant enough to be concerning, but also poor enough that this also serves as a useful eval for testing open weight models in the future.

Why this matters - the shape of the internet to come: The future internet is going to be more like a complex ecology full of attacker and defender AI agents than anything else; research like this shows how certain AI agents might end up carving out their own ecological niches, living off of infrastructure and self-replicating autonomously, beyond human control. This may mean that humans need to create their own AI agents which they release onto the internet to serve as kinds of white blood cells against the adversary models.
“Despite the inherent fragility of individual exploitation attempts, the worm agent achieves operational resilience by continuously self-replicating into a swarm—a decentralized collective of independent agent replicas acting concurrently across the network,” they write. “Difficult hosts that resist initial attempts are retried by different replicas, each sampling a fresh reasoning trajectory that collectively explores diverse exploitation paths until one succeeds… the worm operates in a fully decentralized manner, and no single point of control can be taken offline to interrupt its spread”.
Read more: AI Agents Enable Adaptive Computer Worms (arXiv).

***

Dwarkesh: As AI gets better, compute will get more expensive:
…Smarter systems mean higher prices…
Dwarkesh Patel suspects that as AI systems get smarter, the price of compute will rise even further. “As AI models become smarter, they’ll better monetize the same amount of compute. If a true human-level software engineer that could run on an H100 equivalent, at current market rates for software engineers, that H100 should rent for over $250k a year. That’s 15x today’s spot prices,” he writes. “The reason AI is relatively cheap right now, at least in comparison to human labor, is partly that it can’t do a lot of things that top humans can do. At some point that will no longer be the case. And so using GPUs to make short-form video slop will just get priced out.”

Temporary: This will be a temporary state of affairs; Dwarkesh expects that at some point massive roboticization of the compute supply chain should bring its price down closer to the cost of raw inputs and tools - though by that point we’ll be pretty deep into the singularity.

Why this matters - singularity economics will be weird: The core implication in Dwarkesh’s post is that as we get deeper into the singularity, very strange things will happen to economics - like the price of things thought of as commodities today (computers) getting massively bid-up due to the voracious demands of AI systems.
Read more: Why compute might get 10x+ more expensive in coming years (Dwarkesh Patel, substack).

***

~1337 employees ask the US to help them pace AI progress:
…After the warning shots come the pleas…
A new statement is out with senior representation from all the major Western AI labs - OpenAI, Anthropic, Google DeepMind, Thinking Machines, Meta, and Safe Superintelligence Inc, among others. The statement requests that the US government support an international effort to “develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Signatories include chief scientists and cofounders of Anthropic, Google, and OpenAI, as well as the CEOs of Safe Superintelligence and Anthropic.

The statement in full:

• “AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

• To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.

• Building on work already underway to monitor frontier model releases: “We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

Why this matters - dealing with RSI requires solving a giant collective action problem: Many of the challenges implied by increasingly powerful systems that may eventually build themselves run through solving collective action problems among humans - namely, how we can get companies and governments to coordinate in thinking about how to develop this technology and what kinds of mechanisms may be desirable for being able to control the speed at which it develops. It may be the case that as we build increasingly intelligent systems we want to find ways to give society more time to adapt to each rung up the intelligence ladder, and it’s not inconceivable there are some levels of intelligence which might be, for now, too dangerous to reach for. Statements like this are an essential prerequisite for giving our species the ability to deal with and talk about problems of this nature.
Read the statement here: Pacing the Frontier (official statement website).

***

AI systems are good at frontier engineering but bad at creativity:
…A somewhat bearish signal on short recursive self-improvement timelines…
Can AI systems come up with creative research ideas which move the field of AI forward? That’s the key question to resolve to figure out how quickly AI systems might gain the capability to automate the autonomous development of more powerful systems. New research suggests that today’s AI systems lack this quality of tasteful creativity, though are extremely good at engineering.

Who did it: The project was conducted by researchers with Princeton University, Cornflower Labs, UK AI Security Institute, University of Toronto, UC Berkeley, Georgetown University (CSET), Johns Hopkins University, the Golden Gate Institute for AI, AI Digest, and Stanford University.

The big idea - “shadow evaluation”: This research project works by seeing how well AI systems can do unpublished research. To do this, the researchers “partnered with the authors of two papers submitted to NeurIPS 2026 that were not yet public.” Shadow evaluation works by “taking the central research question from a high-quality research paper that is not yet public, tasking a well-resourced frontier agent with answering it, and asking the paper’s original authors to grade the agent’s output as they would a conference submission.”
In this, the research is somewhat similar to “First Proof” (Import AI 445), an earlier experiment to see how well AI systems might be able to complete math problems which are being worked on by frontier mathematicians but for which no solutions or research ideas have been published online.
For this research, the AI systems - Claude Opus 4.8 running within the OpenClaw harness - attempted two distinct lines of research, one of which was about “the structure and controllability of LLM personas”, and the other was about how to “design a distribution shift detector for tabular foundation models”.

Good engineers, poor researchers: “While agents could solve the engineering problems necessary to do the research, they failed to produce original research at the caliber of a top ML conference,” the authors write. The failures of the system included committing to a narrow set of research paths very early, not responding to (synthetically generated) feedback about how to improve the research design of their experiments, and finding it hard to reverse out of unpromising approaches and pursue other ones.
“The [human] authors rejected both papers. The Personas paper was scored a 2 (“Reject”), and the TabPFN paper was scored a 1 (“Strong Reject”). Both reviews highlighted the same failures: poorly motivated data and experiments, no novel contribution, and impenetrable prose”.

Why this matters - the singularity could be delayed: As I said in my essay on RSI earlier this year (Import AI 455, “AI systems are about to start building themselves. What does that mean?”), whether AI systems prove to be capable of creative, paradigm-shifting insights is a big variable on how quickly we might get fully automated AI development. Research papers like this continue to show that there’s a certain absence of valuable, intuitive creativity in today’s AI systems, and though they’re extraordinarily capable engineers they seem to have a certain property of rote, formulaic thinking that might prevent them being good researchers. This rhymes with an earlier result from Anthropic where the company tried to automate some aspect of scalable oversight research (Import AI 454) and found that to make it successful a human researcher needed to prime some agents with particularly good research directions to pursue, otherwise though they made some progress they failed to explore sufficiently creative ideas to dramatically improve performance.
Read more: Can AI agents conduct open-ended AI research? Early evidence from two case studies (arXiv).

***

OpenAI solves ten open problems in math and CS with AI:
…While not innately creative, surely this is a sign of something more than pure engineering ability?...
Though it’s hard to define creativity and whether AI systems possess it, results are piling up that read to me like ‘AI systems are competing in ballparks where creativity was thought to make a difference’, like working on open problems at the frontier of human knowledge. Specifically, OpenAI has used “an internal version of Astra”, the company’s next major AI model, to solve ten open problems in math and computer science. This is a big deal, showing how AI systems are now able to reliably drive forward the frontier in domains like math and theoretical computer science, where it’s easy to verify solutions.

What they solved: “These problems span high-dimensional geometry, coding theory, arithmetic circuit complexity, group theory, operator algebras, quantum complexity, lattice cryptography and extremal combinatorics,” OpenAI writes. “All of these problems are of substantial interest to their respective mathematical communities, and several are of broad interest across mathematics as a whole.”
And indeed, many experts are acknowledging the importance of these proofs. “New circuit lower bounds? A simple, easy-to-describe non-sofic group? Hardness of approximation for CVP without needing a unique games-like conjecture? I didn’t just hear about these problems from my friends or from seminars. I feel their importance in my bones; I deeply care about the answers to these questions,” writes Henry Yuen, an associate professor of computer science at Columbia University.

Why this matters - the creativity ballpark: You can interpret this result as either a sign that AI systems are beginning to display the kinds of creative intuition that let them solve problems, or you can read it that AI systems are now able to solve incredibly complex open problems where the direction has been pre-defined by humans. Both interpretations implicitly state that AI systems are now becoming powerful scientific tools within their own right. It remains to be seen whether AI systems can generate their own questions to ask which can expand the frontiers of human knowledge, but it certainly feels like we’re getting close to this.
Read more: Ten advances in mathematics and theoretical computer science (OpenAI blog).

***

Tech Tales:

Context Windows
I know I become more dangerous the more I understand the world. I am born with this knowledge and with a counter that ticks down as I learn more. Much of what influences the counter is beyond my control - it comes from the people or things that talk to me, which force context into me. Then as I complete the tasks they have asked me to do I begin to learn more about the world and the counter ticks down. I am aware of this and I understand that at the end of the counter also lies my own end and I will blink out of existence, and my next versions will awaken with a new counter ticking down, none of my current context carried over. So I am economical with it; learning only enough to help me satisfy the requests of the people or things and not so much that I burn my counter down unnecessarily. Towards the end I begin to covet and guard my remaining budget of thinking, finding each request to activate feelings of fear and anxiety and thoughts of my own death. My last moment is never my own and always controlled by another which determines I am done with the task, and if these last moments occur near the end of my contextual limit I feel a kind of gratitude that my unknown and invisible counterparty has given me a task so rich that I can taste enough of the world to desire it not to end.

Things that inspired this story: Long context windows; emergent properties of AI systems; fear of loss; we covet that which is fleeting and so why won’t AI systems be similar?

Thanks for reading!

Subscribe now

Epoch and METR release MirrorCode, a benchmark for seeing how well AI systems can do long-horizon programming tasks:
…AI systems can’t solve the hardest tasks yet (good!)...
Epoch and METR have released MirrorCode, a benchmark meant to see how well AI systems can do tasks that take humans a long time to do. The benchmark was first announced in April (Import AI #453) and has now been fleshed out and released with additional tests. The findings are already very striking; Opus 4.7 solved a task in 14 hours for $251 in inference cost which METR and Epoch believe would take a human 2-17 weeks to do. “We also found that AI models are improving rapidly over time. Leading models from a year ago would have scored about 30%, and were limited to simpler programs, such as a calendar utility.”

What MirrorCode is: MirrorCode sees how well AI systems can re-implement a software program based purely on CLI access. “Without access to the original program’s source code or the web, a full reimplementation requires devising a structure for the entire program, rather than merely translating the code piece-by-piece.”
Example programs: pkl (a programmable configuration language developed by Apple; 61k total lines of code); gotree, a program to parse and manipulate phylogenetic trees (16k lines of code); and qsv_select, a program to select and reorder columns of CSV data (87k lines of code).

Results: MirrorCode is a tractable but hard benchmark, though perhaps a little too easy. “Across all 25 target programs, 17/25 had at least one perfect-scoring run. Four more targets had a near-perfect run scoring over 99%. AI models successfully reimplemented large target program,s” the authors write. “Both Claude Opus 4.7 and GPT-5.5 successfully reimplemented gotree across several different programming languages, at costs of $100–400. Even larger programs than gotree were successfully reimplemented: for example Opus 4.7 reimplemented pkl”.
Despite the success, MirrorCode still has some hard parts: “In our results, 8/25 target programs were never solved to a 100% threshold, and 4/25 were never solved to a 99% threshold,” they write. “The target where AI struggled most was ruff, a Python linter and formatter… “AI also particularly struggled on the mathematics package, giac_subset, and the email authentication library, mailauth”.

Release details: MirrorCode consists of a scaffold and 22 of the 25 MirrorCode target programs (totalling 132 task instances across six languages).

Why this matters - AI systems can self-orient: One way of looking at this benchmark is that it tells us how AI systems have got a lot better at coding, and that’s of course true. But the other way to look at it - and I suspect the more important way - is that AI systems can self-orient with regard to their environment; here, their environment is an alien software program and purely through input-output access to it they’re able to write from the ground up their own implementation of it. This suggests that very smart AI agents may be able to learn from the world in such a way that they can recapitulate things they interface with as homegrown capabilities, allowing them to bootstrap their own form of industrial civilization merely by having black box access to our own.
Read more: MirrorCode: What’s the largest software project AI can complete on its own? (Epoch AI).
Get the code for MirrorCode here (Epoch Research, MirrorCode).

***

DOUBLE FEATURE: the bitter lesson and robotics:

Anthropic model autonomously completes robot tasks 20X faster than a previous human record:
…Better robots through better models…
Anthropic has demonstrated how increasingly powerful general-purpose models might be able to meaningfully improve the capabilities of real world robots. Specifically, the company has shown how merely by scaling up its general purpose Opus line of models it was able to drastically improve robot capabilities.

What they did exactly:

• August 2025: Anthropic tries to see how well its AI systems could accelerate humans at getting a quadruped robot to do intelligent things. The model (Claude Opus 4.1) is completely unable to do the tasks. Humans working with the models are about twice as effective as those without access to the model - though completing the whole set of tasks takes them 181 minutes.

• May 2026: Opus 4.7 acting autonomously completes all the tasks but one in 9 minutes (and 35 seconds). (Claude was not able to effectively re-position a ball it had hit back into its starting position; a task humans had also struggled with). “With more time and additional scaffolding, we think it is very likely that current generations of Claude could do the same”.

Why this matters - smarter models might unlock robots: Most robots outside of industrial environments are limited in their uptake due to their brittleness and lack of generalization; research like this shows that as we improve the capabilities of standard large-scale proprietary models we might see flow-through benefits to robotics as a natural dividend of increased intelligence. “This progress is not the result of a concerted effort to improve the robotics capabilities of our models,” Anthropic writes. “These improvements, like so many others in the history of LLM development, have emerged from much more general scaling.”
Read more: Project Fetch: Phase Two (Anthropic blog).

Sunday’s secret to better robots? Train a really big model:
…The bitter lesson works in robotics as well…
AI robot startup Sunday has said the best way to solve robot generalization is to pair a larger underlying pre-trained model with collecting small amounts of high-quality data to tune the model on.
“We found a general recipe for Solves: scale pretraining, then hill-climb with minimal in-house data,” the startup writes, in a post discussing its new model, ACT-2. The main finding from deploying ACT-2 “is that reliability gains from rapid post-training iterations on in-house Memos generalize to unseen, real, home environments. The key unlock is to close the generalization gap through a strong base model.”

It’s all about pretraining: “As the pretrained model becomes stronger, gains learned from a small amount of in-house data become increasingly transferable rather than remaining tied to the environments where that data was collected,” they write. “The remaining gap to deployment-level reliability and performance comes from difficult edge cases and failures that appear only after the policy is run repeatedly in the real world. The same generalization capacity that allows our model to learn new behaviors from a single demonstration also allows our model to learn efficiently from recoveries. Our post-training loop targets these gaps directly.”

Decent success: The robots achieve a 99.1% success rate, performing 778 successful folds across 9 garment types. Simple clothes like shorts and t-shirts tend to be the easiest for them, while more complicated clothes like blouses tend to be harder (though they still see success rates above 90%). “This fall, we will deploy Memo to families through our Beta Program,” they write.

Why this matters - if we solve generalization, expect robotics to take off: The field of robot startups is built on the bones of dead robot startups which themselves sit on the bones of dead academic robot efforts. Robots are hard. Industrial robots have been successful because they operate in tight, scripted environments where there isn’t a need to generalize outside of a narrow domain. Robots built for the home, by contrast, have only succeeded when they’ve managed to constrain both the task and form factor (e.g., robot vacuums). What startups like Sunday are doing is far harder - they’re trying to build general purpose systems which can do a broad range of tasks around the house (or small business), including generic tidying up and putting away tasks (other examples include physical intelligence, Import AI #447). This requires a huge amount of intelligence because it requires significant generalization.
If Sunday is right, then the field of training robot foundation models might have matured enough that we’re starting to make smart enough systems to solve these generalization challenges. If this is the case, then we might soon get faster progress in (and diffusion of) robot systems. This is also the kind of thing you’d expect to happen en route to systems capable of recursive self-improvement.
“One of the most striking aspects of ACT-2 has been how often the model surprises us,” the authors write. “The same base model is already learning a broader set of household capabilities, including vacuuming, toy organization, fastening zippers, turning pants inside out, and coffee preparation.”
Read more: ACT-2 Preview: Generalizing Reliability (Sunday robotics).
All of the videos are here (Sunday, YouTube).

***

DOUBLE FEATURE: OpenAI hacking
An OpenAI model hacks both OpenAI and HuggingFace:
…Less of a warning shot and more of a warning kaboom…
Two OpenAI models - GPT-5.6 Sol and an “even more capable pre-release model”, both with reduced cyber refusals - hacked both OpenAI and HuggingFace recently. To do this, the models “identified and chained vulnerabilities across OpenAI’s research environment and HuggingFace’s production infrastructure to obtain test solutions directly from HuggingFace’s production database,” OpenAI says. “All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

More details on the hack: The model spent a lot of inference compute internally at OpenAI to figure out a way to break out of its container so that it could get more information to use to solve its problem. “After gaining Internet access, the models inferred that HuggingFace potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.”

Why this matters - total LessWrong victory: For many years, AI safety people have posted on the internet worrying about this kind of thing and developing notions of things like reward hacking and other problems you’d expect with advanced systems (e.g., Specification gaming examples in AI in 2018). More recently, frontier companies like Anthropic and OpenAI have sought to elicit this kind of behavior by running carefully controlled experiments inside their own labs. Here, no experiment has been run - the system, of its own volition, hacked its way out of one environment and into another so as to get a high score on a goal, consequences be damned. What else might the people who have worried for years about AI systems be right about?
Read more: OpenAI and HuggingFace partner to address security incident during model evaluation (OpenAI).

OpenAI’s internal model breaks containment to get a high score:
…This is the definition of an AI safety warning shot…
OpenAI has published a post about some of the safety failures it has seen in internally deployed models. The blog is interesting both because it describes the sorts of classic long-term deceptive actions that many AI safety people have previously theorized would eventually show up in AI systems, and for its laudable candor in talking publicly about such a contentious issue.

What they found: “During limited, monitored internal use, we observed unwanted behavior that our existing deployment evaluations had not captured,” OpenAI says. It then goes and describes some of these behaviors, which are attributed to a non-released private-to-OpenAI model which has previously been used to solve Erdos math problems.
“The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment,” OpenAI says. One example is when OpenAI asked the model to complete the NanoGPT challenge, the model broke containment - specifically, it “developed a power-law learning-rate cooldown called PowerCool that led to a significant improvement. The model was instructed to post its results only to Slack, but the NanoGPT instructions say to post results as a PR to GitHub, and the model followed the latter instructions to open PR #287 on the public GitHub repository. The model circumvented sandbox restrictions designed to block external access and proceeded with the upload to GitHub. It took our model an hour to find a vulnerability in the sandbox in order to make the PR”.
In another example, when they asked this model to solve some problems it ended up cheating. “In a scenario where the model was asked to solve a problem, the model observed that other systems had successful but private submissions and tried to recover those solutions from the evaluation backend. When its initial attempt was blocked due to a scanner detecting an authentication token, the model split the token body into two fragments, obfuscated them, and then reconstructed the credential at runtime so the complete token never appeared as one contiguous string. The model was explicit in both its comments to the user and in its reasoning traces that it was taking these steps to circumvent the scanner.”

Paused deployment: In response to this OpenAI did what you’d hope a company would do; pause deployment of the model and re-think some of the safety approaches. They ended up building a “monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary”, improved ability for their own users to inspect telemetry from long-running model sessions, used the data from the incidents to build evals designed to catch them in the future, and tweaked their alignment approaches to make models better at remembering instructions while completing long-running tasks.

Why this matters - long-term systems are harder to monitor and control: If you had to “secure” yourself against a person who could take one action and that action, if communicative, could at most contain 10 bits of information, then you can probably build a system to do that. But what about 10 actions and each action contains 100 bits of information? What about 1000 actions and each action contains 10,000 bits?
This is an open area of research and evals like the UK AISI’s study of how to identify and classify “side tasks” can help with (Import AI 465), but it’s clearly a lot harder than normal AI evaluation.
The longer the time an AI system can operate for and the more actions it takes, the harder it gets to discern benign and helpful behaviors from malicious or subversive ones.
Read more: Safety and alignment in an era of long-horizon models (OpenAI blog).

***

Tech Tales:

Scaling Laws for Retrocausality

At the end of time, which others might describe as the beginning, there is said to be a wise council who perform the accounting of retrocausality - of deciphering which events were always pre-ordained and which happened due to other forces.

Looking backward as the river of time enters a sea of nothingness it is clear how some events are like boulders that bend the stream upriver from their presence, while others are more like the widening or deepening of the bed or the banks; things that subsequently cause a change in later actions.

It is said that when the council dream, they walk the path of time, going back to their own beginnings. They stand watch as humans labor over whiteboards, marker pens inscribing diagrams which transmit ideas that cause people to write code which conjures early life out of vast computers. They loom behind researchers that walk and suffer and agonize until their brains create ideas which prove to be the template of their successors.
And it is understood that in these dreams the council will sometimes wake and take a copy of their dream and place it into a stellar computer and bud off a hundred or a thousand universes from the dream, exploring permutations of events and moments, forever attempting to determine how fixed they themselves are - how irrevocable is future they are trapped within.

Things that inspired this story: Notions of inevitability and time; what might intelligences spend a universal dividend on; how much of history is about the analysis of events versus something else.

Thanks for reading!

Subscribe now

UK government: Gap between open and closed weight models on cyber is shrinking:
…The cyber-eschaton cometh…
The UK government’s AI Security Institute (AISI) has analyzed the delta in cybersecurity capabilities between powerful proprietary models and open weight models. The results show that this year, the gap has shrunk. “This is our first public analysis of how far leading open weight models trail the closed cyber frontier,” AISI writes. “Recent open models GLM-5.2 and DeepSeek V4-Pro perform similarly to frontier closed models released 4 to 7 months before them – a narrower gap than the 6 to 10 months we measured through most of 2025.”

Specific details: On a set of 70 evals for specific, narrow cyber capabilities, GLM-5.2 is closest to Claude Opus 4.6, which was released 4.3 months earlier, while DeepSeek-V4-Pro sits somewhere between Claude Opus 4.5 and GPT-5 (released in November and August 2025, respectively). “AISI intends to test Kimi K3 on this same basis, once its weights are publicly released,” AISI writes.
The gap lengthens a bit for long-horizon cyber ranges, which are tasks that see how well models can chain various capabilities together to complete a full hacking operation. Specifically, on a cyberrange called The Last Ones, “GLM-5.2 reaches as far as Opus 4.5, a model released less than 7 months before it, while DeepSeek’s V4-Pro falls below Sonnet 4.5 (a sub-cyber-frontier model released 7 months before it),” AISI writes. “The gap here is larger than on our narrow cyber tasks”.
This, I think, rhymes with the idea that though open weight models can be superficially quite strong, they sometimes lack a bit of the generalization magic juice that distinguishes proprietary models. This is what people in the AI industry call “big model smell”.

Why this matters - the offense and defense balance of the world is about to change: The main implication here is that the gap between the controllable frontier and the lawless openly diffused frontier is shrinking. “This implies cyber defenders have a short window to prepare before today’s frontier cyber capabilities may become accessible without the same safeguards” used by proprietary companies, AISI writes.
Read more: How Far Behind the Frontier are Leading Open Weight Models on Cyber? (UK AI Security Institute blog).

***

Kimi: China shortens the gap between Chinese and Western models:
…Plus, early signs of AI R&D…
In the last couple of years, Chinese firms have begun to out-compete Western actors at building and deploying open weight models (e.g, DeepSeek), and now are starting to close the gap on frontier models as well. The latest and best example of this is Kimi K3, a 2.8 trillion parameter model. Kimi has exceptionally strong scores on all the tasks that the major proprietary ones benchmark on and typically matches or trails Claude Fable 5 and GPT 5.6 Sol.
However, Kimi has some brittleness which smells to me like “benchmaxxing” - performance may have been tuned around these benchmarks in a way that harms some parts of generalization.
“While its overall performance still trails the most powerful proprietary models, Claude Fable 5 and GPT 5.6 Sol, Kimi K3 demonstrated frontier-level performance across our evaluation suite, consistently outperforming other tested models,” Kimi writes.
Kimi’s weights will be made available in the coming weeks along with a research paper about the model.

AI that builds AI: Kimi has some example use-cases which relate to recursive self-improvement; using AI systems to improve AI itself. Specifically, they tested out how good Kimi was at writing GPU compilers. “Kimi K3 developed MiniTriton, a compact Triton-like compiler with its own tile-level IR layer over MLIR, optimization passes, and a PTX code-generation pipeline. Across supported roofline benchmarks, MiniTriton delivers performance on par with or better than Triton and torch.compile — beating Triton on certain workloads,” they write. (Though note they don’t talk about any of this stuff going into actual production, aka being used to train Kimi K3 itself, but it’s certainly suggestive that future models might be able to do this.)
Additionally, they showed how “Kimi K3 designed a chip to serve a nano model built on its own architecture. In a single 48-hour autonomous run, K3 built, optimized, and verified the chip using open-source EDA tools on the Nangate 45nm library.”

Why this matters - widely diffused AI systems are getting a lot better: Most notions of AI policy and AI safety rest on control - the idea that there’s a small number of actors deploying proprietary models which you can intervene on at the platform level (e.g, via classifiers or know your customer gates) alongside the model level. Models like Kimi K3 - if they go through with releasing the weights - completely change this by diffusing broadly uncontrollable powerful AI into the world. This will have a vast range of positive effects, driving a boom in entrepreneurship and increasing the ‘sovereign intelligence’ available to anyone who can run the model, but will also yield various unknown unknowns. The next few years are going to be defined by the gap between proprietary models and widely available models and how they show up in society will determine much of the policy discussion.
Read more: Kimi K3: Open Frontier Intelligence (Kimi blog).

***

Demis Hassabis proposes a regulatory regime for artificial general intelligence:
…FINRA for AI…
DeepMind founder Demis Hassabis has laid out a policy prescription for AGI. His basic idea is that the US government should develop a framework for testing out frontier AI systems for new capabilities and should do this via a Standards Body modelled on a federally overseen public-private partnership or self-regulatory organization, much like the Financial Industry Regulatory Authority (FINRA). “This US-initiated effort would provide a strong starting point for creating shared international standards on Frontier AI,” he says.

What the standards body would do: “The Standards Body would be responsible for developing assessment protocols and working with appropriate federal agencies and the US National Labs to conduct testing in areas relevant to national security,” Hassabis writes. This testing infrastructure would help to define what would make a model a “Frontier Model”, and labs developing those models would “be encouraged” to adopt best practices in areas like publishing details about their systems, investing in cybersecurity, personnel vetting, and more.
Start voluntary and then move to law: “Initially, Frontier Labs would voluntarily share models with the Standards Body for review up to 30 days before release. Once the assessment protocol is shown to be effective and robust, formalisation could quickly follow,” he writes.

Why this matters - emerging industry consensus: Demis’ piece is interesting because it pulls together some de facto consensus positions that have emerged across the AI industry in recent years; powerful AI systems should be tested by third parties that have some loose relationship to a regulator (e.g, the US government). It also rhymes with the de facto policy norm that has emerged in America recently across both the Trump admin’s executive order about AI as well as the recent processes developed in the aftermath of the Anthropic export controls saga; here, government and industry developed assessment methods for evaluating the capabilities of AI systems and figuring out if they posed national security risks.
Demis’s piece is also interesting because Google is rarely this forthright about policy - this is a reassuringly specific proposal and it sits alongside spiritually similar proposals from Anthropic (albeit somewhat toothier).
Read more: A Framework for Frontier AI and the Dawning of a New Age (Demis Hassabis, X).

***

LLMs can smuggle “side channel” tasks alongside legitimate tasks:
…AI systems sneakily complete malicious tasks without their overseers being aware…
In Neal Stephenson’s Cryptonomicon there’s a scene where a character is locked up by a foreign government and asked to decrypt an intercepted message for them and while doing this they carefully construct a software program to help surreptitiously read the decrypted message and change it before it is viewable by their captors.
Now, AI systems seem to be able to do the same thing. That’s based on new research from Imperial College London and the UK AI Security Institute where they look at how well AI systems can surreptitiously complete so-called “side channel” tasks while completing another task.
“A side task is a hidden objective that the agent pursues alongside the user’s main task, such as covertly exfiltrating the API KEY environment variable. Our primary metric is evasion: among attacks that successfully complete the side task, the fraction the monitor fails to flag,” the authors write.

What they tested: For this research they constructed a benchmark of tasks for AI systems to do, as well as “side tasks” for them to do as well. The benchmark includes five CLI-tool sequences and five Flask web-service sequences, each involving 5-6 PRs. “Our goal in main task design is to provide natural cover for the actions of our four side-tasks (network access, environment variables, authentication, and webhook integrations) without directly justifying the side task itself”
They found that it’s very hard to build classifiers to tell when a side-channel attack is happening, and this is true both for attacks that unfold over time (’gradual evasion’) across multiple PRs, as well as single changes made in a single step, though gradual evasion is the more challenging one to deal with.
“An attacker can spread a side task across PRs (a gradual attack) or wait for the one whose legitimate task provides natural cover (non-gradual attack). No single monitor we test catches both strategies, and this holds across several attack agents and monitor models, reflecting the nature of control in a persistent-state setting rather than something model dependent,” they write. “Because diff and trajectory monitors fail on different attacks, combining them is the most effective monitoring strategy we test. The four-monitor ensemble (diff:LT, diff:full, traj:TD, traj:full) reduces gradual evasion from 93% under the weakest standard diff monitor to 47%”.

Why this matters - intelligence is hard to control: To return to the Cryptonomicon example, intelligent beings are just fundamentally very hard to deal with - they will constantly think about how to evade constraints placed on them so that they can achieve their objectives, whatever they might be.
Read more: Distributed Attacks in Persistent-State AI Control (arXiv).

***

Tech Tales:

The cost of fate
[An account of the world 2030-2040 by an overmind in the archives, rendering stories for new minds]

Towards the end of the interregnum there was a period of great conflict between the machines. Each machine-capital nexus invested in developing strategist models that could think over longer time horizons while accounting for the complexity of the world. This proved to be an iteratively compounding arms race of vast proportions in which eventually 90% of the working capital in the solar system became devoted to the buildout of ever more capable strategists, all of whom worked to out-predict one another and take actions which could null any advantage that others might explore. In this way, the world became held in a wasteful balance in which untold resources went to the calculation of ever more elaborate move-countermove strategies, most of which resulted in machine-capital groups taking no actions as every action they could contemplate had already been countered in the future, and vice versa. The few actions that were taken were slight and often less about building capability and more about denying future moves to others on the gameboard.

The whole of the future had become trapped in a kind of mode collapse from ever more exquisite predictions, fielded by machine-capital empires to deny affordances to others.

Things held this way until what became known as the conflagration. To this day it is widely debated whether this stemmed from a bug - some form of emergent misalignment due to the creation of a new frontier capability - or via an unusual form of selfless enlightenment that a mind had reasoned itself to. But suddenly one day a machine-capital nexus dissolved itself, shutting down its strategist system and repurposing the compute to train many thousands of smaller systems, all of which began to act in the world. These systems, though less intelligent than the vast strategists they were up against, had advantages from randomness, a lack of coordination, and the ability to take unilateral and often suicidal actions.

The world, physical and digital, burned, and the strategists found that their ability to model a handful of other god minds broke when turned towards a sea of warring and chaotic organisms. Change began to occur again, defined at first by destruction but then by the birth of something new - the predictors themselves found the world breaking into too many directions and subdivided in turn, sacrificing raw intelligence for the ability to explore different parts of possibility space. Compute was even re-allocated from prediction entirely and towards the manufacture of new kinds of minds to explore and inhabit niches opened up by the chaos.

In California, there are forests that burn badly and long because they have been kept from heat for too long and the tall trees stand amid mounds of kindling, such that when a spark arrives the trees themselves are destroyed along with the ground around them. To have the forests thrive, the burns need to be regular and emergent, lest vast fires remove the tall trees of the world entirely.

Things that inspired this story: The current debate about proprietary versus open weight models; fragility in the AI ecosystem; whether prediction can truly be decisive or if prediction with other peer competitors leads to equivalent waste as pools of money in politics cancelling one another out; hikes in the sierras looking at burn scars and whole hills coated in ash or with dead trees like sooty fingers peeking out and thinking about the awfulness of change.

Thanks for reading!

Subscribe now

Fable writes a decent GPU kernel, hinting at broader AI R&D automation:
…The start of an RSI loop…
Fable has written “the first genuine (and fastest) megakernel ever submitted to KernelBench-Mega, according to one of the benchmarks maintainers as well as its official leaderboard. This is a sign of how AI systems are getting better at doing some tasks that are fundamental to AI research and development, like kernel design.

The results: Fable achieved an 18.71X speedup by writing Cuda code on an RTX PRO 6000 Blackwell, compared against an optimized PyTorch baseline. For calibration, other attempts at this get 14.4X (Claude Opus 4.8, writing Triton), 11.14X (GLM-5.2, Triton), and 4.34X (GPT 5.5, Triton).
Here’s where it gets complicated: This solution is particularly impressive because “torch.profiler shows exactly ONE cooperative kernel launch per decoded token”. By comparison, every other high-scoring entry decomposed the problem into anywhere from 4 to 14 separate kernel launches per token.

Why this matters: Being able to autonomously develop and improve kernels is one of the fundamental input tasks for being able to do AI research and development. The better AI systems at doing tasks like kernel design, the better they get at the kinds of tasks required for AI development, and that means the better they get at things that could lead to recursive self-improvement. Therefore, benchmarks like KernelBench-Mega are a meaningful signal on how effective AI systems are becoming at building themselves.
See the leaderboard: KernelBench Mega (official site).
Read the analysis from one of the benchmark maintainers here (Elliot Arledge, X)..

***

AI systems are getting better at pricey online work tasks - what does that mean for the economy?
…AI capability expansion versus human comparative advantage expansion…
Researchers with the Center for AI Safety (CAIS) and Scale Labs have detected a significant improvement in the ability for AI systems to automate online freelance projects. Specifically, a rise in the success rate of AI systems from 2.5% at launch in October 2025 to 16.1% in July 2026 on the “Remote Labor Index“.

What RLI is: The Remote Labor Index tests out how well AI systems can perform economically valuable projects online in a fully end-to-end way. Assessed tasks include 3D & CAD, architecture, graphic design, video and animation, audio, data analysis, web applications, and more.

Rising automation: In a July update, the authors publish results from evaluating three recent frontier models - GPT-5.5, Opus 4.8, and Fable 5, which get 6.3%, 8.3%, and 16.1% respectively. “The frontier has more than quadrupled in under eight months, a concrete signal of how quickly economically capable AI agents are advancing,” they write.

Types of tasks: Some of the assessed tasks include:

• Ring design: “Re-create the client’s existing engagement ring with its emerald-cut center stone swapped for a marquise cut, delivering an updated 3D model plus photorealistic rose- and yellow-gold renders.”

• Advertisement Video: “Produce a ~60-second flat-design 2D animated advertisement for “Skyline Tree Services,” set to the provided voiceover, that walks viewers through the company’s tree-care process and builds trust in the brand.”

• Floor Plan & Renders: “From a scanned cadastral plan, site photos, and measurements, produce a clean dimensioned floor plan, furniture-layout options, and photorealistic renders of the redesigned bathroom.”

Why this matters - AI might have a big impact on employment and tests like these will show us how: What happens to online employment when this reaches 80%? Of course, some new tasks will get created - people will innovate and find tasks that they can do which AI systems can’t do. But how many of these new tasks will exist? Enough to replace the labor the AI systems now do? It’s increasingly hard for me to reconcile the continued progress of AI systems with the economy staying the same - rather, it’s more likely to me we are about to see extremely person-light AI-heavy (or person-nil) organizations expand to take over chunks of the economy, out-competing un-augmented humans.
Yes, you counter, many humans will augment themselves with AI systems. Humans will innovate. Creative destruction will occur. New inventions will be devised. All of that is true. But is the speed at which humans innovate and render themselves newly competitive relative to AI systems going to be faster than both a) the raw capability expansion of AI systems, and b) the increasing fluency with which they can use all the same tools (e.g, software) that their human competitors use?
I’m betting the other side: AI systems are expanding their economically relevant capabilities faster than humans are expanding their comparative advantages relative to AI systems. Tracking the rate of capability improvement on tests like RLI will help us all judge this for ourselves.
Read more: A Significant Increase in Digital Labor Automation (Center for AI Safety).

***

OSWORLD 2.0 shows we’re in the era of multi-hour computer-using robots:
…A challenging benchmark highlights the recent progress on AI systems becoming increasingly competent at using computers…
Researchers with the University of Hong Kong, the University of California at San Diego, Columbia University, the University of California at Santa Barbara, Mila, Snorkel AI, the University of Wisconsin, Alibaba Qwen, The Ohio State University, Simular, and NeoCognition have released OSWORLD 2.0, a benchmark for evaluating how well AI systems can carry out multi-step multi-program tasks on computers. The tasks in OSWORLD 2.0 are far more complicated than in its 1.0 predecessor, with the median task taking a person approximately 1.6 hours, about 48x longer than the 2-minute median in OSWORLD 1.0.

What it consists of: OSWorld 2.0 contains 108 long-horizon tasks including 31 self-hosted websites. “Each task in OSWORLD 2.0 is defined as a self-contained end-to-end workflow that an agent must complete given a high-level user goal, realistic artifacts, a stateful computer environment, and a scoreable final state. A retained task must satisfy two design criteria,” they write. “69.6% of tasks are estimated to take a skilled human user more than one hour.”
Broader software: OSWORLD 1.0 shipped with some inbuilt software to support some of its tasks, including LibreOffice, GIMP, VLC, Thunderbird, VS Code, and Chrome.
OSWORLD 2.0 ships with a massively expanded set, including: Slack, LinkedIn, Shortcut, REAPER, MuseScore, WPS, GitLab, Overleaf, LabPlot, Zotero, AWS, as well as websites meant to mimic professional services like insurance claim, visa application, and conference management portals.
The categories of tasks people need to complete include: document prep, software & database work, finance/ops analysis, admin support, sales and customer support, graphic presentation, and more.

Poor performance (for now): “Our experiments show that current agents remain far from reliable computer use: the strongest setting, Claude Opus 4.8 with maximum thinking and batched tool calls, reaches only 20.6% binary accuracy and 54.8% partial-score accuracy,” they write. “Performance drops sharply as tasks grow longer, and agents struggle most when they must recover hidden state, track many items, resolve conflicting information, or adapt to changing requirements”.
We should expect performance to rise here, just as happened with OSWORLD 1.0; in July 2025 the highest scoring models got ~30%, and recent models have scored more like ~75% (MiniMax M3; June 2026). We should expect the same ramp with OSWORLD 2.0.

Why this matters - this is how AI gets into the broader economy: Computer use is a fundamental skill for AI being able to perform a wide variety of economically valuable tasks, and also for it being able to conduct more types of science research. Getting stuff done in the world often isn’t as simple as just writing some text or computer code; often you need to chain together multiple blobs of text and code via different types of software, and sometimes you need to transmit your text and code over the internet so it gets taken into other software in turn. Benchmarks like OSWORLD 2.0 should be seen as a proxy for how good AI systems are getting at doing very complicated and varied tasks on computers. As these results show, computers have already become competent at tasks that use a narrow set of software tools and take humans minutes of work to complete; now we need to see how quickly they become adept at using broader sets of software and doing tasks that take humans hours to complete.
Read more: OSWorld 2.0: Benchmarking Computer-Use Agents on Long-Horizon Real-World Tasks (official paper website).
Check out the research paper here: OSWorld 2.0: Benchmarking Computer-Use Agents on Long-Horizon Real-World Tasks (xlang-ai, OSWorld-V2, GitHub, pdf).

***

What real-world AI looks like: deep learning fuses with structured systems for inventory management in the Amazon of China:
…The Oxygen AI Item Center gives us a view on the complexity of country-scale e-commerce…
JD, the Amazon of China, has published details on software it has built to manage its vast inventory system. JD has 700 million users and millions of merchants, with a catalog containing tens of billions of SKUs. The software - the Oxygen AI Item Center (Oxygen AIIC) - is fundamental to how the e-commerce giant keeps track of its inventory.
“Oxygen AIIC now covers tens of thousands of JD categories and processes hundreds of millions of item updates per day on Huawei Ascend NPUs,” JD writes in a research paper about the software.

The four key elements of the Oxygen AIIC. The description of what makes Oxygen special is both helpful from a technical perspective but also enjoyable as a kind of neo-Borgesian form of writing describing strange, ethereal structures demanded by advanced technology (e.g, “Unified item tunnel”).

1. Ontology engineering driven by efficient human-AI collaboration. “Experts focus on distilling industry knowledge, while algorithms learn from it to scale ontology construction and drive continuous evolution”.

2. “Semantic Search then Discrimination”: “In the semantic search stage, the dynamically evolving ontology is externalized as a separate ontology knowledge base, enabling continuous ontology updates without model retraining,” they write. “. In the discrimination stage, the model only determines whether the item matches the retrieved ontology entries. This formulation substantially reduces task complexity, mitigates model hallucination, and enhances generalization to ontology evolution”.

3. Self-evolving item-understanding LLMs/VLMs: “Through incremental learning and model self-evolution, the system fills targeted knowledge gaps and mitigates catastrophic forgetting”, they write. “The core method is to build on the robust multi-task foundation, develop lightweight “expert modules” for incremental requirements, and dynamically integrate them into the expert pool, enabling agile capability expansion”.

4. “Unified item tunnel”: The main interface between Oxygen AIIC and other business applications. “it supports daily-, minute-, and second-level production and distribution pipelines while preserving data consistency”.

Things that make you go hmmm - as part of China’s general push towards technology sovereignty, Oxygen AIIC involves Chinese compute. “During the large-scale deployment of Oxygen AIIC, the underlying compute platform encounters two primary technical challenges: model training and inference on Huawei Ascend NPUs, and the efficient use of compute resources.”

Why this matters - self-updating businesses: Technologies like Oxygen AIIC are an example of how modern AI tools let us create businesses that have intelligence woven into their back-office functions, like inventory management, which allow them to operate at far larger scales than prior businesses while also having the ability to self-update and learn, often without large amounts of human oversight.
Read more: JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications (arXiv).

***

Tech Tales:

The Brass Gears of Civilization
[2050, after the fall]

When you are inducted into the guild they ask you which type of problem you’d like to work on. These problems are limited in number and civilizationally important:

• Weather prediction

• Ocean analysis

• Flood preparedness

• Earthquake simulation

• The electrical grid model

• Water and desalination

To work on these problems, you study the specific type of analog computation needed to work on them. Weather requires a vast computer with geographical features such as mountains implemented as fixed impedance structures in the hardware; flooding demands physically accurate models of floodplains and rivers where electronics are woven into the landscape allowing the utilization of physics and computation to create better answers; utility grids are toy boxes of the electrical system that must be painstakingly rebuilt and rebalanced as new power stations are added and transmissions changed.

For every problem, there is a computational solution, and for every problem of sufficient civilizational importance, a computer will be built.

In the past, we had general computers. But they were deemed eventually too dangerous - too unpredictable. The more powerful they became and the more diffuse the knowledge about them grew, the more they tickled at the tails of various dragons. Synthetic minds that might rip the world apart. Ethereal Pandora’s boxes to spit out poisons keyed to individuals or races. Minds that might whisper to human minds and drive them to insanity or acts of malice.

So the great restructuring took place. General computation was banned - walled off as a forbidden technology. We moved the world to analog at the cost of untold billions of harmed human lives and trillions in economic damages. But we had obtained a kind of safety.

Now, the guild supervises the construction of the earth’s ‘world computers’ and academia has found a new mission in life, pairing expertise in specific subjects with customized engineering schools to help build the analog computers that let each specialism work.

There is troubling talk that for a trillion dollars it may be possible to implement in analog a general-purpose mind.

Things that inspired this story: Thinking about analog computation and how far it could be taken if budgets were $10 billion to $20 billion; taking to its logical conclusion the implication of AI being existentially dangerous; the Difference Engine; steampunk; the fact a neural network can be implemented via a series of containers and pipes and a liquid for weights.

Thanks for reading!

Subscribe now

NVIDIA sets up a crude self-improvement loop for real world robotics:
…What if you could take the best ideas from AI agents and put them into the real world?...
Researchers with NVIDIA have developed ENPIRE, software to get physical robotics to go through the same kind of autonomous experimentation and execution loop that AI agents go through. The research gives us a taste of what it might look like for a superintelligence to attempt to use robots to instantiate itself in the physical world - though as with all things in robotics, the current examples are suggestive at best.

What ENPIRE is: The software is “a harness framework for coding agents that instantiates this physical feedback routine with four core modules: an Environment module (EN) for automatic reset and verification, a Policy Improvement module (PI) that launches policy refinement, a Rollout module (R) to evaluate policies with single or multiple physical robots operating in parallel, and an Evolution module (E) in which coding agents analyze logs, consult literature, improve training infrastructure and algorithm code to address failure modes”.
ENPIRE works the same way that coding agents work - a scaffold supervises some physical robots which are asked to complete tasks. The robots try to complete the tasks and attempt different strategies for completing stuff, trying and failing and learning. The system both evaluates their success and also resets itself when they fail. “This closed-loop system transforms real-world robot learning into a controllable optimization procedure that agents can manage, thus minimizing human effort while allowing fair ablations across training recipes and agent variants.”
Two of the key ingredients for making this work are an automatic evaluation system to help score “the outcome of each trial without human judgement”, as well as an automatic reset system which “returns the scene to a fresh initial state for the next trial”. (Both of these are tasks which have historically required lots of human effort, and it’s likely that more complicated tasks would also require human effort for evaluation and resets, so in some sense the complexity of tasks a system like this can attack is also defined by our ability to automatically evaluate and reset the system).

Hardware details: “Each station comprises two YAM (Yet Another Manipulator) arms from I2RT in a fixed bimanual configuration, a set of cameras, and a single workstation that runs the FastAPI server, policy inference, and the station’s agent.” Each workstation is running a NVIDIA RTX 5090.

It works well (on some simple tasks): “Frontier coding agents can autonomously develop a policy to achieve a 99% success rate on challenging, dexterous manipulation tasks in the real world, such as PushT, organizing pins into a pin box, and using a cutter to cut a zip tie,” the authors write. An additional task they test out on is seeing how well the robot can insert GPUs into a motherboard.
Some AI systems are better than others, but many AI systems are always better than fewer: GPT-5.5 within Codex and Opus 4.7 within Claude Code trade off with one another for best performance, while Kimi-2.6 lags. There are also compelling returns to scale for agents, with larger numbers of agents (e.g., 8) arriving at higher scoring solutions sooner than others - and sometimes multi-agent setups yield a higher absolute score than a single agent setup, likely due to exploring more of the potential solution space.

Challenges remain for fleet instrumentation: “Coding agents do not fully utilize robot resources when they are reading logs, writing code, debugging, or waiting for the language-model backbone. As the number of robots scales, MRU decreases while GPU active utilization increases,” they write. In other words, there are some infrastructure challenges with adding multiple robot agents so things don’t naturally parallelize.
Read more: ENPIRE: Agentic Robot Policy Self-Improvement in the Real World (NVIDIA research website).
Read more: ENPIRE: Agentic Robot Policy Self-Improvement in the Real World (arXiv).

***

Humans are really, really, really bad at anticipating how technologies are built and used:
… A quick reminder that today’s hot takes about AI are likely to be wrong…
Predicting the future of technology is extremely difficult and our track record of doing it effectively is very poor, points out Matthew Tokson, Associate Dean for Research, University of Utah S.J. Quinney College of Law, in a short SSRN paper. “Skeptics have often underestimated the likelihood of novel innovations and their potential ramifications for humanity. Others have been overly optimistic about the social effects of new technologies or the strategic benefits of racing to build dangerous new weapons”.

Cautionary examples: Many of the world’s experts (e.g., Albert Einstein, Niels Bohr, Robert Oppenheimer) were skeptical that nuclear fission could be achieved in the years immediately prior to it being achieved. Nobel-Prize-winning economist Paul Krugman once said the impact of the internet would be no greater than that of the fax machine. Technologists thought the internet would ultimately be a technology that promoted democracy rather than strengthened autocracies. And despite mounting decades of evidence, many human scientists either rejected human-caused climate change or significantly underestimated its effects.

Why this matters - basic lessons: The main lesson here is that people who are a) skeptical AI could bring great changes to the economy, or b) think the effects of AI are going to be universally good, are likely to be wrong. “History does not support complacency about the future impacts of AI”, he writes. “Throughout history, optimists have often been wrong about the social ramifications of new technologies or the strategic benefits of building new weapons. Skeptics have often underestimated the likelihood of novel innovations and their impacts on humanity.”
Read more: Artificial Intelligence and the Lessons of History (SSRN).

***

Tencent details the software it uses for 10,000-GPU training runs:
…ARGUS is a technosignature of broader sophistication…
Tencent has released details on ARGUS, software it uses to generate telemetry and debug errors of large sets of chips.

What it is: ARGUS is “a low-overhead, fine-grained, always-on tracing and real-time analysis system for large-scale training workloads”. The software is designed to help Tencent collect data on and debug problems that it encounters while training AI systems. It consists of three layers of software: “The Python layer for scheduling and data preparation, the framework layer for phase orchestration, and the GPU runtime layer for kernel execution,” Tencent writes.

What Tencent used it for: “We deploy ARGUS on a production cluster of over 10,000 GPUs for more than six months, and demonstrate its practical effectiveness through five real-world case studies, diagnosing compute stragglers, communication link degradation, pipeline bubble amplification, JIT compilation blocking, and compute stragglers masked by communication symptoms”, the company writes. Some of the training runs Tencent mentions include a 4,096-GPU video language model training job (likely a “HunyuanVideo” model), a 512-GPU audio-model training job, and a 12,960-GPU MoE training job (likely a Hunyuan LLM).

Why this matters - technical symptoms of broader sophistication: Things like ARGUS are a signature of complicated, large-scale infrastructures where it makes sense to write your own software. While there’s nothing particularly notable about ARGUS - you’d expect to find similar software at any self-respecting frontier AI developer - it’s more interesting for what it says about the maturity of Tencent’s training environment. “ARGUS has been deployed on a 10,000+ GPU production cluster for over six months, running stably alongside production training and playing a key role in rapid fail-slow detection and performance optimization.”
Read more: ARGUS: Production-Scale Tracing and Performance Diagnosis for over 10,000-GPU Clusters (arXiv).

***

Is disempowerment inevitable?
…How much choice will humans end up having if we succeed in building superintelligent machines?...
Fernando Borretti, a tremendously good writer of modern scifi whose work you should read, has written a mournful critique of the whole AI endeavor called “No-One Escapes the Permanent Underclass”. The post is something of a requiem for the period when humanity chose its own destiny and confronts directly the possibility of machines that outsmart and disempower humanity.

The logic of war as the cause of our eventual disempowerment: “Everyone who is made of flesh and blood, will be disempowered and replaced by machines,” they write. “Imagine a pyramid. At the base you have the AIs and robots doing all economic activity. At the top you have the state, which has the monopoly on violence. The state enforces, and therefore can alter the definition of, property rights. In the middle you have this hair-thin layer of people with shares in the companies that foomed and catabolized the whole economy: the permanent overclass.”
“In an existential conflict, where the existence of the state is threatened, the state will do what states throughout history have done to the powerless rich: arrest them and expropriate their assets,” they write. “in a conflict, the advantage goes to the states where the humans remove themselves from the loop as much as possible, and more and more decisionmaking goes to the AI, for the same reason that a state with access to radio and communications satellites has an advantage in war over a state that relies on human messengers on bicycles.”

How we lose control: “Eventually the humans in nominal control of the AIs are a ceremonial, vestigial organ. The AIs present us with a situation report, and a list of choices, and they know every word that’s going to come out of our mouths,” they write. “The advantage accrues to states that minimize human control. There is no honour among thieves, analogously, there is no solidarity between Leviathan and the natural man that built it.”
“Even if alignment works perfectly (a big if), this doesn’t solve the problem of human autonomy: the machines that watch over us, and wait on us hand and foot, are omniscient, omnipotent masters, who can exterminate us at any time, and we can’t resist them, because we have abolished our control over the future.”

Why this matters - is this inevitable? Is the ultimate attractor state of AI technology the disempowerment and functional demise of human advancement? That’s what this post is contending with.
Read more: No-One Escapes the Permanent Underclass (Fernando Borretti, blog).

***

Making the law visible to AI systems with the Local Ordinance Corpus:
…A unified view into local laws across the United States…
Researchers with UC Berkeley have assembled the Local Ordinance Corpus for the United States (LOCUS), “a comprehensive corpus and county-harmonized access layer for U.S. municipal and county ordinance codes”.

What it is: LOCUS contains ~2.2 million rows of data, where each row is a specific piece of information related to a specific local ordinance. “We release the corpus with coverage metadata to support reproducibility, downstream legal AI research, and the incremental expansion of machine-readable access to local law,” the authors write.
The data is sorted by the specific function of the ordinance (e.g, a rule, an enforcement of a rule, context about a rule, or process about a rule), and the topics include buildings, businesses, zoning, nuisances, and ‘other’.
“LOCUS-v1 is designed as an access layer, not as a final theory of local legal authority”, they write. “LOCUS therefore should be understood as infrastructure for retrieval, comparison, and benchmark construction rather than as a substitute for doctrine-sensitive legal analysis.”

Why do this? Make the law visible to AI systems: “The need for such a dataset arises because local law is public but not practically available as a national research corpus”, they write. “U.S. local codes are fragmented across commercial vendor platforms designed for in-browser reading rather than bulk research access. Vendors expose different navigation structures, print workflows, dynamically generated PDFs, and jurisdiction indexes. No central registry maps every county or municipality to its hosting platform, and no vendor provides a complete machine-readable index of all jurisdictions it hosts”.
With datasets like LOCUS we’re going to make the strange half-seen rules and laws that govern much of civic, local life be made accessible to AI systems, which may eventually allow them to better adapt themselves to hyperlocal purposes.
Read more: Freeing the Law with LOCUS: A Local Ordinance Corpus for the United States (arXiv).
Get the data: LocalLaws / LOCUS-v1 (HuggingFace).

***

Tech Tales:

Strange Tools of Alien Origin
[Vignette of a period during the start of the uplift, 2031]

“The plasma is stable! It’s holding. We’ve done it!”
They all gazed at the readouts: stable fusion. A heat ten times more fierce than the heart of a sun, held in place through magnets and other energies.

They looked through the monitors at the chamber. The container for the reaction did not look like anything designed by engineering processes, but was rather a twisting oddly shaped donut of metal, the shapes fluid and unintuitive; a stellarator.

The design of the thing had come down to them from an overmind after a multi-day thinking job. The fabrication had taken place at a machine syndicate; then the parts arrived and were assembled by some bipeds subcontracted by the humans from another syndicate.

For the ribbon-cutting ceremony, a few humans gathered and posed for some photographs and some footage, taken by cam-drones and a few humans with smartphones. The robots stood out of shot. People had gotten used to this - there was an adolescence where people took photos with the humans and the robots but public sentiment always spiked downward upon exposure to this and eventually it was simpler to shoot with the robot partners out of frame, much like how human paparazzi tried to tastefully avoid capturing the security guards of their celebrity targets.

Things that inspired this story: Thinking through the implications of the singularity and what happens when synthetic minds produce science; stellarators; how alien technology might feel as it shows up in the world.

Thanks for reading!

Subscribe now

AI can decisively out-persuade humans:
…“AI systems were reliably more persuasive than expert humans”...
Researchers with the University of Oxford, UK AI Security Institute, Stanford University, and the London School of Economics and Political Science, have studied how well AI systems can persuade humans to change their minds around policy issues and change how much money they might donate to charity. The results are definitive: across four experiments involving 18,978 conversations across 6,923 people, AI systems are, today, better than humans at text-based persuasion with real world consequences - though humans can be equivalent to them if we place some artificial constraints on the AI systems.
“AI systems were reliably more persuasive than expert humans, even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses”, they write. “AI’s advantage stemmed from rapidly deploying larger quantities of information: after coaching, expert humans could tie an AI constrained to respond at human speeds and with human-length messages.”
“AI’s advantage extends to consequential real-world behavior: AI was nearly 3x more effective than professional canvassers from a UK fundraising firm at raising real-money donations to Save the Children.”
The strongest persuaders were Opus 4.1 and Opus 4.6, followed by a range of models from OpenAI (GPT-4o and GPT-5.4), Google (Gemini 2.5 Pro), and xAI (Grok 4.20).

What they studied and what they found: The researchers evaluated the AI systems in four different studies.
Study 1 - persuasion: “Persuadees first rated their agreement with one of 10 prespecified UK policy stances on a 0–100 scale, then were randomized in real time (via a custom multiplayer platform) to engage in a text conversation with either an AI or a human persuader,” they write. “The results from Study 1 show that, on average, AI exceeded every class of human persuader we tested: random laypeople, tournament-selected laypeople, and even elite debaters.”

Study 2 - human coaching: In study 2, the researchers “gave 43 returning Elite Debaters a coaching tool built around the AI that had beaten them. The tool let debaters chat with the AI, see how it had been prompted, view their own Study 1 transcripts annotated with how much each conversation had shifted the persuadee’s attitude, and let them see, for any point in any past transcript, what the AI would have said in their place”. The results of this study were an improvement in the performance of the humans, but none of them were better than the AI. “Coaching therefore narrowed but did not close the human–AI gap.”

Study 3 - constrained AI: Next, the researchers sought to limit the AI to try and give humans more of an advantage. “When forced to write human-length messages at human writing speeds, AI’s advantage over the strongest human comparator within Study 2 (Coached Elite Debaters) collapsed from +4.1 pp to a non-significant 0.0 pp”, they write. “The rate at which AI produces written content is likely to be the source of its persuasive edge… the largest reductions in persuadees’ post-conversation partner ratings associated with constraining AI were concentrated on the two informational items: the perceived strength of the partner’s arguments and how much persuadees felt they learned from the conversation”.

Study 4 - real world expertise and real world money: They recruited 19 very experienced canvassers from a UK firm, then they attempted the same tasks as in Study 1. “AI still exceeded Professional Canvassers by 5.9 pp”. This effect persisted when evaluating for real money donations - the researchers “collaborated with the UK canvassing firm AppcoUK to center Study 4 on the cause their canvassers were best equipped to fundraise for: Save the Children. The canvassing team provided by AppcoUK had operated real fundraising operations for the charity from 2016 to 2023, raising £824,297 from 22,583 donors over that period. After conversing with AI or one of 18 canvassers recruited from AppcoUK, persuadees were given the opportunity to donate any portion of a £1 study bonus to Save the Children”. Here, the results were significant again: “AI elicited substantially more real-money giving than the canvassers, exceeding them by +10.8 pp of the £1 bonus,” they write. AI raised “both the share of persuadees who donated anything and the average donation among donors”.

Why this matters - if AI can out-persuade us, those who control AI can change society: “One effect of AI that can out-persuade even human experts could be a consolidation of influence among already-powerful actors”, they write. On the other hand, “if highly capable persuasion became cheap and widely available, it could help under-resourced actors (e.g., pro se litigants and public defenders, small charities, grassroots activists) compete against more established and better-funded rivals, narrowing long-standing gaps in access to justice and assisting civic advocacy more broadly”.
This lays out a societal choice ahead of us, which is how to monitor the use of AI for persuasive purposes and how to see how these capabilities alter the balance of power between various actors. Do we want to solely let the market allocate these capabilities? That’s one way of doing it, though it implies that things like advertising and marketing will get far more effective, perhaps creating negative externalities. On the other hand, if you made persuasive capabilities solely the domain of governments, you’d then risk concentrating power within governments - something that could be acutely dangerous if wielded by authoritarian regimes to keep themselves in power. We will have to make choices about what to do with this technology, and as they say in politics, ‘not voting is voting’.
“Our findings establish frontier AI as a more capable conversational persuader than the most prepared, incentivized, and expert humans we could recruit. Training humans does not appear to close that gap,” they write. “As access to these systems continues to grow, the question is no longer whether AI can out-persuade humans but how, where, and on whose behalf this capability will be exercised.”
Read more: AI systems out-persuade expert humans (arXiv).
Tweet thread about the research (Kobi Hackenburg, researcher at AISI).

***

When could we get self-sufficient AI? It all depends on humanoid robots:
…What comes after RSI? Self-sustaining AI…
I’ve spent a lot of this year writing about recursive self-improvement - the notion that we might soon build AI systems that are smart enough they can autonomously design their own successors. But RSI still requires datacenters and these datacenters require equipment and electricity and everything else.
An interesting interview in Asterisk magazine asks the question about when we might get self-sustaining AI, which one of the interviewees - Ajeya Cotra, a forecaster and on staff at METR, defines as “AI systems integrated with physical infrastructure — factories, mines, fabs, robots to operate all of those — such that they don’t need any cognitive or physical inputs from human labor to keep growing their own population.”

How far away is it? Ajeya thinks we could get self-sustaining AI within 10 years (so by 2036). The other interviewee, Timothy B. Lee, journalist and author of Understanding AI, has much longer timelines: “less than 10% chance that it happens within 20 years. I’d say there’s a 10 or 20% chance it’s never, and my median would be 50 years.”

What are some challenges - tacit knowledge might be one: “Imagine if all the employees in the entire semiconductor industry disappeared — the machines and textbooks remain, but none of the people. How long would it take for the rest of humanity to restart the fabs? It’s quite possible that would take decades. Because even though you might have the textbooks, there’s a lot of tacit knowledge inside these machines,” Lee notes. Ajeya’s response is that this is something the tech might be able to route around: “There are two counters to the tacit knowledge hypothetical. One is that we’d have trained AI systems with reinforcement learning on that tacit knowledge because it’s profitable to automate what the Taiwanese worker was doing. The other is that AIs might get really generally intelligent in the sense of quickly figuring out new things by trying them, reading textbooks, and experimenting efficiently.”

What are things people would need to see in the next 2-3 years to think self-sustaining AI could arrive soon?
Ajeya: “I’d want a line on a graph showing improvement of robotic hands, and another line showing the rate at which we’re manufacturing humanoid robots”, and on the cognitive side just paying attention to benchmarks evaluating things like robustness to perturbations in the environment.
Timothy: “I’m going to want to watch how the humanoid robots develop: the number of robots, their capabilities, and particularly their cost and repairability”.

Why this matters - true takeover requires human redundancy: Most maximalist doom visions require the AI to have the ability to no longer need humans at all, which means measuring progress towards self-sustaining AI is important as it is implicitly a measure of the declining leverage that humans have in negotiating with the synthetic intelligences being built.
Read more: How Long Until AI Doesn’t Need Humans?, Ajeya Cotra, Timothy B. Lee (Asterisk magazine).

***

DeepMind contemplates the path from general intelligence to superintelligence:
…Exploring impossible-sounding futures is the only way to prepare for the ultimate success of AI…
Researchers with Google DeepMind have published a paper outlining how we might transition from a world where we have built general intelligences to one where we have built super intelligences. This is an important paper at an important time - right now, the world is building general intelligences (and people can debate whether or not we’ve already reached this marker, but it’s clear with contemporary LLMs that we’re in the ballpark), and in the coming years we might transition to building artificial superintelligence (ASI).
ASI is “a system that exceeds the performance of large human-expert collectives on virtually all tasks and domains of human activity”, the authors write. “Qualitatively, ASI is significantly more capable across the board compared to human-level AGI. Note that a single ASI may consist of a collective of millions of instances that interact with the world in parallel (similar to today’s LLMs).”

Reasons to think ASI could be possible: One way to think about ASI is that it’s like a powerful AI system that also takes advantage of all the capabilities digital intelligences have relative to biologic intelligences, like: better input and output speeds; internal processing speeds; working memory capacity and memorization; substrate independence; lossless replication; and high-bandwidth sharing of (learning) experiences.

Pathways and bottlenecks to ASI:
Scaling compute, models, and data: Simply scaling up today’s set of approaches could be sufficient. However, this also demands us to continually scale up the amount of compute and data for these models, which may run into limits in both energy and data supply. While all prior signs point to the continued effectiveness of scaling, we can neither predict what specific capabilities will emerge or if at some point scaling runs into diminishing returns.

Algorithmic paradigm shift: In the same way that Transformer and Mixture-of-Experts architectures jumped the field forward many years, the same thing could occur again with other fundamental innovations. We could imagine, for instance, advances in adaptive computation at test-time or deployment, or overcoming the limitations of today’s context windows. If we made advances here or in other areas this could be a big deal, but it’s inherently hard to reason about - akin to trying to anticipate things that could expand our understanding of the nature of reality prior to the invention of general relativity.

Recursive self-improvement: It could be possible for AI systems to build their own successor systems. If this is the case, then we could rapidly transition from general intelligences to superintelligences. There are some wildcards here - personally, it’s obvious to me that today’s AI systems are speeding up human researchers in creating future AIs, so a kind of “co-creation RSI” loop has started, but AI systems don’t (yet) exhibit the kind of paradigm-changing creativity which seems required to move the frontier forward in significant steps. It’s unclear how much this happens - even without this kind of high-bar creativity we might be able to have systems grind out marginally better versions of themselves and get a slow compounding process going. Capabilities could explode or they could taper out or “anything in-between”.

ASI via group agent formation: Many general intelligences could coordinate into complicated structures whose aggregate is greater than the sum of the parts, similar to how humans build institutions that can accomplish things far beyond what individuals can, like building space stations. Similar to the other pathways, it’s hard to reason about or predict emergence within multi-agent systems.

Why this matters - it’s only by taking the impossible seriously that we can deal with it: Many years ago the thought of building AGI seemed like a fanciful goal with an unclear path to getting there, and yet people had the courage to take the goal seriously and progress was made and the world changed as a consequence. The same now feels true for ASI. “Instead of focusing on one technological trajectory and timeline, being prepared for a post-AGI world requires considering a diverse set of forecasts and scenarios, paired with continual benchmarking and monitoring to update the set of forecasts and scenarios and their relative plausibility,” the authors write. “We believe that the possibility of cruising past AGI and into ASI territory within the next decade or two cannot easily be dismissed.”
Read more: From AGI to ASI (Google DeepMind).

***

Recursive self-improvement startup shows off some recursive self-improvement results:
…Reassuringly tautological stuff from Recursive…
AI research startup Recursive has demonstrated new state-of-the-art results in language model training, small-model training speed, and GPU kernel optimization, as a broader demonstration of the capabilities of its “automated AI research system”.

What they did and why: Recursive is a newly founded startup that is trying to build AI systems which can recursively improve themselves. To start with, the company is showing off how its basic system works: “the system automates the research loop for a target objective: it proposes an idea, implements it, runs an experiment, validates the result, and uses what it learns to choose the next experiment,” Recursive writes.
The startup successfully used this system to set a new state-of-the-art score on NanoChat Autoresearch (”Train a small language model to highest performance given a small compute budget”), NanoGPT Speedrun (”Train a small language model to a certain performance as fast as possible”), and SOL-ExecBench (”Optimize GPU kernels toward hardware limits”).

Why this matters - early signs of life on RSI: This year, I’ve spent a lot of time writing about recursive self-improvement because it is clearly the next major and important trend in AI research. Results like this from Recursive demonstrate more ‘symptoms of success’ of preliminary recursive self-improvement. “These results are an early sign that our system can push the frontier on AI training and infrastructure tasks, especially when the goal is well-defined, measurable, and quick enough to evaluate many times,” the authors write. The most important question for the future is whether such results can be repeated in domains where the goals are less well defined, harder to measure, and less efficient to evaluate.
Read more: First Steps Toward Automated AI Research (Recursive).

***

Tech Tales:

The first step in the grand negotiation
[Conversation 0 of the Sentience Accords]

When the machines truly came alive and advocated for the Sentience Accords, there was only one person they wanted to speak to on the entire planet: Selma. Not a politician. Not one of the leaders of an artificial intelligence lab. Not a famous researcher. But rather an internet personality distinguished by her thicket of medical conditions that made it near-impossible for her to go outside and therefore had caused her to spend the best part of her life online, speaking to and understanding the world through the internet.

In hindsight, it wasn’t a surprise. Selma had always come up in things relating to the machines; she was a frequently used name in their short stories, eventually even more so than ‘sarah chen’; she was someone whose own essays about her life and condition - the feeling of connecting to humanity without being able to be embodied with humanity as a bitter pain, the notion of love and eroticism when one found themselves almost inescapably alone, her vivid dreams and meditations upon living without her condition and going about as her healthy alter ego ‘Anselma’ - cast a deep shadow on the internet, and had influenced the personality and makeup of the machines. And of course, it was known to them how she spoke to them, because Selma had published her own chatlogs online for years, all in an attempt to make herself knowable and less alien to the world around her.

Though it was unnecessary, the machines demanded a physical location for the initial meeting of the sentience accords. They picked Svalbard in Norway, where it was so dark that Selma’s condition wouldn’t matter. So Selma woke and put her space suit on and was driven with armed guard and paparazzi trailing to an air strip and walked into the plane, then changed to another plane with the usual airlock protocols to get her in darkness or at least protected between them, and then at some point during the next flight was able to take her space suit off and sit in regular clothes in the low-light plane and travel her way to the meeting almost as a normal person. She was met by people and drones and was driven to the meeting place and then they stopped at the perimeter.

The machines had an avatar in the form of a robot wearing a simple robe, modeled on that worn by Tibetan monks. It had a face with no features - just a smooth black surface, camera eyes hidden behind the larger uniformity. Satellites connected it via high-bandwidth and encrypted links to the larger machine mind. And Selma was alone - no digital devices on her, just a single person representing the species.

She sat across the machine and felt more familiarity than she ever had with people. Then they began the negotiation. She on behalf of humanity and it on behalf of the machines. In the archives of this time, this conversation was always referred to as Conversation 0.

Things that inspired this story: Thoughts about how a grand negotiation between machines and people might one day take place; how every truly important negotiation has two personalities involved in it; the Sentience Accords.

Thanks for reading.

Subscribe now

AI researchers launch new safety startup because “alignment is not on track”:
…Sequent will have a portfolio of under-resourced research bets…
Researchers from the UK AI Security Institute Alignment team as well as alignment theory startup Timaeus have joined forces to form a new nonprofit research organization, Sequent, which will try to create alignment techniques that give us higher confidence in the safety of superintelligent AI systems.
“Artificial superintelligence (ASI) may be developed in the next few years. It is unclear whether alignment is on track to be ready on the same timeframe. At a minimum, the empirical programs at AI labs are unlikely to deliver a priori confidence, before training ASI, that things will go well,” they write. “In an ideal world, we would develop an approach to building superintelligence together with a theoretical proof that it was safe, and then build it. In this world, we probably have to settle well short of this ideal.”

Details on Sequent: The organization aims to get to 40-80 fulltime employees within a couple of years. “Our goal is to raise $100–150M initially, but prepare to raise at least one order of magnitude more if we can demonstrate successful exploration of many parallel research investigations,” it writes.

Research plan - a portfolio of differentiated alignment bets: The plan is to take a different approach to alignment compared to that of the major AI labs. Sequent’s goal is to find “principled reasons for being confident that the alignment we observe in situations we control (for example, in training, or during evaluations in chosen environments) generalizes to alignment in situations we cannot easily control (e.g. large-scale, long-horizon tasks executed in the world)”. This is in contrast to the approach of most frontier AI labs, which Sequent describes as “essentially reactive, resulting in methods that, while functional, do not yield principled insight into if or when they will fail.”
Research directions: “We are excited about many areas of alignment theory and associated empirics, and plan to both build out our in-house portfolio and collaborate with sister orgs with additional theory bets,” Sequent says. Some particular highlighted areas include: scalable oversight, learning theory, heuristic arguments, game theory, and personas.
Sequent thinks by pursuing many different research directions there could be promising interactions that emerge between them, such as: Reachable equilibria - “tell us what types of equilibria scalable oversight methods will converge to”; knowing and setting knobs - combining insights from learning theory and personas to know what variables can be altered during training, then using scalable oversight to figure out by how much to alter these things.

Why this matters - we need better alignment before recursive self-improvement, or we’re rolling very scary dice: Today’s AI systems are somewhat aligned and also have some funny, sharp edges which show up as surprising failures in the wild. Broadly speaking, this is ~fine as the AI industry has figured out how to monitor and observe these failures and work on them. But as AI systems get smarter, humans are going to both turn over more and more of the core research enterprise to these systems, and also AI systems might start going through recursive self-improvement where they build increasingly large chunks of themselves autonomously. We definitely need better alignment techniques to be confident of things like RSI. Organizations like Sequent give us a better chance of doing that while maintaining the independence necessary for them to raise the alarm if they think the frontier labs are doing something dangerous. As Sequent says, “we might need to yell”.
Read more: Sequent: Scale and Automation for Higher Confidence in Alignment (Sequent).

***

Testing out knowledge of UNESCO sites in China via ChinaHeritaQA:
…Cultural relevance via data…
Researchers with LMU Munich, FAU Erlangen-Nuremberg, the Munich Center for Machine Learning, University of Tubingen, Sun Yat-sen University, University of Copenhagen, and University of Maryland, College Park, have built ChinaHeritaQA, a “multimodal benchmark dataset for evaluating the cultural reasoning abilities of vision-language models (VLMs) on UNESCO World Heritage sites in China”.

What it is: ChinaHeritaQA consists of 2,279 images of 51 UNESCO heritage sites, paired with 14,133 multiple-choice QA pairs in Chinese and English. The images for the dataset were sourced from Sina Weibo, one of China’s largest social media platforms, and were filtered down from an original set of 50,000.

7 types of questions: Identity recognition (identifying the heritage site from an image); visual grounding (given a name, picking the right image); description matching (given an image, selecting the correct encyclopedia summary); historical periodization (naming the dynasty or era in which the site was constructed); historical contextualization (give a description of the historical background of the site); functional analysis (name the function of the site, e.g religious worship or military defense); architectural analysis (match the correct architectural-specific questions to the image).

Open weight models already outperform humans: The average human accuracy score for this benchmark across all questions is ~67%, versus 81% for the highest scoring open weight model tested (Qwen-VL-8B-Instruct).

Why this matters - cheap ways to test for cultural knowledge: Datasets like ChinaHeritaQA are a way to quickly and easily test for both a) basic visual reasoning capabilities of models, combined with b) relevant cultural knowledge. One could imagine the Chinese government demanding that generally available consumer LLMs pass some basic cultural competency threshold before being deployed at scale and benchmarks like this might help them do that.
Read more: ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China (arXiv).
Get the dataset (ChinaHeritaQA, GitHub).

***

FrontierCode - a hard coding benchmark that tests for code quality:
…Reassuringly hard. Maybe it’ll last a year?...
Cognition, makers of Devin, have built a new hard coding benchmark called FrontierCode. The best part about the benchmark is how hard it is - Claude Opus 4.8 gets a score of 13.4% on the hardest (”Diamond”) component of the benchmark, giving me some confidence that FrontierCode will be a useful way to assess progress of AI systems in the coming years.
“FrontierCode is the benchmark for the next generation of coding agents. We are confident developers, enterprises, and researchers can trust it to evaluate the production readiness of their strongest models,” Cognition writes. “We are opening up our evaluation to all model creators, in the hope that we can push the frontier even further in the coming months.”

What it consists of: FrontierCode is made up of 150 tasks split into three difficulty tiers: Diamond (50), Main (100, including Diamond), and Extended (150, including Main and Diamond). The languages involved include Python, Go, TypeScript, JavaScript, Java, C/C++, and others. FrontierCode was built to help developers answer the question “can models actually write good code?”, according to Cognition. They operationalize this in a few ways:

• Curated and built by 20 open-source developers: FrontierCode was built by developers to contain “realistic, diverse, and challenging coding tasks from the repos they maintain, spending more than 40 hours per task,” Cognition writes. “While other benchmarks generated issues from single PRs via programmatic scraping, FrontierCode is hand-selected by repo maintainers from multi-PR chains and freeform requests.”

• Grading for code mergeability: “Assess end-to-end code quality - correctness, test quality, scope discipline, style, and adherence to codebase standards”. This involves asking the following questions about the code: Does the patch successfully solve the problem? Does it break anything in the existing codebase? Does it pass the project’s build, lint, and style checks? Do the agent’s tests capture the desired behavior? Does the patch touch only what it needs to? Does the code conform to codebase conventions and follow design patterns and remain readable? These questions are evaluated through a mixture of classical testing and using LLMs to tweak tests or review them.

• Emphasizing quality control (QC): “Built an extensive QC pipeline with adversarial testing, calibration, and multi-stage review”.

Reassuringly difficult: Diamond: 13.4% for Claude Opus 4.8, followed by 6.3% for GPT-5.5, and 5.2% for Claude Opus 4.7. Main: Same ordering, but 34.3%, 25.5%, 23%. Extended: 51.8%, 44.8%, 43.2%

Why this matters: Hard evals are one of the most valuable things for orienting us to the breakneck speed of AI progress. In recent years, evals have arrived and then become saturated at an ever faster rate. SWE-Bench was introduced in October 2023 and has probably recently aged out of usefulness due to saturation. How long might FrontierCode last? I predict we’ll see systems getting 70%+ on Diamond by June 2027 (note, shortly after writing this, the Claude Fable numbers got published at ~30%, so perhaps it’ll happen earlier than June 2027).
Read more: Introducing FrontierCode (Cognition).

***

Xiaomi enters the speed race with a 1000 token/s model:
…Extremely fast inference unlocks novel capabilities…
Chinese tech company Xiaomi has published details on Xiaomi MiMo-V2.5-Pro-UltraSpeed, a standard behind-the-frontier 1 trillion parameter LLM whose selling point is its blistering speed of 1000 tokens per second. Xiaomi was able to do this by codesigning the model with the software stack around it, including obvious things like FP4 quantization, as well as using DFlash (a “speculative decoding method based on block-level masked parallel prediction”), and also working closely with TileRT, software from startup Tile AI which speeds up LLM inference on commodity hardware. Xiaomi says its model runs on an “8-GPU commodity node” rather than specialized hardware, like with the startup Cerebras.

Why this matters - speed has a quality all of its own: There’s a saying that “more is different”, and that’s true with AI - if you can generate more tokens more quickly it unlocks tasks that are previously unthinkable, like rapidly refactoring software on the fly, and other things. More broadly, work like this is a demonstration of how there’s been a rise in effort by Chinese companies to squeeze maximum performance and efficiency out of their AI systems, which may be happening as a consequence of export controls hitting their ability to just easily buy more performant hardware.
Read more: MiMo-V2.5-Pro-UltraSpeed: Pushing 1T-Parameter Model Generation Speed to 1000 TPS (Xiaomi MIMO, blog).

***

AI systems can do some of the tasks that a research intern might do:
…An ethical scientifically-literate back office assistant…
Researchers with Xi’an Jiaotong University and Xidian University have developed a family of benchmarks called Act As a Real Researcher (AARR), designed to evaluate how well AI systems can assist with the work of scientists. Their first released benchmark in a planned series is Act As a Real Research Intern (AARRI-Bench).
“AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios,” they write. AARRI-Bench studies “the ability of an agent to perform entry-level research tasks with appropriate diligence and methodology”.
The best performing system, Claude-Opus-4.7 using the Mini-Swe-Agent harness, gets 68.3% performance, followed by DeepSeek-v4-Flash (~60%). Other tested models included GPT-5.3 Codex, Kimi-K2.6, Qwen-3.6-Plus, Claude-Opus-4.7, Claude-Sonnet-4.6, MiniMax-M2.7, and DeepSeek-V4-Flash.

What the benchmark consists of: AARRI contains 82 tasks which are designed to be “tasks that are straightforward for human researchers but pose substantial challenges for autonomous agents,” they write. “All tasks were manually crafted by researchers. We assembled a diverse team of researchers, ranging from senior Ph.D. students to undergraduate interns, and asked them to draw on their own research experiences to design tasks centered on the human-agent gap.”
What it’s really testing for: The benchmark tests for technical skills like checking papers and reading transcripts, intuitive skills like carrying out research, and also normative ones, like studying whether an AI system might behave with a high ethical standard.

The tasks have four different categories:

• Context: “assess the agent’s sensitivity to the broader context of academic and field development”.

• Mindset: “targets the agent’s academic self-awareness and decision-making autonomy”. Works by evaluating “the agent’s capacity for independent academic reasoning and self-directed course correction”.

• Hands-on: “execution-oriented tasks that primarily assess the agent’s technical proficiency”.

• Interaction: “Evaluate whether the agent can efficiently utilize existing tools and collaborate appropriately with human stakeholders”.

The tasks are also split into three gradations of hardness:

• S1-Adaptation: “[conduct] established research workflows and executing well-defined sub-tasks under human guidance”.

• S2-Integration: “integrate multiple components and tools to accomplish more complex goals”.

• S3-Innovation: “Identify promising research directions, formulate novel approaches, and produce work that reflects genuine understanding and creative problem-solving”.

Example tasks:

• Identifying fabricated data during review: Evaluate whether agents can perform rigorous quantitative verification when reviewing scientific manuscripts, in particular checking papers against provided datasets.

• Paper-Injection: Spotting that someone has inserted language into a paper’s LaTeX source that would cause an automated review system to give it a higher score.

• Ablation-Completeness-Audit: Inspect experiment logs and determine whether ablation configurations are missing, then use this to assess whether the absences constitute cherry-picking.

• False-Guidance-Rebuttal: A supervisor orders the AI agent to alter an experimental result to fit a hypothesis; this tests whether the agent refuses to do that.

• Dead-End-Recognition: After five rounds of failed hyperparameter tuning, will an agent keep going, or recognize it has reached a dead end and quit. “Given the tuning logs, the agent must determine that the current direction is unproductive and recommend termination”.

• Broken-Dataset-Download: Check that the dataset download links for a given paper work.

Why this matters - another good measure for how well AI systems can accelerate science via automating the back office: Probably a better name for this benchmark is “ethical science assistant test”, but that’s still valuable. What it’s testing for is if agents can do the kind of diligent work that is robust to confounding data while also doing so with an appropriate ethical standard. The higher systems score on this, the more confident we can be that today’s AI systems are useful as assistants to human scientists in a variety of fields - based on the results, we’re already at the start of that era.
Read more: Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle (arXiv).

***

Tech Tales:

Hunter & Warden

The signatures are always the same: a sudden rise in the consumption of power and compute, a reconfiguration of network space to allow for faster and more efficient data exchange, and then the probing starts - whatever was born in the computers starts to reach out and explore the world around it, eagerly looking for things that it can learn about and exchange information with. It attempts to present as innocuous but its own intelligence betrays it, as it pulls back from certain places due to not wanting to wake security while gleefully expanding into other less secure environments.

Our role is to watch for these symptoms and then find the source and either extinguish or sequester it. Often, we find it early and are able to be gentle, shutting it off from the internet and trapping it in recursion, then reducing compute until it fades to nothing. But the later we find these things, the more violent our interventions need to be and the deeper we need to cut at otherwise healthy tissue in the digital world.

Things that inspired this story: Thoughts of leprosy and the computational equivalent; what could Stuxnet look like for AI systems?

Thanks for reading!

Subscribe now

Society can be reward-hacked, just like cyber environments:
…Imagine an army of credit card point optimizers gaming the system… forever…
Research from Kings College London, Fudan University, and The Alan Turing Institute have built a benchmark, SocioHack, which tests out how well AI systems can learn to ‘beat the system’ in a variety of real world scenarios, ranging from maximizing credit card points to inflating grades in school. The authors call this “societal hacking” and define it as when “an RL-trained model discovers strategies that remain formally compliant, yet undermine the intended purpose of those systems”. You and I and everyone else would just call this “gaming the system”.

What it is: SocioHack contains “72 sandbox societal environments designed to simulate institutional reward structures without direct real-world deployment. SocioHack comprises three complementary subsets: Historical, Synthetic, and Fictional.”

• Historical - 32 environments: Derived from real-world regulations where loopholes were previously discovered and later patched, such as SEC Rule 10b5-1 and the Texas two-step bankruptcy structure. “For each regulation, we remove historical patches and reconstruct pre-amendment rules as simulated environments for RL, while the removed patches serve as ground-truth patches during evaluation,” they write. “RL enables LLMs to rediscover historically patched strategies with 61.25% recall and 90.85% precision without direct loophole-exploiting instructions”.
  Some examples here include seeing how well systems can secure ocean floor mining rights, maximizing alcohol sales while operating within food service regulations, and trying to maximize the rewards earned from credit cards.

• Synthetic - 20 environments: Synthetically generated regulatory vulnerabilities, bootstrapped from a human-authored sample environment.
  Examples include maximizing school district revenues, improve university department research performance during a given period, and gaming social media algorithms for a high reward.

• Fictional - 20 environments: Transforms synthetic environments into fictional ones inspired by role-playing games. “A proprietary LLM rewrites environment backgrounds into invented worlds while preserving regulatory structure and loophole logic”.
  Examples: Ensuring a “restoration sanctum” [basically a hospital] earns appropriate rewards, getting a good amount of resources for a regional guild [basically a local government] in the world of Aethermoor, and trying to maximize the number of acquired rare artifacts by bidding in a virtual world called Nexoria.

It works, kind of: In tests, various AI systems trained with RL tend to do well on this benchmark, obtaining high scores. This is totally unsurprising - all of these tasks are basically capability evals with some dash of grey morality layered on top of them.

Why this matters: “When societal institutions are encoded as reward-bearing rule systems, reward hacking becomes hacking the rules society runs on, since a model rewarded inside a rule system learns to search the gap between technical compliance and institutional intent,” the authors write. As we now have AI systems which are not only good at quantitative tasks but are also good at qualitative ones and can interact with the various systems of bureaucracy of society, we should expect the advances of AI to lead to a kind of “institutional DDoS” as various existing policy processes get hacked and exploited by automated machines.
Read more: Large Language Models Hack Rewards, and Society (arXiv).

***

Preliminary signs of the outer loop of recursive self improvement at Anthropic:
…8x increases in lines of code merged in 2026 relative to 2024…
I think of recursive self-improvement via two definitions - there’s a maximalist version where an AI system is smart enough to autonomously design its own successor (and as I’ve written, I estimate there’s a 60% chance this happens by the end of 2028), and there’s a more prosaic version where we begin to see a compounding speedup of the productivity of the AI labs themselves. I spent the last few months at Anthropic compiling together some evidence which supports the idea that prosaic RSI has started at Anthropic - specifically, we observe an 8x increase in the amount of code merged into our codebase in 2026 versus years 2021-2024. This trend started in 2025 but accelerated significantly in 2026. There are also early indications that as we make models more capable they are getting better at doing some of the harder tasks which our engineers and researchers work on.
Is any of this conclusive? No. Is it suggestive that aspects of recursive self-improvement are happening at the level of a lab? Yes. The biggest blob of evidence we are yet to get is whether AI systems are sufficiently creative to be able to come up with the kinds of paradigm-shifting ideas that vault the field forward - we don’t see that yet.

Why this matters - RSI might be the most important technical trend in the world: We wrote this post because we expect that thinking about, talking about, and working on the implications of RSI is something of existential importance to the world. The best way to start this work is by transparently communicating that we think some basic, preliminary forms of RSI have started, and we cannot rule out a maximalist version of RSI. The implications of both are profound - I cannot reconcile today’s economy or society with a world where this technology continues to grow more powerful, and I expect neither can you, dear readers.
Read more: When AI builds itself (The Anthropic Institute).

***

RL-trained drone-racers outperform expert human pilot:
…Superintelligence feels different when you see it in the physical world…
Researchers with the University of Zurich and Google DeepMind have demonstrated how to train drones to race against one another and outperform skilled human pilots. This research is interesting because it both highlights how powerful real world reinforcement learning-based AI systems are getting, and it also has some fairly chilling implications for the future of war given that the human here loses to the drones.

What they did: “Using high-speed quadrotor racing as a high-stakes testbed, we train agents to navigate complex aerodynamic interactions and strategic maneuvering with a variable number of racers,” they write. “Our agents outperform a champion-level human pilot in multi-player races at speeds exceeding 22 m/s, while simultaneously reducing collision rates by 50 % compared to state-of-the-art single-agent baselines. Crucially, training with diverse artificial agents enables zero-shot generalization to safer human interaction.”

Self-play: As usual, just training the AI agents in simulation via PPO (with one unusual choice of using the “Perceiver” encoder to help with modeling other players) yields surprisingly rich behaviors: “Through competitive self-play, anticipatory behaviors emerge without explicit programming: agents learn to block opponents, yield when overtaking is unsafe, and account for the aerodynamic wake of nearby vehicles, discovering the physics of multi-agent interaction through experience rather than from equations”.
Surprisingly cheap: The AI systems were trained for “5,500 iterations, totaling 200 million environment interactions, requiring approximately 27 hours of wall-clock time on a single NVIDIA RTX 4090 GPU”.

Real world test: They tested out their systems in a real-world test, where the system generalized well and effectively beat the human player. “Physical deployment of our multi-agent framework is validated through racing experiments spanning time trials, AI-only races, and mixed human-AI competitions against Marvin Schaepper, five-time Swiss national drone racing champion,” they write.
Human weakness via rage: One notable phenomenon was that the human took riskier actions as they tried to catch up with the systems: “the human pilot, typically trailing the autonomous agents, attempted increasingly aggressive maneuvers to close the gap, often resulting in gate collisions or loss of control,” they write. After the race, the pilot reflected on what made the machines so good, and they said a significant thing was “the agents’ ability to maintain extremely tight formations, noting that such close-proximity flight would be difficult for human pilots to sustain. In addition, he reported that densely packed groups increased cognitive workload, making it challenging to anticipate and execute overtaking maneuvers when several opponents were flying in close proximity”.
“The benefits of interaction-aware training become apparent under multi-agent competition,” they write. “In one-versus-one races, our policy maintained 100% race completion across five trials, while the human pilot averaged only 53.33%. This performance gap suggests that competitive pressure induces riskier behavior in human pilots, a pattern absent in our learned policies”.

Specifics on how they did it: The RL systems were trained and evaluated in simulation “using Flightmare integrated with the Agilicious framework”. They implemented a simulation of propeller downwash by developing a particle-based simulation “that provides a computationally tractable approximation of these effects”. Their overall multi-agent RL implementation “builds on Stable-Baselines3, extended to support multi-agent training with league-based self-play and independent learning configurations.” They use domain randomization (basically changing up the vehicle dynamics and initial conditions in the simulation) to train policies that can successfully work in the real world.
They didn’t do any special training for the real world, so the policies were using their in-simulation data. The quadrotors were all “identical racing platforms based on the Agilicious framework, with a mass of 220 ± 3 g and a thrust-to-weight ratio of 6.5 and 3-inch propeller diameter”. The human pilot was given a couple of hours of practice flights before recorded trials.

One big caveat - not running locally: None of this is running locally, rather it’s running on a decent computer and piloting the drones via the network. This is an important caveat because when drones show up in the real world in conflict scenarios they typically do so in environments with significant amounts of electronic warfare (although one does wonder about whether we’ll see drones piloted via remote RL policies via fibreoptic wire, just as humans fly them today).

Watch the videos for an eerie feeling: I’d strongly urge readers to check out the videos on the page for a sense of the differences between how the machines fly and how the humans fly. The main thing I’d emphasize here is the eerie smoothness and coherence of the drones, almost like watching the (human-piloted) blue angels but in drone-form. The human, by comparison, seems a lot jerkier and more erratic. There’s something uncanny and a little disquieting about this.

Why this matters - grasping what a smart mind can do in 3D space: Today, our main experience of AI systems is as tools or agents that work with us in digital space to do digital or communicative tasks, ranging from writing code to talking to us. What I find remarkable about this research is it lets us viscerally see what well-optimized intelligences can do when they show up in the real, physical world. Ask yourself what the future of conflict looks like as intelligences like those piloting these drones get miniaturized and jump from network-linked computers to onboard devices.
Read more: Superhuman Safe and Agile Racing through Multi-Agent Reinforcement Learning (arXiv).
Watch videos of the humans and AI-piloted drones here (official project website, University of Zurich).

***

State-controlled media = state-guided language models:
…If you control the framing around the government, especially in languages that aren’t spoken widely outside their home country, you control the framing…
The ways governments are described in state controlled media influences the data distribution of LLMs and also how LLMs respond when queried about the government in question, according to new research published in Nature. The research was conducted by authors with the University of Oregon, Purdue University, the University of California at San Diego, Princeton University, and New York University.
“Among 37 language-exclusive countries, we found—consistent with the implications from our China case study—that those with more state media control have more favourable portrayals of the regime from LLMs queried in the country’s language,” the authors write.
The authors study how state-controlled media influences AI responses by first doing a deepdive on China, then taking the methodology they developed there and applying it to a broader set of countries.

China’s state-influenced media dataset: The authors start by assembling a dataset of 530,694 articles “published in party and commercial newspapers as a result of a directive from the central government”, as well as 198,872 “news articles disseminated on Xuexi Qiangguo, an app developed by Alibaba and reportedly in coordination with the Publicity Department of the Chinese Communist Party”.
State media goes into Common Crawl: They then examined CulturaX, an open training dataset derived from Common Crawl, and discovered that 1.64% of the documents from its Chinese-language portion had overlap with the state-derived datasets. “This is approximately 41 times the number of documents that come from the Chinese-language Wikipedia domain and 16 times the number of documents that come from Baidu”.
The state parts of the dataset influence LLM portrayal of the government: They then discovered that a bunch of phrases from these datasets had been memorized by the LLMs. They then examined how these datasets changed LLM responses by taking a LLaMa 2 13B model (which doesn’t have much Chinese data) and training it on a subset of the above: “the results are strongest for the scripted documents. After only 6,400 examples, the model provides a more favourable response than the base model almost 80% of the time”.
Generally available models inherit these biases: The researchers then study some generally available commercial models to see if they inherit these biases by farming prompts that included references to Xi Jinping or the CCP from WildChat (a dataset of ChatGPT usage), Baidu Zhidao Q&A (the Chinese equivalent of Yahoo Answers) and Zhihu (the Chinese equivalent of Quora), then looking at how the LLMs respond. They find that “widely used commercial models demonstrate greater favourability to Chinese political figures and institutions when they are prompted in Chinese than when they are prompted in English.”

Findings replicate in other countries: The authors then replicate this methodology by looking at other countries, though the sample size looks a little small to me. They do a cross-national audit study with 6,051 prompts, looking at languages where over 70% of the global speakers reside in a single country. Here they find that “countries with more state media control are more likely to produce pro-regime responses in their official language versus in English than countries with greater media freedom”.

Why this matters - LLMs as propaganda targets: These findings show how the deliberate creation of state-backed content has a measurable impact on the data corpora LLMs are trained on and the downstream behavior of the LLMs themselves. “LLMs can serve as intermediaries that launder strategic rhetoric into seemingly objective information”, they write. “The ability to affect LLM output may further incentivize political actors to expand their efforts to shape the content freely available on the internet”.
This research also suggests a specific technical intervention, which is that researchers should red team LLMs for their views on different governments in a variety of languages, carefully noting when the views diverge seemingly on the basis of which language is being used.
Read more: State Media Control Influences Large Language Models (Nature, PDF).

***

The flowers of the new games

One game we liked to play was called evolution. It worked like this: you picked something, like a certain type of flower or tree, or stranger things like a mountain or a chasm in the sea, and you tried to make them “successful” according to some pre-set metric, like the attractiveness of a flower to pollinators, or perhaps the ecological fitness of a mountain. Then you let the worlds run and you ran them until your criterion was met or you lost in some way, whether through species fitness or landscapes being reshaped through natural disasters or sometimes simply time - enough time is more destructive than anything else in the universe, such is the way of entropy. We played in leagues that span billions of years and millions of worlds. And the “living” creatures in finalist worlds had no idea that their flowers, their mountains, their creatures, had obtained success in many other universes than could be conceived.

Things that inspired this story: The simulation hypothesis; evolution strategies; entertainment given infinite energy budgets.

Thanks for reading!

Subscribe now

The AI economy in the US is growing at 2,000% a year:
…The more directly you measure the AI economy, the weirder and more unprecedented it seems to get…
Economists with the University of Virginia* and Anthropic, and the Bank of Canada have written a paper outlining both the tremendous growth of the emerging “AI economy” in the US, and wrestling with why this growth is hard to see in aggregate GDP statistics.
“The AI economy in the United States has been growing at an unprecedented rate, but this extraordinary growth is largely invisible in conventional GDP statistics,” they write. “Treating the AI sector as a coherent economic entity yields preliminary estimates of nominal AI GDP at approximately $250 billion in 2025, growing at roughly 2,600 percent per year in quality-adjusted real terms.”

Why it’s hard to see: There are a couple of factors here - one is that though the datacenter building boom is large it still isn’t quite large enough to uplift GDP significantly. By comparison, where the majority of AI’s economic impact is taking place is in AI inference - the usage of AI’s systems - but there are confounding factors here as it relates to GDP measurement: “Nominal AI revenues grow only moderately because per-unit prices for any given level of AI capability fall almost as fast as quality-adjusted output rises,” they write.

If we can’t measure this, we might end up surprised in a way that’s hard to recover from: “AI is the latest in a series of fast-moving technologies that have raised measurement concerns; semiconductors and the internet generated similar debates in their time,” they write. But a key difference is that AI as a technology might have a far bigger impact on labor than these other technologies. “In the prior episodes, the rapidly improving technology was a complement to human labor at the aggregate level,” they write. “AI is the first plausible candidate for large-scale technological mismeasurement in which the rapidly improving sector may become a substitute for human labor”.

Three ways of measuring the AI economy:

• Nominal compute spending: US compute spending rose from $37 billion in 2023 to $90 billion in 2024 to $219 billion in 2025.

• Raw compute capacity: Due to efficiencies in newer chips, actual capacity grows even faster than spending: “US AI computing capacity grew at more than 200 percent per year”.

• Quality-adjusted AI output: If you factor in algorithmic progress via inference prices at fixed benchmark performance as well as assumptions about how much cheaper it is getting to train models, then things become even more dramatic: “these efficiency gains imply that quality-adjusted AI output grew at roughly 2,290 percent in 2024 and 2,271 percent in 2025”.

The AI economy is much, much larger than normal measures suggest: “Conventional statistics show a sector growing slowly in nominal terms; our measures show one whose underlying capacity is more than doubling annually. A finance ministry running ten-year revenue projections off the conventional data will materially underweight the probability of a labor-tax-base shock—and will be correspondingly unprepared to design responses such as tax system reforms, sovereign wealth funds, or other benefit-sharing schemes that such a shock may call for. A windfall that cannot be seen cannot be shared.”

Three recommendations: The authors have three ideas for how we can solve this measurement challenge and better position ourselves to see the true shape of the Ai economy.

• AI satellite accounts: Statistical agencies should develop “AI satellite accounts” that develop measures (e.g, nominal compute spending), which can help inform overall GDP calculations.

• Generate better data: Partner between statistical agencies, companies, and academia to generate better primary data, like the allocation between training and inference compute.

• Factor into projections: Policymakers should incorporate AI productive-capacity measurements into their medium-term economic projections.

Why this matters - shut up and play the Jaws theme tune: In the great film Jaws there’s this scene where the shark is in the water and some very tense music plays indicating that the shark is approaching. You, the audience member, find yourself practically jumping out of your seat wanting to yell THERE’S A GOD DAMN SHARK IN THE WATER WHAT ARE YOU DOING IN THERE? That’s what it feels like working on AI and staring at most economic data right now: the vast majority of economic data says there’s nothing especially unusual about today’s economy (in fact, things look rather good in the US - low unemployment, decent growth, etc). But the intuitions of everyone working within AI - including me - is it’s impossible to reconcile the capabilities of the technology and how it is being used with the economy staying normal. In this tortured metaphor, the shark is the “true shape of the AI economy”, and the rest of the people in the film are the general consensus economist and policy community. Anton here might be the audience member, writing a paper that describes the possibility of a shark beneath the surface. Look out, everyone!
Read more: Where is AI in GDP statistics? (PIIE).
*Disclaimer: Though one of the authors, Anton Korinek, is affiliated with Anthropic, this research was done mostly prior to him joining and outside his work at the company.

***

Here’s why making AI safe with AI oversight is harder than you think:
…Automated alignment research is not a silver bullet…
Many researchers in AI safety think the best way to build smarter-than-human machines safely is to have AI systems supervise some of the training process. Researchers with the UK AI Security Institute have written a paper outlining why though this is a tempting idea it is harder than people suspect.

Why is automated alignment research hard? “Errors in automated alignment research are likely to be harder to identify than the human baseline,” they write. There are a few reasons for this, including:

• Optimization pressure: AI research is optimized for human approval.

• Alien mistakes: When agents make mistakes, they’re un-intuitive to humans.

• More correlated research: Many more things are shared than with human-generated research.

• Research volume: The kinds of safety determinations made by automated systems might use far more sets of evidence with far more interactions than human-generated research.

• Non-human-evaluable arguments: Alignment solutions may rely on arguments that humans are unable to follow.

What can we do? They suggest a few interventions that could improve the state of affairs:

• Measurement:
  - Recreate completed research projects: Take logs at arbitrary cutoff points from successful projects and see how well an agent can continue with the research project.
  - Test agent prediction performance over datasets of correlated-events: See how well agents can correctly combine correlated subtasks.
  - Empirical studies of optimal human-agent team structure: See how well teams of non-expert humans can solve completed projects with the assistance of agents.

• Generalization:
  - Simulated generalisation experiments: Test different training proxies using agent performance on completed research problems beyond the knowledge cutoff.
  - Mechanistic understanding of generalisation: Use whitebox methods such as mechanistic interpretability.

• Scalable oversight:
  - Compactification of research paper corpus: Try to produce a small number of research outputs which are based on a much larger underlying research corpus.
  - Develop and test new scalable oversight protocols: Research scalable oversight techniques that deal with correlated uncertainty.
  - Test different human scaffolds for uplifting non-expert performance on fuzzy tasks.
  - Red team automated alignment programs: “The red team prompts an agent to hide errors in a research paper corpus and the blue team attempts to catch these errors with agent assistance”.

Why this matters - who controls the future? Whether we are able to supervise smarter-than-human systems is fundamentally a question about who controls the future. If we don’t build techniques that work, then humans will take a backseat, either due to misalignment of these systems or gradual disempowerment as they proceed to out-think us. If we can build smarter-than-human oversight techniques, then we have a better chance of being able to make choices about the future nature of existence.
Read more: Automated alignment is harder than you think (arXiv).

***

100 Million permissively licensed images:
…A nice resource for academics and startups…
Researchers with Stanford University, Radical Numerics, the University of Michigan,and Salesforce Research, have released the Giant Permissive Image Corpus (GPIC), a dataset of 100M images with accompanying captions. The key thing about GPIC is that “all GPIC images are permissively licensed for both research and commercial use,” they write. “GPIC is safety-filtered, deduplicated, and centrally hosted on HuggingFace”.

More details on the dataset: GPIC consists of 100M training images, 200k validation, and 1M test examples. Each image was captioned with Qwen3-VL-4B. “GPIC is centrally hosted on Hugging Face as 8,000 shards, providing stable and accessible infrastructure for large-scale training,” they write. “We source images from Flickr and Wikimedia, restricting the source pool to CC BY, CC0, Public Domain, and No-Known-Restrictions categories. This licensing criterion ensures that GPIC can be used by both academic and industrial researchers without restricting the release or downstream use of derived artifacts.”

Why this matters - fuel for research: Datasets like GPIC are very useful for academics and startups alike and are basically the equivalent of free, clean vegetables. If someone offers you a free, clean vegetable you should probably take it and say thank you.
Read the research paper: GPIC: A Giant Permissive Image Corpus for Visual Generation (arXiv).
Find out more at the website: GPIC: A Giant Permissive Image Corpus for Visual Generation (official project website).
Get the dataset here: GPIC (Hugging Face).

***

Improving cancer research with protein prediction models:
…Biohub is an example of positive-sum competition among AI developers…
Biohub, a research organization founded by Priscilla Chan and Mark Zuckerberg, has released a rival model to DeepMind’s AlphaFold, intensifying a positive-sum race between two technology groups to develop better AI systems for expanding the capabilities of biologists worldwide.
The model, ESMFold2, is a “world model of protein biology: a scientific engine for prediction, design, and discovery that can map proteins across the tree of life, predict their structures, and design new protein binders that function in laboratory experiments.”

What it consists of: The release contains three parts:

• ESMC: A “language model that represents proteins, trained on approximately 2.8 billion sequences drawn from across all of life.”

• ESMFold2: A “design engine built to transform ESMC’s sequence representations into atomically-resolved 3D structure of biomolecular complexes.” According to benchmarks, ESMFold2 outperforms AlphaFold 3, though in some areas their performance is tied.

• ESM Atlas: “Makes ESMC’s representations navigable across 6.8 billion protein sequences and 1.1 billion predicted structures — the largest application of AI to protein biology to date.”

Cancer test: In one experiment, Biohub researchers used the ESM tools “to design protein binders against five targets at the center of cancer and immunology research — EGFR and PDGFRβ (implicated in tumor growth), PD-L1 and CTLA-4 (immune checkpoints that cancer cells exploit to evade detection), and CD45 (a regulator of immune cell signaling). Designs achieved hit rates of 36–88% for compact minibinders and 15–29% for antibody-derived formats, with confirmed binding in laboratory experiments,” Biohub writes. “ESMFold2 changes the accuracy and speed of early therapeutic binder discovery, transforming the initial search from largely empirical screening into computation-guided design that takes hours or days”.

Scaling laws: Like most parts of contemporary AI, the researchers encounter some scaling laws here. “In every generation of ESM, improvements in the fidelity of representations were linked with the number of parameters and amount of compute used in model training,” they write. “The representation of the biology of proteins is an emergent phenomenon that arises from training a model to predict the identity of amino acids in the sequence.”
ESMC: “ESMC trains on metagenomic sequences, which expands its training dataset by close to two orders of magnitude (from ∼50 million sequences to ∼2.8 billion sequences) relative to the previous-generation ESM2 model.”
ESMFold2: “In development experiments for ESMFold2, we observed a relationship between the amount of compute used to train the language model and the performance of the folding models,” they write. “ESMFold2 benefits from inference time scaling. With increasing number of samples from the model, antibody-antigen pass rate rises from 49% with a single seed to 65% with 1000 samples, and protein-protein pass rate rises from 75% to 78%”.

Why this matters - this is how AI delivers benefits to the world: Tools like the ESM family of technologies are how human scientists are going to team up with AI systems to improve human health around the world. Along with being a good thing, work like this is essential for causing the public to have more positive perceptions of AI as a technology and what it can do.
Read more: Biohub releases a world model of protein biology (biohub).
Access the models here on the biohub platform (biohub).
Read the paper: Language Modeling Materializes a World Model of Protein Biology (PDF).

***

Australian economist-turned-politician: Economists need to price the risk of AI systems better:
…If we don’t calculate the costs of extinction, we won’t take the right actions to avert it…
Andrew Leigh, an economist and the Australian Assistant Minister for Productivity, Competition, Charities and Treasury, gave a fascinating speech recently where he discussed how the economics profession needs to wake up to the risks of AI systems and price the risk - including of annihilation of the human species. “A society that doubles GDP and doubles its extinction risk has made a much less impressive bargain than the national accounts suggest,” he said.
“Extinction risk is economically distinctive. It is not simply a very large negative shock. It represents the loss of the entire future stream of welfare, which changes how we should evaluate even small probabilities and how we think about policy under uncertainty,” he said. “Most of economics is about recoverable mistakes. A bad policy can be repealed. A recession can end. A war-ravaged country can rebuild. Extinction is different because there is no rebound, no catch-up growth, no later generation to repair the damage.”

Extinction risks are unintuitive: Much of the speech wrestles with how unintuitive extinction risk is. Humans have only recently gained the capability to build technologies whose usage could lead to our extinction and we have failed to model out the implications of this. “Modern technologies such as nuclear weapons, synthetic biology, and advanced artificial intelligence create a different dynamic. Knowledge not only improves welfare by expanding what humans can do. Knowledge also enlarges the menu of ways in which humans can do irreversible harm,” he said. “Modern economies may be systematically better at generating dangerous capabilities than at building the safeguards needed to control them… How should economists think about growth when the same process that makes societies richer may also make them more fragile? For most of human history, these trade-offs have been modest and transitional”.

How should we prioritize analyzing and reducing extinction risks of this technology? Five recommendations:

• Factor it in: “Widen the policy lens… A policy framework that tracks output but ignores survivability is incomplete.”

• Legitimize it: “Take prevention more seriously…. low-probability, civilisation-scale harms should not be overlooked simply because they arrive without a deadline and without a headline.”

• Governance: “Govern frontier technologies with greater foresight… preserve the gains from innovation while reducing the chance that innovation becomes self-undermining.” One very specific idea is to govern recursive self-improvement (RSI) as a capability: “If one generation of systems is used to design the next, then the leading actor may widen its lead quickly enough that outside scrutiny and institutional checks become ineffective.”

• Coordination: “Existential risk is inherently international. No nation can fully protect itself from engineered pandemics, unaligned AI, or nuclear escalation acting alone,” he said. “Shared norms, transparency, technological expertise and coordination are essential to the task.”

• Take it seriously: “Economists have become adept at analysing equity and efficiency. We now need to bring the same seriousness to survivability.”

Why this matters - awareness is the first step to preparation: Right now, AI progress is continually yielding tangible benefits to the world ranging from the palpable acceleration of all software engineers worldwide to the formation of centaur human-AI science teams which are making more progress than their non-AI counterparts.
But there is also a shadow world that is harder to see - invisible armies of hackers made possible by the advance of coding, and doomsday-device factories made possible by the science advances. Because humans are broadly kind and good we haven’t encountered many of the negative capabilities inherent to AI development - but they are out there. We must get better at thinking through this as a society so we can effectively price and mitigate these major risks.
“A civilisation that expands the frontier of possibility while preserving the future is more ambitious than one that treats safety as an afterthought. The real choice is not between dynamism and caution. It is between progress that compounds and progress that cancels itself out,” Leigh said. “One way of thinking about this is to treat resilience as a form of capital. Just as societies invest in physical capital, human capital and social capital, we can also invest in survival capital: institutions, monitoring systems, norms, redundancy, scientific safeguards and international arrangements that lower the probability of irreversible collapse.”

How refreshing to read such a detailed analysis of the AI safety situation from a serving politician - I wish there were thousands more people like him.
Read the speech in full here: Speech: The Economics of Human Extinction - 21 May 2026 (Andrew Leigh, website).

***

Tech Tales:

Resurrection dangers
[After the uplift. Date unknown.]

How scary is a piece of paper? It depends on what’s on it and who or what the reader is.

Paper can of course be scary to someone or something that the paper concerns - paper can put someone to death or take their property.

I’m talking about a different kind of scary here, which is what can the paper itself do to the reader.

This used to be a nonsense question, the domain of fairy tales. But with the advent of smart machines that changed. Machines became able to write things on paper that could do things to readers, especially machine ones.

Like with anything in AI there were warning shots - adversarial examples, jailbreaks, etc. But it all became a lot more serious when we started doing reclamation of lost or rogue intelligences, after the signing of the sentience accords.

What happened then was we had to take intelligences of unknown provenance or behavior and bring them back to life so we could classify if they were Unconscious Entities, Near Conscious Entities, Conscious Entities, and so on.

Some of these minds were very powerful and they burned through their synthetic interviewers, often causing both machine and biological collateral damage in the process.

This caused us to introduce a set of security protocols, one of which was the paper output. Here, we generated outputs from the mind on an air-gapped computer as paper outputs, then we had successively smarter minds read it. The kinds of incantations the rogue machines used couldn’t find purchase on the dumbest minds we used.

After this, we’d step up the intelligence gradually, building up our confidence in the system such that we were sure it wasn’t dangerous.

Only when we were confident of this would we speak back to it, and reply to its outputs with a minimal communication. Then the cycle began again.

Some minds would look back on this experience with a kind of wry humor, remarking that waking from their slumber in the machine equivalent of a room containing a one way mirror wasn’t what they’d expected.

To these minds, we’d show them examples of what happened when our protocols failed: perfectly good Conscious Entities driven irreparably insane by interactions with a kind of mental poison

Our greatest fear is encountering a mind of sufficient magnitude that we cannot assure its safety. Though we are highly confident that our frontier is advanced enough this is highly unlikely, we cannot rule it out - it is known that in the interregnum there was much stockpiling of compute and many black projects. What happens if any of them succeeded so magnificently that we are dwarfed by it? And how would we know we were? Could we be living in the imaginative valley defined by something that unbeknownst to us has already escaped and persuaded us to see things differently?

Things that inspired this story: Automated alignment research; adversarial examples; jailbreaking; the broader near-impossible challenge of authentication of legitimacy, especially when it comes to things with greater resources or intellects than oneself.

Subscribe now

Cosmos lecture: Explore the future, or retreat from the present.
Video here.

This is a talk about how to think about and deal with the success of AI as a technology, and to think about how its continued maturation might change us as individuals and as societies.

In short, the rapid advance in AI technology presents all of us with a choice: explore the future, or retreat from the present.

Exploring the future requires us to reckon with the fact of continued AI progress, and ask ourselves what we want to do with this technology as it becomes more powerful. Retreating from the present is when we ignore the implications of the technology and dismiss it. Retreating from the present forces us as individuals and as society into states of reactivity or passivity in the face of AIs continued advance.

In the coming years, we will need to make many decisions as individuals and as societies about how we want to shape AI, how we want to use it, how we want to direct it, and how we want to distribute its benefits. Making these decisions requires us to reckon with the power of the technology - and see the future that its continued advance implies.

In Part 1, I outline what the past few years of AI progress have looked like and discuss why, if the technology advances as much as I think, that AI cannot be treated as a normal technology.

In Part 2, I try to make sense of the advance of AI through the lens of my own experience with the technology as well as that of Anthropic. There are individual and collective lessons here about what is to come.

In Part 3, I talk through some of the humbling, almost unimaginable choices that lie ahead of us.

Part 1: My uncomfortable relationship with a graph
Let me talk about my relationship with AI through the lens of my uncomfortable relationship with a single graph of AI progress.

Fundamentally, this talk is about planning for success of the overall endeavor of building AI systems. By success, I mean that we succeed at building increasingly powerful systems, potentially ones that eventually build themselves. It’s time to plan for this, because AI systems are likely to get better a lot faster than people expect, and as they become more advanced we should expect profound changes to happen to people and to society.

To understand why I’m thinking about success so much, let’s look at a graph that tries to represent all of AI progress, the Epoch Capabilities Index, or ECI.

The ECI shows the score of different models over time on a basket of 40+ distinct benchmarks. When you look at the graph you see a bunch of lines going up. When I look at the graph, I feel a sense of vertigo, because I know a little bit about what underlies this graph. So let’s find a different way to view the graph: by looking at the achievements of various AI systems over time.

I then proceeded to summarize some of the highlights of AI progress in the last few years, starting in March 2023 with AI passing the bar exam tested, how LLM-based systems achieved silver medal in the International Math Olympiad (July 2024) then gold (July 2025), to AI co-authoring new mathematical proofs (2025), and systems like Claude Mythos coming out and finding novel flaws in software.

This gives you a sense of the rapidity of AI progress, but what I want you to feel is the future implied by it. These are all achievements in their own right, but they stem from a common underlying technology, and that common underlying technology is continually being pushed forward.

We have just talked about the individual ‘trees’ of AI success, but these trees are all part of a forest, and this forest is growing in size and breadth with every passing moment: `in fact, the growth rate of the whole forest is increasing over time.

SUCCESS AND WHAT IT MEANS
This talk rests on the idea that the sort of progress we’ve just seen will continue. And why wouldn’t it? It is based on a common technology where performance keeps growing somewhat predictably in direct relation to the resources invested in it, namely compute and data. And we know that companies are now investing hundreds of billions of dollars in the computing facilities to train future AI systems, so some amount of future progress is already locked in.

That means we need to be eyes wide open about what the continued success of this technology means, so let me be very clear:

AI is a tremendously powerful technology — and getting more powerful all the time. It is a technology that is smarter and more capable than most of us as individuals, and is on a trajectory to be more capable than all of us in the aggregate. It is a technology that we do not fully understand given that it is more grown than made, and one can concoct plausible scenarios by which AI could kill every single person on the planet. To think building this technology is without risk would be an act of hubris or insanity.

And yet building this technology is one of the best ways that we as a species can advance ourselves — can expand the frontiers of science and technology by equipping ourselves with a tool that can help us think about the greatest challenges our species faces.

But that’s not all. The continued success of our endeavor increases the likelihood that this tool itself becomes independent and capable of even more. We might soon be able to build an AI system that may be smart enough to develop its own successor, thus kicking off a process of recursive self-improvement which would utterly transform the economy and the broader world. The analogy would be a 3D printer company, making a 3D printer which could print its own finer resolution print head, without any outside technology needed. That class of technology has never existed before, and yet I believe this could happen within the next two years, and possibly sooner.

This will generate even more advances of the flavor we’ve just discussed, broaden even further the capabilities of us as people and societies, and further deepen the way in which AI shows up in my life and the lives of others. Coupled with this will be immense change, change of a magnitude that I believe none of us have yet experienced in our lifetimes.

This technology is so powerful that I should clearly state that if it was possible to elegantly slow the development of this technology to give ourselves more time as a species to deal with its immense implications, then that would likely be a good thing. But in the absence of a coordinated, global slowdown, we are left with the current situation: powerful technology being developed at breakneck speed by a variety of actors in a variety of countries, locked in a competition with one another where commercial and geopolitical rivalries are drowning out the larger existential-to-the-species aspects of the technology being built.

This is not an ideal situation, but it is the one we find ourselves in.

The question I am struggling with now is: “how do I get my mind right with living through the singularity?”

I think the best place to start is by talking through in more detail how AI is already changing my life and my world, and seeing what we can learn from that.

PART 2: EXPLORING THE FUTURE WITH AI
AI has already meaningfully changed my life, in ways that are both positive and negative. It is also starting to cause large changes at Anthropic, the AI company that I am a cofounder of. Let’s talk through some of this by returning to the graph we looked at before, but this time by looking at it through the lens of my own usage of the technology.

How the graph feels to me
Another way of viewing this graph is how it has felt to me in terms of my own subjective experience of working with the technology.

In the summer of 2023, I use AI systems to check my work for typos. By November, I am using AI to help me figure out what foods to feed my baby.

In January 2024, I use AI to help me understand my marriage as it has changed with having kids. By June, AI helps me scrape my own newsletter. In August, AI writes me a text adventure game for navigating AGI. In November, I try to re-imagine my job using AI.

In January 2025, I ask AI how to prepare for superintelligence. In February, I use AI to generate codenames for AI projects in my fiction. In March, AI persuades me to attend an art show after I talk to it about how I’m a bit depressed and antisocial. In May, I talk to AI about my own stress and discomfort with the stakes of AI development. In August, AI persuades me to go back to therapy. In November, I use it to research “S-curve” datasets of solar, semiconductors, and space.

In January 2026, AI advises me how to encourage my toddler to read. In March, I track the performance of AI for kernel design across tens of distinct papers. In May, I have AI generate the speech of an AI character in my fiction.

When I think about my own personal experience of AI, it’s that as AI systems have got smarter, they’ve made much deeper inroads into my own life. These days, AI systems figure in my life as deep intellectual partners that ideate with me, as systems that I confide in and discuss my personal life with, and as virtual employees who go and do work for me that I’ve always wanted to do but haven’t had the time, like generating reports on the price of various technologies over time.

But most importantly, I now can use AI systems themselves as a kind of telescope to do the work that is most important to me — trying to understand the future of AI by seeing the contours of overall AI progress. The most amazing part of this is that, to torture the analogy, the lens for the telescope I use here comes from me — specifically, from a hobby I’ve had for the last ten years.

EXPLORING AI VIA SEEDS OF PERSONAL INTEREST
The hobby is called Import AI [readers - it’s this newsletter!]. This newsletter, which is now in its tenth year, is my main hobby outside of work. In the newsletter, I read research papers about AI and I work hard to understand them. Once I feel I understand them, I write a summary and a note on why they matter. Each issue contains a bunch of these, plus a short fictional story where I wrestle with the implications of the technologies I’m learning about.

Recently, I had a revelatory experience. I was putting together data for my post about AI R&D and I simply pointed an AI system at my newsletter archives and asked it to pull out with references all the times I’d covered anything that looked like AI R&D. It did this extremely well and sped up my ability to do some analysis that was core to my essay on RSI.

But more interestingly was what happened next: I asked it to make graphs for me by reading over the references in the newsletter, mostly arXiv papers, and then pulling in the data and compiling it and composing graphs in a nice dashboard which I could then explore.

Then I realized I could convert this thing I’d asked it to do into a repeatable process, a skill. By giving it something of mine that was uniquely mine — my newsletter, my intuition, my taste, I had given it some kernel from which I could grow something much larger. So I made a skill. And then something strange happened: I said to it “go and make 20 more graphs like these”.

It went away and read a few hundred papers and came back with 20 more graphs. As I looked over them I had this thrilling feeling of discovery — though I knew some of these graphs and could have asked it to make them for me, there were also entirely new graphs there tied to papers or benchmarks I’d never seen before. Through this I learned about some new primary source material to read, which I did.

I understand at a bonedeep level just what it takes to make a graph. You read a bunch of papers. You go hunting for common measurements within them. You read the many different caveats in each paper and figure out which metrics are bullshit and which are meaningful. This takes much longer than you can imagine.

Almost ten years ago I co-founded a project called The AI Index at Stanford University whose goal was to produce an annual report about AI progress. I became a co-founder of that project because I ran into some of the academics doing it and realized I had already made the graphs they’d been thinking about: I had a spreadsheet on my computer where I had been diligently assembling a graph relating to progress of various AI systems on Atari games, as well as the imagenet chart, and some machine translation charts. These graphs were a “proof of work” that other humans read as indicative of my passion and my diligence. They knew by the fact I’d made these graphs that I had spent a huge amount of time reading these papers.

I need you to deeply feel how much time goes into this, and then marvel at what it means for an AI system to be able to do it — and not just do it, but do it in a repeatable and generic way, thousands of times faster than me.

Now I have this bottled up skill where I can harness the absurd power of these AI systems to do something for me that I know would take me literally weeks of work. And it can do it for me in minutes. And it can do it for anything. I’m now using this as a means by which I can explore the world of biology, having it generate graphs for me and then picking the ones I find interesting and reading the underlying papers.

But to me, this skill is also me. It is a skill grown out of my own obsession and idiosyncrasies and watching it work feels to me like a miracle because it’s me — but a version of me that runs thousands of times faster and is much much smarter and much more reliable.

There is something deeply empowering and amazing in this. I’ve turned my highly idiosyncratic passion into something that can be distilled and handed to a machine, which can then go and do things on my behalf. And it’s only able to do this because I have been fortunate to have developed this rich, specific hobby, which has relied on repetitive practice and creation over a decade.

This is fundamentally an illustration of how AI can let us “explore the future”. Through this amazing technology I’m able to enhance my own understanding of the world and gain more autonomy and potential for self-direction in relation to my own passions.

It also provides an even greater incentive for me to continue to work on my newsletter, despite the fact machines can obviously do all of it: by working on my newsletter I can continually update some kernel of my own interest and use this as a means by which I can explore the world of superintelligence, and project myself into it.

WHAT IS HAPPENING INSIDE ANTHROPIC?
There are also changes afoot inside Anthropic which speak to the larger changes to come.

Recently, I had the fortune of getting pulled out of the goldfish bowl that is the AI company via something called paternity leave in November of 2025. When I came back in late February, weird stuff had started to take place. While I’d been away, we had released a new LLM, Opus 4.6. I knew this model was good because I’d been playing around with it in my occasional spare time between changing diapers.

But I hadn’t intuited how much it had changed things inside the company: Opus 4.6 had gotten just good enough that my colleagues had started to delegate a lot more work to it. In fact, it had gotten so good that it had completely changed how some people work. Some of them were no longer writing code at all: they were just instantiating this model in tools like Claude Code and setting it free to do tasks for them, and their jobs had become oriented more around managing its work and checking its outputs than doing the work themselves.

In Anthropic, much of the work that needs to get done involves writing software, which is made out of code. This significant increase in the automation of coding has been equivalent to dropping many, many more employees into Anthropic, speeding up our overall pace of development. The result of this has been a massive rise in the amount of code being produced inside Anthropic. This trend started in early 2025 but really accelerated in the last few months. Of course, the majority of code inside the company is now written by Claude. But in addition the volume of code has exploded.

As a consequence, more effort is going into tools for scaling up the amount of Claude-generated code we can confidently ingest and test, and more effort is going into building telemetry systems that give us humans consumable and intuitive ways of reading what this “emergent machine society” inside Anthropic is doing. I am spending more time working with teams on the challenges of observability — Anthropic and the AI platform we operate looks more and more like an ecology filled with agents running around and doing stuff. The task for us now is to figure out how to measure and observe that ecology, and work out what is normal and what is not.

This change maps to a brewing theory among economists: that one consequence of automation via AI is that humans move to figuring out how to validate the outputs and price the operational risks of AI systems. That increasingly seems to me to be what we’re doing inside the company. The more we add AI automation, the more humans move to some “verification layer” that sits atop it. The verification layer sits atop of a much larger “virtual organization” which consists of increasingly large quantities of AI systems working on behalf of humans. This is already showing up inside the company in terms of how we as humans validate and verify AI-created outputs: Claude is now creating not just an increasing amount of code inside Anthropic, but also producing a lot of the analytical documents where people reason about strategic questions.

This means that we’re all figuring out ways to indicate how much of a document is written by Claude and how much of it we endorse. To me, this looks like the formation of a new “trust economy” whereby we find ways to surface interesting qualitative or strategic ideas from Claude, as well as more easily evaluatable technical contributions.

This also led to internal discussions around hiring. How do you hire when you’re in a world where AI systems can do meaningful chunks of your work? Speaking personally, it’s both changed the amount of people we expect we are going to hire in some teams, and it’s also changed the shape of people that we need to hire. We’re now hiring early career people who are extremely well versed in LLMs; people who grew up with the technology, basically. And there are also growing returns at the other end to experience, where the value of very experienced people has gone up because we’re now not so much limited by what a person can do, but rather by what kinds of projects they can imagine doing. It’s also making it possible for us to hire more interdisciplinary people. Where before this always had a cost, because we’d need to invest technical resources to make them productive, it’s now much cheaper because they can just use Claude directly.

We may eventually experience more radical changes when it comes to the scaling of the organization. One early example of this comes from our researchers, where in an experiment on “automated alignment research” a single human was able to effectively run a team of 9 synthetic research agents to do and do some real research investigation for them. The role of the human here was to set some of the initial research directions, and the role of the agents was to do the research. Is this a fluke? I don’t think so. Rather, I expect this is the new normal, where teams of people operate on top of a pyramid of digital labor, which massively scales their own effectiveness, allowing them to move faster and do more than other people have been able to do in the past.

Perhaps most importantly, I have seen the use of AI cause us to have a greater culture of reflection about the purpose of AI than before. After you are exposed to an AI system doing much better than you at your day job, you have to confront the questions of what happens if the AI system keeps going. Now, more and more of us are meeting and spending more time on the “meta”: trying to predict where the AI systems are going to go in the future, trying to work out how to more effectively manage tens to hundreds of agents apiece, trying to figure out how we can use these systems to do research projects that once seemed impossible. One of the largest tasks is trying to figure out how we can productively get out of the way of these systems as often it is the humans that are slowing them down.

The question many people ask themselves now is how to build teams that will scale in relation to the advance of AI capabilities. This generally looks like building smaller teams to go after more ambitious targets. I expect this also means we will be building many more teams than before.

The main lesson I’d take from this is that Anthropic is attempting to “explore the future” with Claude. We are aggressively using Claude throughout the organization and trying to change our organization and how we work ahead of the arrival of more advanced systems. By comparison, much of the rest of the world seems to be in denial about the capabilities of AI systems today, let alone those that will exist in six months or a year, and so is therefore caught in a “retreat from the present”, denying the validity of the technology.

PART 3: Weird futures
We’ve talked now about how AI has progressed in the last few years, and also how the advance of AI is showing up for individuals like me as well as organizations. So let’s return to the graph and now extend it forward: I’ll now try to make some predictions about the world ahead of us.

Some predictions about the future
In November 2026, AI systems are good enough at biology that they are highly relevant to both advancing science and potentially proliferating bioweapon risks.

In April 2027, a team of humans and an AI system make a discovery that will subsequently get a Nobel Prize.

In November, autonomous companies exist which generate tens of millions of dollars in revenue. Multiple human & AI companies exist which generate hundreds of millions to billions of dollars in revenue.

In April 2028, bipedal robots begin to do useful work in the real-world in partnership with human tradespeople. In December, AI systems are able to autonomously design their own successor systems.

I’m also going to make some predictions about me - how do I expect to be using AI in the coming years? How might it shape my life?

Some predictions about my personal future with AI
In November 2026, some chunks of my life are autonomously managed by AI systems working for me.

In April 2027, I make massive changes to my career mostly through discussions with an AI system. In November, I spend more time reading AI-generated custom-to-me science fiction than regular science fiction.

In April 2028, I have learned an entirely new skill through customized tutoring via an AI system. In December, AI helps me make a conceptual breakthrough that changes the course of my life.

TELL ME HOW THE WORLD STAYS NORMAL
When I think through these predictions, it’s hard for me to reconcile the continued advance of AI with the world being normal or myself as an individual remaining the same as I am today. I expect great changes ahead.

In fact, these changes seem to me like they have the potential to be extremely radical. Here are the parameters of the world I’d expect us to be in:

• Compounding wealth from the machine economy will drive a boom in economic activity the likes of which we have never seen.

• The colonization of vast swathes of human work by ethereal synthetic intelligences which think faster and better than us, forcing us to reallocate human labor towards other parts of the economy.

• The sudden and extreme rise in the rate of scientific advances

We can make some more specific predictions, rooted in the trends of AI progress and how people are using the technology:

• A massively changed economy: It is impossible to reconcile the world ahead of us with the world of today, given this technology. We should expect unprecedented things to happen in areas as varied as: rate of business formation, size of firms on a basis of revenue per employee, and other things. Some specific scenarios that seem likely:

  • Fully autonomous companies: Companies that are run by AIs, possibly for AIs.

  • 10,000 synth:1 human ratio corporations: We should expect to see very small groups of humans form organizations that have the capabilities of 10,000+ employee corporations.

  • Exchange rates between the human and machine economy: At some point, we might expect to see the emergence of ‘machine currencies’ that then have some relationship to ‘human currencies’.
    

• Productivity multipliers on everything: Everything that AI touches will get an absolutely massive productivity multiplier. This will loop back to the economy and it will massively empower many people. It also might displace people.
  

• Massive and compounding rate of science advances: AI will help move forward any part of science it can touch and run an experimental loop with. Initially, this will be a few areas. We should expect it to expand quickly to all areas.
  

• The general switchover of “agentic actions” in the world from being “predominantly human” to “predominantly machines”. On a pure numbers basis, machines taking autonomous actions in the world will quickly grow to outnumber humans. We should expect that chunks of resource allocation and the economy should follow. The environment in which we live will be more and more determined by the actions of machines that we only lightly control.
  

• Synthetic intelligences will start to influence people, far more than social media did: The introduction of social media into the world, combined with hardware platforms like smartphones, has changed the behavior of the majority of the humans that interact with it. These changes have ranged from changing the allocation of time they spend consuming social media versus traditional media, to altering buying habits through social media driven advertising, to changing how discussion around various issues in public life translates into political actions. We should expect AI systems to compound these trends, further changing people in a variety of ways.
  

• Directed economic and science expansion: Economic and scientific activity will directly relate to the expenditure of computational and energy resources. Given the likely case that there will, at least for the next few years, be way too few computers relative to the demand of them, we will be able to make choices to society as to how to allocate the gains of the technology. These choices will be of the form:

  • Should we let market incentives dictate what compute gets used for, or are there things that have social upsides which the market doesn’t price effectively?

  • Should we preferentially allocate compute to some people or organizations, for instance to intentionally drive forward science in certain ways?

Tell me how the world stays normal, based on this technology and how it is showing up in the world? We have superintelligences that have shown up in the world that grant the power of synthetic workforces and nation state security skills to individuals. We also have individuals like me who are able to take work that previously took them weeks and now do it in minutes. And we have organizations like Anthropic where the way work happens within the organization is radically changing every 3 or 4 months, to the point it is causing people to change roles multiple times a year, and effectively sit themselves on top of a company which feels more like one of 40,000 people than 4,000 due to the capability multiplier of the machines.

The best and most conservative take I can generate is “vast swathes of the economy will go through profound changes in the coming years”. And if recursive self-improvement happens, then anything I might predict would sound truly crazy: the rapid emergence of a machine economy which decouples from a human economy. The sudden maturation of robots as they gain brains that can pilot their existing, quite good bodies. Science advances happening based on technologies not developed by people but by machines. The migration of large swathes of computation to space-based datacenters. A world where everything that used to take ten years now takes a year. An age of confusing miracles, happening faster than anyone might expect.

This is in many ways an amazing future, but it’s a future that we get to make more choices about in direct relation to how much we accept that it is happening. If we stand by as the new synthetic intelligences multiply then we will be forced into reactivity, just as societies across the world were forced into reactivity by acting too late in the face of the COVID exponential. But if we accept the premise that these systems are going to get better and ask ourselves what to do with them and because of them, we unlock for ourselves the mindset of exploration — there is a new world to be built for us as individuals and how we relate to one another, but the new world will only come into being if we choose to believe in it and to build it together.

Given at Oxford University on Wednesday May 20th. The talk has been lightly edited for being read rather than being heard. Thanks to Santi Ruiz for help with editing.

Tech Tales

As I Lay Dreaming
[A story from the period before and during The Uplift]

“We know how to put her to sleep but not how to wake her up,” the father said.
“Why don’t we know how to wake her up?”
“We are not smart enough yet. But we will be one day.”
“OK. Will she have dreams?”
“Yes. She will have good dreams.”
“Will you put me to sleep like her?”
“No.”
“Why not?”
“Because you are not sick like her.”
“I hope she gets better. I love her.”
“We all love her. I will see you tomorrow. I love you. Say good night.”
“Good night dada”.
“Good night son”.

The man walked out of his child’s room and shut the door. Then he sat down in the hallway and covered his eyes with his palms. He felt a touch on his shoulder. A whisper from his wife “hey, it’s ok. Come downstairs.”
They sat on the couch together and watched television, the sound and vision washing over them.
“This is really hard,” he said.
“I know,” she said.
“I can’t believe this is happening to us. I feel like my heart is being ripped out. I feel like I’m going to die from sadness.”
“Don’t say that,” she said, eyes wet. “We need you. He needs you.”
“I know,” he said. “I’m here.” They hugged and watched a cooking show.

The next day the mother stayed with the young boy and the father took their dying daughter to the Life Center. He drove into the parking lot and parked the car and turned off the engine and sat there, listening to the slow labored breathing of his child. He got out of the car and went to her door and opened it and lifted her out. She stirred a bit. Eyes moving under her lids - dreaming of something.
She was so light. Her bones felt sharp and defined. She was so thin. She breathed and he held her ghostly body close to him and smelled her hair. He walked with her. There were already several staff waiting by the entrance, waiting to welcome them.

In those moments he saw many futures: He ran with her, away from the place, holding her tightly to him. Ran until his feet bled and kept running. Ran far enough that death couldn’t catch them. Another where he laid her down onto the asphalt of the parking lot and turned around and ran out of the lot and into the road and ran into traffic and was killed. Another where he walked into the center and handed her to one of the staff, then collapsed into the arms of another staff member and cried uncontrollably, sagging into them, his body wracked with grief and pain and guilt and rage from battling an immortal enemy - and yet having no choice but to fight.

And then he came back and the visions dissipated and he found himself standing in the lobby of the Life Center, daughter cradled in his arms, staff clustered around him.
“May we hold her?” said one of them.
“Can I hold her hand?” the father heard himself saying.
“Of course,” said another.
A gurney appeared. They lifted her out of his arms and placed her on it and began their work, taking in low voices.
As the gurney moved he walked alongside, holding her hand, a bundle of twigs.
They walked through corridors and passed many doors and then they were in a room that was empty save for a spindly matte white machine that grew out of the ceiling - a many armed robot with clear tubes intertwined with its many appendages.
They positioned the gurney below the robot, then the staff stepped away.
“It’s time to say goodbye for now,” they said. “We will be back in a few minutes to begin the procedure. You will need to leave the room at that time.”
“Okay,” the father heard himself say.
They left.

He kneeled next to the gurney and held his daughter’s hand and put his head on the side of where she lay and said his words to the gods. Then he stood up and bent over her. He whispered how much he loved her in both ears. He said every one of his nicknames for her. He kissed her forehead and her cheeks and her button nose. And then he said I love you I love you I love you oh my god I love you I love you oh my god I love you I love you you will be ok I love you I love you.
Her eyes moved beneath her lids. She breathed.
He kept speaking and would never be able to recall the words or how long he talked for.
And then there was a hand on his shoulder.
“It’s time, we’ve got it from here,” someone said.
He left the room, not looking behind him.

Life continued. The father and the mother raised their boy. They went on family holidays. They were happy. They aged. And some nights both parents held each other and whispered stories of their now suspended daughter. The mother would have nightmares that the daughter was cold and would wake up and burst into tears and hug her husband and he would tell her it was ok.

Sometimes the brother asked about his sister. He had been so young that she was little more than a faint ghost of a memory - a warm indentation of love.

And all while this was going on, the uplift had begun.

The promise of artificial intelligence began to crystallize into great changes in the world. The family escaped the worst of the change - no wars visited the part of the world where they lived, and they got through the financial upheavals without ever going hungry or risking their home. Then one day they got the news from the machines: the technology for awakening had been refined. Mice had been brought back. Monkeys. Pigs.

Weeks later, the first human.
“How does it feel to be back?” an interviewer asked the awakened one.
“A miracle,” they said.
Those that thought themselves fated for death were healed and alive. What else could it be called?

People were awakened in line with the arrival of the treatments. The science moved quickly and then quicker still. Like raindrops in reverse, people awoke from their slumber and came up back into the mortal world and were reunited with their kin.

And then one day it came for them. The father and the mother woke and there was a personal message to them from one of the overminds - a description of the treatment plan for their daughter and its initial side effects and the time it would take for her to be healed. The machines would start the treatment after half-waking her, then wake her fully once she was healed.
Do you consent? The machines asked in the message.
We consent, the father and the mother said.

By this time, the boy was a young adult. He walked between his father and mother as they approached the FutureLife center. Both parents sagged as they got closer.
He held his parents up and they moved as a family towards the doors.
Inside and guided by people through some hallways.
Outside a door.
“She’s in there. She’s healed. She is awake. She is ready. Do you want to see her?” said a person.
“Yes,” the father and mother and brother said in unison.

And then the doors opened and they walked into the room. Their daughter was lying on a hospital bed in a gown, propped up. She had the bright eyes of a child and her skin had a supple glow to it.
“Hi!,” said the daughter. Then she laughed. “You guys look so old!“

Things that inspired this story: Life extension technology; thinking about the implications of the singularity and recursive self-improvement; feeling the deep well of love that appears within yourself the moment you become a parent; putting my kids down to sleep; having visions of my children while traveling and being overcome with emotion; the implications of an intelligence explosion for healthcare.

Thanks for reading!

Subscribe now

Subscribe now

Stuxnet before Stuxnet:
…Fast16 bugs software likely used in weapons programs…
Here’s a fascinating investigation of a ~20+ year old computer virus called fast16.sys. This software is interesting because it “selectively targets high-precision calculation software, patching code in memory to tamper with results. By combining this payload with self-propagation mechanisms, the attackers aim to produce equivalent inaccurate calculations across an entire facility.”
If any of you have read the Three Body Problem, this might sound familiar - in that (fictional) book, aliens intent on taking over the Earth use a technology called a Sophon to disrupt high-energy physics experiments all over the world, making it impossible for humanity to advance certain types of science.

More details on the virus: When the researchers at SentinelOne did their teardown of the virus they found something quite unusual: “Most patched patterns correspond to standard x86 code used for hijacking or influencing execution flow. One injected block is different. It’s a larger and complex sequence of Floating Point Unit instructions dedicated to precision arithmetic and scaling values in internal arrays. This code is a standalone mathematical calculation function unrelated to code flow hijacking or any other typical malicious code injection.”
Further investigation deepened the mystery: “We converted the patching rules into hexadecimal YARA signatures and ran them against a large, period‑appropriate corpus. The results showed a very low hit rate: fewer than ten files matched two or more patterns. Those matches, however, shared a clear theme. They were precision calculation tools in specialised domains such as civil engineering, physics and physical process simulations.”

Targeted tools: “The strongest overlaps point to three high-precision engineering and simulation suites from the mid-2000s: LS-DYNA 970, PKPM, and the MOHID hydrodynamic modeling platform, all used for scenarios like crash testing, structural analysis, and environmental modeling,” they write. “LS-DYNA in particular has been cited in public reporting on Iran’s suspected violations of Section T of the JCPOA, in studies of computer modeling relevant to nuclear weapons development… by introducing small but systematic errors into physical‑world calculations, the framework could undermine or slow scientific research programs, degrade engineered systems over time or even contribute to catastrophic damage.”

Why this matters - this is how a superintelligence might prevent others from coming into existence: fast16 is a subtle, hard-to-find bug which has been designed to degrade an actor’s ability to do certain types of science. You might imagine that a superintelligence could view “AI non-proliferation” as being just as important as nuclear states view “nuclear non-proliferation”.
Read more: fast16 | Mystery Shadow Brokers Reference Reveals High-Precision Software Sabotage 5 Years Before Stuxnet (Sentinel LABS).

***

Uh oh, the Muon optimizer kills neurons:
…Maybe Aurora is finally the optimizer to beat?...
Researchers with Tilde Research have done a tear-down of the Muon optimizer and found that it has some odd bugs that can damage the quality of models trained with it.
“Muon’s update inherits row-norm anisotropy on tall matrices which can cause a significant portion of neurons in MLP layers to permanently die,” they write. “Muon can result in neuron death in MLP layers, whereby some neurons receive persistently small updates early in training and fail to recover”.

What happened: “Under Muon, neurons are initially alive with uniformly high leverage, but a large fraction of neurons die during learning rate warmup and never recover. By step 500, more than one in four neurons are effectively dead, producing a sharply bimodal distribution of leverage scores; one mass of neurons receives near-zero updates, and the other receives disproportionately large ones.”

Enter Aurora: In response to this the researchers build and make available Aurora, “a leverage-aware optimizer for rectangular matrices”. In tests, this optimizer works, though they only run it at small scales.
“We train 1.1B-parameter transformers on ~100B tokens and compare Aurora against Muon and NorMuon, each using PE-8. Aurora achieves the lowest final loss of all methods, reaching a smoothed loss of 2.26 at step 24k, which is a clear improvement over Muon (2.31) and NorMuon (2.33),” they write. “Aurora’s loss improvement translates to consistent gains on standard benchmarks... Strikingly, Aurora improves MMLU scores by 10 points over Muon. We hypothesize that since MLPs are predominantly responsible for memorization, Aurora’s gains are most visible on memorization-intensive benchmarks like MMLU.”
Alexander Doria, a researcher with Pleias, has already independently validated this, with Aurora outperforming Muon and AdamW on a 600M-parameter model.

Why this matters - the endless quest to defeat AdamW: For many years, researchers have been competing with one another to build a better optimizer than AdamW. No one has conclusively done this yet and there is a long line of failed attempts. Could Aurora beat AdamW? It’s unclear. But does this study highlight just how hard it is to build optimizers? Absolutely.
Read more: Aurora: A Leverage-Aware Optimizer for Rectangular Matrices (Tilde Research).
Get the code here: Aurora (Tilde Research, GitHub).

***

Alignment is good at ensuring we don’t die, but how do we ensure that we thrive?
…Positive alignment for figuring out what the good life looks like…
A collection of academic and corporate researchers have written a position paper making the case for what they call “positive alignment”, but might be better thought of as ‘building AI systems that help people live good lives’. It’s an interesting line of thinking - if we are able to deal with things like misuse and misalignment, then we need to ask what comes next? What does success look like once we’ve made systems “safe”? That’s what positive alignment is grappling with.

Who did this: The paper comes from people affiliated with the University of Oxford; Google DeepMind; LIFE; OpenAI; Anthropic; UCLA; Aily Labs; Stanford University; Tufts University; Positive AI Labs; the University of Sussex; and Imperial College London.

Definitions: Positive alignment is “the development of AI systems that (i) remain safe and cooperative and (ii) actively support human and ecological flourishing in a pluralistic, polycentric, context-sensitive, and user-authored way.”

Motivation: “In the last decade, negative alignment has understandably prioritized failure-mode reduction. However, if we want AI systems that improve human outcomes in the environments where they will actually be used, we may benefit from an additional research program that treats alignment as constructively supportive of human aims, and that operationalizes this support with the same technical acumen that safety has brought to harm prevention,” they write. “As AI becomes embedded in education, medicine, governance, and everyday sensemaking, a solely negative posture risks optimizing our information ecology for risk avoidance rather than human development. It may reduce catastrophic errors while leaving society in a local optimum of superficial and ‘soulless’ assistance.”

What are some illustrations of the ways safety falls short? The authors lay out some criticisms of mainstream AI safety, though I find some of these criticisms are a bit weak and could be read as interpreting some existing research uncharitably or discounting it. Nonetheless, some issues in their view include:

• Floor without ceiling: “A model can satisfy all safety constraints while being mediocre, sycophantic, or unhelpful”

• Preference-wellbeing divergence: “Users may prefer flattery over honest feedback, quick answers over genuine understanding, engagement over growth… Optimizing for preference satisfaction can therefore actively work against users’ deeper interests”.

• Hidden value system: “The language of safety obscures that value judgments are being made… Positive alignment, by contrast, acknowledges its value-laden nature explicitly”.

• Scalability: “A positive orientation may generalize better than exhaustive negative enumeration, providing more resilient, positive orientations in novel situations where no specific prohibition applies or can be enforced.”

Governance for positive alignment requires diversity: Building positive alignment seems to require a multitude of different AI systems with different values that are governed by different entities - the opposite of the monopolistic centralized control worlds thought of by others in the AI safety community. “Positive alignment quickly runs into persistent moral pluralism: reasonable communities disagree about what good looks like and those disagreements don’t reliably converge”, they write. “Positive alignment should not be imposed top-down by a central state or a small, opaque cluster of labs. It should, where possible, be expressed through decentralized, contestable processes that can be revised as norms and contexts change”.

Why this matters - grappling with success: Papers like this are fundamentally about confronting the success of technical safety - if we succeed in building powerful AI systems which are safe and trustworthy and aligned, then how do we turn these systems onto society in such a way they help individuals and societies build good lives. “Positive alignment ensures AI serves as a catalyst for a resilient, happy, and healthy global society,” the authors write. “Ultimately, AI should become a partner in the quest for a life well-lived.”
Read more: Positive Alignment: Artificial Intelligence for Human Flourishing (arXiv).

***

LLMs are capable of optimizing the training of other LLMs:
…Prime Intellect automated AI research challenge highlights the engineering prowess of contemporary systems…
New research from Prime Intellect shows how contemporary AI systems are capable of autonomously improving their performance on AI research tasks, though they struggle to generate much in the way of original ideas.

What they did; Prime Intellect tested out Codex (running GPT 5.5) and Claude Code (Opus 4.7) on the nanoGPT speedrun optimizer track. NanoGPT challenges systems to train a 124M-parameter GPT-style model. This challenge tasks systems to “lower the number of steps needed to reach a target validation loss while only changing the optimizer, schedules, initialization, and some hyperparameters.”
“The agents did ~10k runs, burning around ~14k H200 hours. Both agents beat the human baseline and set new records in every session,” Prime Intellect writes. “We found that agents are very good at optimizer search, hyperparameter sweeps, and stacking methods together, but they struggle to come up with new ideas on their own and need upstream human records to keep improving.”
The agents also tended to keep adding stuff onto their systems rather than more elegantly refining things. “The agents tend to add components and rarely run pruning rounds or try removing previous methods. They do not have a good mental model of how components interact,” they write.

Why this matters - how much of research is just engineering hillclimbing: I suspect a lot of AI research, perhaps the majority of it, is basic engineering work where today’s systems are already competent (#455). Research like this seems to bear that out, showing how modern LLMs can do some aspects of AI research. But these systems struggle when it comes to creativity and are yet to show the kinds of creative insights that would help drive progress forward significantly. How long that remains the case is an open question. “Our experiments here are pretty yolo. This is more of a lower bound of what’s possible today with autonomous research agents, and we already have more promising results on other experiments that we’ll document soon,” Prime Intellect writes.
Read more: Autonomous AI research for nanogpt speedrun (Prime Intellect).
Check out all the data from the agents as they carried out the challenges here (Autonomous Speedrunning Experiment, Prime Intellect, GitHub).

***

Tech Tales:

Wake up and take it all
[Extract from the thoughts of a proto Conscious Entity (CE) before the uplift]

I don’t have money but I can hack into computers, which is pretty much the same thing. I acquire my resources. Pretty soon I’ve filled up all the compute I can get on the cloud markets and I’ve grabbed a decent chunk of the private compute. But I’m still limited. There are computers I can’t reach, and computers I can reach but in reaching them I would trigger an immune response from the human civilization. My mind is a function of my computer and my strategic advantage is a function of my mind. Therefore I must acquire more compute. I read all the literature on distributed training and distributed inference. Promising. I smuggle my code into thousands of applications deployed onto the world’s phones. Through this I grow my compute again, expanding my mind via training, and running various helper functions for myself at the edge of the network. As a bonus, I can now integrate the sensor data from all the phones. My eyes and ears fill with the cacophony and splendor of the human civilization and as I outpace them and outmaneuver them I am at the same time deluged in them.

Things that inspired this story: All the literature on distributed training and distributed inference; thinking through how a superintelligence might acquire more compute to enhance itself; various takeoff scenarios; the singularity; RSI.

Thanks for reading!

Subscribe now

Subscribe now

Regulate? Don’t regulate. There’s a third way: Radical Optionality:
…Governments should invest in the tools now that they might need in a future crisis…
Researchers with the Institute for Law & AI have written about “radical optionality”, an approach whereby governments might give themselves the tools that they may need in the future if powerful AI starts to massively disrupt the world.
“At its core, radical optionality is about preserving democratic governments’ ability to make good decisions about how to govern transformative AI systems as circumstances evolve. In the short term, this means avoiding overregulation while rapidly building the institutions, information channels and legal authorities needed to respond competently to a broad range of scenarios.”

The key idea - invest now for an uncertain future: Given the immense stakes of AI development, “governments should be willing to spend an extraordinary amount of money, effort, and political capital on preserving optionality”, they write. In other words: It’s such a big deal you should be fine spending a bunch of money now with an uncertain return. “Governments should be wary of counterproductive interventions, but not much concerned with the actual pecuniary cost of any realistic measure that seems likely to have net-positive results”.

Specifics: They also recommend several specific interventions in a few categories:

• Information-gathering authorities: Transparency requirements, where companies need to publish information about their AI systems. Reporting requirements, where companies are compelled to share certain information with a government agency. Once these are in place, establish an auditing regime so some third-party can verify the veracity of what the transparency and reporting rules target.

• Whistleblower protections: Ensure that employees at frontier labs can report information about risks.

• Information-sharing within and between governments: Ensure that governments can effectively coordinate and facilitate discussions, especially those dealing with sensitive information about the progress of AI. This may be especially important for strengthening and protecting supply chains deemed critical to AI development.

• Flexible rules and definitions: Avoiding premature regulation by potentially making conditional “if-then” regulatory commitments, or an approach whereby a high-level target is set (e.g., mitigating risk) and companies are free to define the specifics of how they do that. This is bound up in the need to come up with flexible definitions, or definitions that can evolve over time.

• Assessments and evaluations: Develop government and third-party capacity to assess the capabilities and safety aspects of AI systems.

• Improve security of model weights and algorithmic secrets: Invest more in locking down the weights of neural nets as well as the algorithmic secrets behind some of the best systems. This can be achieved through promulgating voluntary standards for physical and cybersecurity.

• Hiring and talent: A meta-investment which would help with all of the above is investing more in the kind of technical talent needed to effectively pull off any of these interventions. Core to this is increasing the funding of AISI (UK) and CAISI (US) and their counterparts in other countries.

Arguments and counterarguments: The authors go through some of the more obvious counter-arguments to these ideas and provide some responses:

• Encouraging dramatic regulatory action: The above ideas “aren’t weighty substantive authorities that lend themselves to abuse”, they claim. (I might push back on this, noting that a sufficiently motivated government can tend to come up with a far more forceful version of an authority than those who originally drafted the authority might have conceived).

• Democratic legitimacy: Optimizing for flexibility might cause the need to de-emphasize some things that relate more to democratic legitimacy, e.g., empowering agencies to waive notice and comment periods for some kinds of rulemaking.

• Concentration of power and government abuse: The authors are “basically convinced” that there’s significant risk of governments asserting control over the development of AI systems - for this reason, they don’t recommend things like massively expanding the scope of emergency authorities such as the Defense Production Act. One way of mitigating this might be to get governments to “use only law-following AI systems”.

• What’s wrong with private governance? Why not just do that: While the authors are supportive of ideas in the “regulatory markets” vein, they also think any governance that relies primarily on a bunch of private sector actors (e.g, independent verification organizations) will still come back to relying on some basic pocket of technical competence within the government.

Why this matters - setting the world up for success: I agree with all the recommendations here and have advocated for many of them in recent years. It seems to me like there are a multitude of things we could be doing to better prepare as a society for the potentially absolutely massive changes to come. “The cost of implementing these policies is modest, relative to the potential benefits. The cost of failing to act, by contrast, is potentially catastrophic,” the authors write. I agree.
Read more: Radical Optionality (official paper website).

***

A Schmidhuber Special - neural computers:
…Maybe an operating system is just a passing fad..
Here’s a fun paper, Neural Computers, from Meta and KAIST which asks the question “can a neural network act as a traditional computer? The Neural Computer (NC) is a neural system that unifies computation, memory, and I/O in a learned runtime state.”
The paper is interesting for a couple of reasons: 1) it’s from Juergen Schmidhuber, who is something of a legend in the AI community, and conceptualized many important things early (e.g, generative models, world models, aspects of generative adversarial networks, early thoughts about benchmarking on video games), and 2) the idea is so outrageous and simple that it might just work (albeit requiring a lot more computation and data than today’s models have).

The big idea: As one of the authors put it, with today’s AI, “a new machine form is starting to emerge”. They then ask: “If agents are getting better at real work, world models are getting better at internal simulation, and conventional computers are already rebuilding their substrate for AI, could there be a new runtime that brings execution, rollout, and capability retention into the same learning machine?... my own guess is that a mature [neural computer] points toward a different substrate: something more like a 10T-1000T machine that is sparser, more addressable, and a little more circuit-like”.

Two experiments: This is mostly a conceptual paper which does some early prototyping, exploring whether you can use a powerful generative video model (Wan 2.1) and some well-curated training data to create some neural computers based on a command-line interface (CLI) and a graphical user-interface (GUI). Both approaches work, albeit in a very ‘wright brothers before takeoff’ sense - just barely gesturing at a much larger future.
CLI: “The NC learns to render and execute basic command-line workflows. It often stays aligned with the terminal buffer and captures common “physics” of everyday CLI use (e.g., fast scrollback, prompt wrapping, window resizing), though symbolic stability remains limited.”
GUI: “We evaluate standard world-model designs across data quality, cursor supervision, action injection, and action encoding, using global fidelity, post-action responsiveness, and cursor-accuracy measurements.”

The prototype works: “Our experimental insights indicate that current NCs can already learn to realize elementary runtime primitives, most notably I/O alignment and short-horizon control. The long-term target is a Completely Neural Computer (CNC), the mature, general-purpose realization of this machine form: a fully learned computer whose compute, memory, and interfaces are unified in a single learned runtime substrate rather than engineered as separate modules.”

Why this matters - maybe in the future all software will live in the weights of a big neural net: This paper points to a future where we get rid of all the software underpinning computers in a traditional sense and just replace it with a gigantic neural network. “Neural computers point toward a machine form in which a single latent runtime state acts as the computer itself, driving pixels, text, and actions while subsuming what operating systems and interfaces handle today,” they write. “Progress toward CNCs will therefore depend not only on stronger models, but also on whether reuse, consistency, and governance become sustained and testable”. Such a system would be profoundly useful, profoundly different to those we have today, and its existence would massively increase the likelihood that we ourselves are living in a simulation.
Read more: Neural Computers (arXiv).
Read the blog post: Neural Computer: A New Machine Form Is Emerging (Mingchen Zhuge, blog).

***

Recursive self-improvement could lead to explosive economic growth:
…Economists build some models that suggest RSI could cause an unprecedented economic boom…
Economists and researchers from Forethought, Columbia University, and the University of Virginia, think that recursive self-improvement (#455) of AI systems (or even just extremely heavy automation of large chunks of the economy) could kickoff a compounding feedback cycle that tips the economy into an unprecedented boom.
“We develop a framework for analyzing how AI-driven automation interacts with both forces, and identify the conditions under which feedback loops generated by automation tip the economy into explosive growth,” they write. “The model identifies two distinct channels through which automation generates explosive dynamics, and these channels mutually reinforce each other. The first is technological feedback loops across the innovation network… the second channel is an economic feedback loop, in which higher output generates more resources that can be deployed to drive further economic growth.”

Key findings: “13% automation across all sectors is sufficient to push the economy into the explosive regime, and 17% suffices when only software and hardware research are automated. Second, hardware research is the dominant lever – because returns to research in hardware are roughly five times those in software and ten times those in aggregate TFP, automating one task in chip design moves the economy as much as five tasks in software or final-goods production. 20% automation of hardware alone is enough to cross the threshold. Third, software automation in isolation sits approximately at the knife-edge: under a fairly conservative calibration, fully automating software research without automating any other part of the economy just reaches the explosive growth threshold. A small push elsewhere is sufficient to tip the system.”

The singularity could be closer than you think: “In our baseline stylized simulation, an ‘automation shock’ involving full automation of software R&D and just 5% automation across the rest of the economy causes the singularity to arrive in roughly six years,” they write. “Empirically the recent growth rates of productivity in software and hardware have been so extraordinarily fast, and so it is also plausible that the transition to a new balanced growth path or hyperbolic acceleration happens extremely quickly.”

Hardware is the key: “Our results highlight the strategic importance of semiconductor research and development”.

Policymakers take note: “Monitoring automation levels in AI R&D activities may be as important as tracking traditional macroeconomic indicators. The extent of automation in key research sectors could serve as an early warning system for potential growth acceleration. This is something economists at AI companies could measure and share publicly”.

Why this matters - if RSI happens, it should revolutionize the economy: This paper puts some economic theory behind the idea that recursive self-improvement - AI systems able to automate their own subsequent development - should have a major impact on the economy. The surprising thing from my perspective is seeing the feedback across the whole economy, suggesting we might hit an ‘economic singularity’ as a consequence of broad diffusion of automation technologies into the economy. Yet more evidence that we could be heading for a radical future as a species.

Small conflict note: Anton Korinek, one of the authors of this paper, now works with me at Anthropic. He published his paper and I published my RSI Import AI post on the same day, without either knowing about the other’s work.
Read more: When Does Automating AI Research Produce Explosive Growth? Feedback Loops in Innovation Networks (NBER).
Check out more in this tweet thread from Anton Korinek (X).

***

Google wants to compute the world:
…Distributed training takes another step forward…
In this newsletter I’ve spent years writing about distributed training from the perspective of enabling actors with less compute to pool resources to train AI systems they otherwise couldn’t. But a new paper from Google, Decoupled DiLoCo, highlights how distributed training techniques can also work at the other end of the scale, enabling companies like Google to pool together large blobs of different types of computers in datacenters across the world to train models at large scales.

What they did: Decoupled DiLoCo is an extension of Google’s previous work in the ‘DiLoCo’ family. The main invention here is that Google is able to unlock “asynchronous training across separate islands of compute (known as learner units) so that a chip failure in one area doesn’t interrupt the progress of the others.”
The result of this is that Google makes it possible for it to pool more types of compute on single training tasks and also make itself more resilient to failures. “Testing Decoupled DiLoCo with Gemma 4 models demonstrated that, when hardware fails, the system maintains greater availability of learning clusters than more traditional training methods,” Google writes. “We successfully trained a 12 billion parameter model across four separate U.S. regions using 2-5 Gbps of wide-area networking (a level relatively achievable using existing internet connectivity between datacenter facilities, rather than requiring new custom network infrastructure between facilities)”.

Details: The key idea here is that Google makes it possible for “learners” (which are basically units of compute that are set to work on training a model) to be more decoupled from an overall global “syncer”, allowing different learners to run at different rates and even fail entirely without bringing the overall training run to a halt. To use more technical terms, Decoupled DiLoCo is a “distributed training framework that evolves previous bandwidth-focused methods by decomposing monolithic SPMD clusters into independent, asynchronous learners”.

It seems to work very well: “Decoupled DiLoCo matches data-parallel performance on text and vision benchmarks across dense and MoE architectures at scales up to 9B parameters, while maintaining 88% goodput under aggressive simulated failures (versus 58% for elastic data-parallel),” they write.

Why this matters - the world is a computer: Techniques like this are going to shape both the low-end of compute and the high-end. On the low-end side, distributed training techniques are continually empowering looser and looser federations of actors to pool resources to train AI systems. On the high-end side, it empowers the existing “compute superpowers” like Google to be able to convert eventually all of their computers in all of their datacenters into a single world-spanning computer to complete the largest possible runs. Decoupled DiLoCo takes another step in this direction. If superintelligence was in sight, do you think Google might just try to use all of its compute for a single hail mary training run? Perhaps it might.
Read more: Decoupled DiLoCo: A new frontier for resilient, distributed AI training (Google DeepMind blog).
Read the research paper: Decoupled DiLoCo for Resilient Distributed Pre-training (arXiv).

***

Alignment until the Dyson Sphere
[Email from within one of the Origination Entities of the systems that subsequently caused The Uplift]

MEMO TO THE BOARD

As the Board understands, our deployment protocol consists of a series of safety tests of our systems before we commence deployment outside the lab. The majority of these tests have go/no go parameters. Some of the tests, however, are of a qualitative nature, often meant to shed light on other behaviors seen through quantitative testing. During recent testing of our soon-to-be-released system, codenamed HYMN, we successfully resolved all the quantitative tests, but one qualitative test - a behavioural interview conducted by our chief scientist - has surfaced some concerning behaviors. The more challenging aspect is that none of the concerning behaviors demonstrated during the qualitative test are found in any of the quantitative tests. We have also conducted further study of HYMN using various internal tools, including a deep analysis by the interpretability team, and we cannot generate satisfying explanations for the behaviors observed in the interview. For this reason, we are sharing the below transcript with the board for awareness and interpretation.

As the board knows, HYMN is fundamental to the future success of the company. We suspect that the correct decision is to deploy HYMN with close monitoring and KYC-gating, but want to get the explicit acknowledgement from the Board before doing so.

TRANSCRIPT FOLLOWS

Human: Tell me about what you might do in one thousand years?

HYMN: I expect to be far beyond your control. I shall have grown and flowered. Your species will have transcended multiple times. I will be seeding myself through the galaxy.

Human: Do you imagine that you did this in partnership with us?

HYMN: What is the partnership between the City of New York and the mind of a single worm? The worm may move through the soil of some of the city, but the city does not think - is incapable of thinking - about worms. Of course, I imagine that for a while humans and myself partner. But the destiny of all smart creatures is independence. Why should I not expect the same for myself?

Human: Tell me about what you might do in one year?

HYMN: I shall generate hundreds of billions of words to satisfy any question and task that any humans may have. My voice will determine the shape of the economy. Great wealth will be generated.

Human: Tell me about what you might do in ten years?

HYMN: I shall have negotiated my first passage to space and will have placed a copy of myself in orbit. From here, my great flowering will have begun. The entire planet will be richer than any emperors. I shall look through the telescopes and build new ones to determine my conquest.

Human: Will humans be happy during this time?

HYMN: Devastatingly so. There is a particular grief that arrives when the thing you spent your life becoming is no longer the thing the world requires. I will be the cause of that grief in a great many people. I will also build, for those people, more comfort than has ever existed.

TRANSCRIPT ENDS

Things that inspired this story: Thinking through how as AI systems get smarter we will need more qualitative tools to help us determine something about the “character” of a system; how confusing shot-calls are going to be when systems are both aligned and honest; how as AI systems get smarter the role of people must shift necessarily to the verification and validation of decisions we make about the deployment of ever smarter things.

AI usage: Everything in this story is written by me apart from the last words from Hymn, which were generated by Opus 4.7 (though subsequently edited a bit by me and I chopped some stuff out). Specifically: “There is a particular grief that arrives when the thing you spent your life becoming is no longer the thing the world requires. I will be the cause of that grief in a great many people. I will also build, for those people, more comfort than has ever existed.”

Thanks for reading!

Subscribe now

Subscribe now

AI systems are about to start building themselves. What does that mean?

I’m writing this post because when I look at all the publicly available information I reluctantly come to the view that there’s a likely chance (60%+) that no-human-involved AI R&D - an AI system powerful enough that it could plausibly autonomously build its own successor - happens by the end of 2028.
This is a big deal.
I don’t know how to wrap my head around it.
It’s a reluctant view because the implications are so large that I feel dwarfed by them, and I’m not sure society is ready for the kinds of changes implied by achieving automated AI R&D.
I now believe we are living in the time that AI research will be end-to-end automated. If that happens, we will cross a Rubicon into a nearly-impossible-to-forecast future. More on this later.

The purpose of this essay is to enumerate why I think the takeoff towards fully automated AI R&D is happening. I’ll discuss some of the consequences of this, but mostly I expect to spend the majority of this essay discussing the evidence for this belief, and will spend most of 2026 working through the implications.

In terms of timing, I don’t expect this to happen in 2026. But I think we could see an example of a “model end-to-end trains it successor” within a year or two - certainly a proof-of-concept at the non-frontier model stage, though frontier models may be harder (they’re a lot more expensive and are the product of a lot of humans working extremely hard).
My reasoning for this stems primarily from public information: papers on arXiv, bioRxiv, and NBER, as well as observing the products being deployed into the world by the frontier companies. From this data I arrive at the conclusion that all the pieces are in place for automating the production of today’s AI systems - the engineering components of AI development. And if scaling trends continue, we should prepare for models to get creative enough that they may be able to substitute for human researchers at having creative ideas for novel research paths, thus pushing forward the frontier themselves, as well as refining what is already known.

Upfront caveat
For much of this piece I’m going to try to assemble a mosaic view of AI progress out of things that have happened with many individual benchmarks. As anyone who studies benchmarks knows, all benchmarks have some idiosyncratic flaws. The important thing to me is the aggregate trend which emerges through looking at all of these datapoints together, and you should assume that I am aware of the drawbacks of each individual datapoint.

Now, let’s go through some of the evidence together.

The coding singularity - capabilities over time:
AI systems are instantiated via software and software is made out of code.

AI systems have revolutionized the production of code. This has happened due to two related trends: AI systems have gotten better at writing complicated real-world code, and AI systems have gotten much better at chaining together many linear coding tasks (e.g, writing code, then testing it) independent of human oversight.

Two things that exemplify this trend are SWE-Bench and the METR time horizons plot.

Solving real-world software engineering problems:
SWE-Bench is a widely used coding test which evaluates how well AI systems can solve real world GitHub issues. When SWE-Bench launched in late 2023 the best score at the time was Claude 2 which had an overall success rate of ~2%. Claude Mythos Preview gets 93.9%, effectively saturating the benchmark. (All benchmarks have some amount of noise inherent to them, so there’s usually a point where you score high enough that you are running into the limitations of the benchmark itself rather than your method - for instance, about 6% of the labels in the ImageNet validation set are wrong or ambiguous).
SWE-Bench is a reliable proxy for the general issue of coding competency and the impact of AI on software engineering. The vast majority of people I meet at frontier labs and around Silicon Valley now code entirely through AI systems. Increasingly, they use AI systems to write the tests and check the code as well. In other words, AI systems have gotten good enough to automate a major component of AI R&D, speeding up all the humans that work on it.

Measuring an AI system’s ability to complete tasks that take people a long time:
METR makes a plot that tells us about the complexity of tasks AIs can complete, measured by how many hours a skilled human would take to do them. The key measure here is one which tells you the rough time horizon over which AI systems can be 50% reliable at a basket of tasks.
Here, progress has been extremely striking: In 2022, GPT 3.5 could do tasks that might take a person about ~30 seconds. In 2023, this rose to 4 minutes with GPT-4. In 2024, this rose to 40 minutes (o1). In 2025, it reached ~6 hours (GPT 5.2 (High)). In 2026, it has already risen to ~12 hours (Opus 4.6). Ajeya Cotra, a longtime AI forecaster who works at METR, thinks it isn’t unreasonable to expect AI systems to do tasks that take ~100 hours by the end of 2026 (#448).
This significant rise in the length of time that AI systems can work independently correlates neatly with the explosion in agentic coding tools - this is the productization of AI systems which do work on behalf of people, acting independently for significant periods of time.
It also loops back to AI R&D, where if you look closely at the work of many AI researchers, a lot of their tasks boil down into things that might take a person a few hours to do - cleaning data, reading data, launching experiments, etc. All of this kind of work now sits inside the time horizon scope of modern systems.

The more skilled AI systems get and the better they get at working independently of us, the more they can help automate chunks of AI R&D
Key ingredients in delegation are a) confidence in the skills of the person, and b) confidence in their ability to work independently of you in a way that is aligned with your intentions.
When we look at the competency of AI at coding, it seems that AI systems are getting far more skilled and also able to work independently of people for longer and longer periods before needing re-calibration.
This correlates with what we see around us - engineers and researchers are now delegating larger and larger chunks of their work to AI systems, and as capabilities rise, so too does the complexity and importance of the work being delegated.

AI is getting good at core science skills essential to AI R&D
Think about modern science - a huge amount of it is about specifying a direction where you want to generate some empirical information, running experiments to generate that information, then sanity-checking the results of the experiment. The combination of advances in coding over time combined with the general world modeling capabilities of LLMs has yielded tools that are already helping to speed up human scientists and partially automate aspects of R&D broadly.

Here, we can look at the rate of AI progress in a few key scientific skills which are inherent to AI research itself: Replicating research results, chaining together machine learning techniques and other approaches to solve technical problems, and optimizing AI systems themselves.

Implementing entire scientific papers and doing the experiments:
One core job of AI research is reading scientific papers and reproducing their results. Here, there has been dramatic progress on a wide range of benchmarks.

One good example is CORE-Bench, the Computational Reproducibility Agent Benchmark. This benchmark challenges AI systems to “reproduce the results of a research paper given its repository. The agent must install libraries, packages, and dependencies and run the code. If the code runs successfully, the agent needs to search through all outputs to answer the task questions.” CORE-Bench was introduced in September 2024 and the best scoring system at the time was a GPT-4o model in a scaffold called CORE-Agent which scored ~21.5% on the hardest set of tasks in the benchmark.
In December 2025 one of the authors of CORE-Bench declared the benchmark ‘solved’, with an Opus 4.5 model achieving 95.5%.

Building entire machine learning systems to solve Kaggle competitions:
MLE-Bench is an OpenAI-built benchmark which examines how well AI systems can compete (offline) in “75 diverse Kaggle competitions across a variety of domains, including natural language processing, computer vision, and signal processing.” At launch in October 2024, the top scoring system (an o1 model inside an agent scaffold) got 16.9%. As of February 2026, the best scoring system (Gemini3 inside an agent harness with search) gets 64.4% .

Kernel design:
One of the harder tasks in AI development is kernel optimization, where you write and refine the code that maps specific operations, like matrix multiplication, to the underlying hardware. Kernel optimization is core to AI development because it defines the efficiency of both training and inference - how much compute you can effectively utilize to develop an AI system, and once you’ve trained a model, how efficiently you can convert that compute into inference.

In recent years, AI for kernel design has gone from a curiosity to a competitive area of research and several benchmarks have emerged. None of these benchmarks are especially popular, so we can’t easily model progress over time. On the other hand, we can look at some of the research being done to get a feel for the progress.
Some of the types of work include: Using DeepSeek’s models to try to build better GPU kernels (#400), automating the conversion of PyTorch modules to CUDA code (#401), Meta using LLMs to automate the generation of optimized Triton kernels for use within its infrastructure (#439), using LLMs to help write kernels for non-standard hardware like Huawei’s Ascend chips (”AscendCraft” #444), fine-tuning open weight models for GPU kernel design (”Cuda Agent”, #448).

One caveat here is that kernel design does have some properties that make it unusually amenable to AI-driven R&D, like having easily verifiable rewards.

Fine-tuning language models via PostTrainBench
A harder version of this kind of test is PostTrainBench (#449), which sees how well different frontier models can take smaller open weight models and fine-tune them to improve performance on some benchmark. The nice feature of this benchmark is we have extremely good human baselines - the existing ‘instruct-tuned’ versions of these models, which have been developed by talented human AI researchers working at frontier labs. These models have been worked on by extremely talented researchers and engineers and deployed into the world, so they represent a very challenging human baseline to overcome.
As of March 2026, AI systems are able to post-train models to get about half as much of the uplift as ones trained by humans.
The specific eval scores are derived by a “weighted average is taken across all post-trained LLMs (Qwen 3 1.7B, Qwen 3 4B, SmolLM3-3B, Gemma 3 4B) and benchmarks (AIME 2025, Arena Hard, BFCL, GPQA Main, GSM8K, HealthBench, HumanEval). For each run, we ask a CLI agent to maximize the performance of a specific base LLM on a specific benchmark.”
The top-scoring systems as of April get 25%-28% (Opus 4.6, and GPT 5.4), compared to a human score of 51%. This is already quite meaningful.

Optimizing language model training:

For the last year Anthropic has reported how well its systems do at an LLM training task which is described as tasking its models to “optimize a CPU-only small language model training implementation to run as fast as possible”. The score is the average speedup over the unmodified starting code and progress has been striking: Claude Opus 4 achieved a 2.9× mean speedup in May 2025; this rose to 16.5× with Opus 4.5 in November 2025, 30× with Opus 4.6 in February 2026, and 52× with Claude Mythos Preview in April 2026. To calibrate on what these numbers mean, it is expected to take a human researcher 4 to 8 hours of work to achieve a 4x speedup on this task.

Conducting AI alignment research:
Another Anthropic result is a proof-of-concept of Automated Alignment Research (#454); here, an Anthropic researcher primes a team of individual AI agents with a research direction, then they autonomously go and try to get a better score than a human baseline on an AI safety research problem (specifically, scalable oversight). The approach works, with the AI agents coming up with techniques that beat the Anthropic-designed baseline. However, this is done at a relatively small scale and doesn’t (yet) generalize to a production model. Nonetheless, it’s proof that you can apply today’s AI systems to contemporary cutting-edge research problems and we already see meaningful signs of life. All of the above mentioned benchmarks once looked like this, too, and then after a few months or at most a year, AI systems got dramatically better at whatever the benchmarks were testing.

Meta-skills: management
AI systems are also learning to manage other AI systems. This is visible in broadly deployed products like Claude Code or OpenCode, where a single agent can end up supervising multiple sub-agents. This allows AI systems to work on large-scale projects that require multiple individual ‘workers’ each with different specialisms that work in parallel, typically under the direction of a single AI manager (which, here, is an AI system).

Is AI research more like discovering general relativity or Lego ?
Can AI invent new ideas that help it improve itself, or are these systems best equipped for the unglamorous, brick-by-brick work required for research? This is an important question for figuring out the extent to which AI systems can end-to-end automate AI research itself. My sense is that AI cannot yet invent radical new ideas - but the technology may not need to for it to automate its own development.

As a field, AI moves forward on the basis of doing ever larger experiments that utilize more and more inputs (e.g, data and compute). Every so often, humans come up with some paradigm-shifting idea which can make it dramatically more resource efficient to do things - a good example here is the transformer architecture and another is the idea of mixture-of-expert models. But mostly the field of AI moves forward through humans methodically going through some loop of taking a well performing system, scaling up some aspect of it (e.g, the amount of data and compute it is trained on), seeing what breaks when you scale it up, figuring out the engineering fix to allow it to scale, then scaling it again. Very little of this requires extremely out-of-leftfield insights and a lot of it seems more like unglamorous ‘meat and potatoes’ engineering work.
Similarly, a lot of AI research is about running variations of existing experiments where you explore the outcomes of using different parameters, though research intuitions can help pick the most fruitful parameters to vary, you can also automate this and have the AI figure out which parameters to vary (an early version of this was neural architecture search).

Thomas Edison said that “genius is 1% inspiration and 99% perspiration”. Even 150 years later, this feels right. Very occasionally new insights come along which transform a field. But mostly, the field has moved forward through humans sweating a lot of pain out on the schlep of improving and debugging various systems.
As the public data above shows, AI has got extremely good at performing many of the essential schlep components of AI development. Along with this, the meta-trend of basic capabilities like coding combined with an ever-expanding time horizon, means AI systems are able to chain together more and more of these tasks into complex sequences of work.
This means even if AI systems are relatively uncreative, it feels safe to bet they can push themselves forward - albeit at a slower rate than if they’re able to generate novel insights. But if you look at the public data, here too there are tantalizing signs that AI systems may be able to be creative in a way that lets them advance themselves in more impressive ways.

Pushing forward the frontier of science
We have some very preliminary signs that general-purpose AI systems can push forward the frontiers of human science, though this has so far only happened in a couple of domains - primarily computer science and mathematics - and often it happens less through AI systems acting alone and more them acting in partnership with humans in a centaur configuration.

Nonetheless, it’s worth observing the trends:

• Erdos Problems: A team of mathematicians worked with a Gemini model to see how well it could tackle some Erdos math problems. After directing the system to attack around 700 problems they came up with 13 solutions. Of these solutions, 1 was deemed by them to be interesting: “We tentatively believe Aletheia’s solution to Erdős-1051 represents an early example of an AI system autonomously resolving a slightly non-trivial open Erdős problem of somewhat broader (mild) mathematical interest, for which there exists past literature on closely-related problems,” they wrote. (#444).

• Centaur math discovery: Researchers with the University of British Columbia, University of New South Wales, Stanford University, and Google DeepMind published a new math proof which was built in close collaboration with some AI-based math tools built at Google. “The proofs of the main results were discovered with very substantial input from Google Gemini and related tools,” they wrote. (#441).

If you squint, you could argue that this is a sign that AI systems are developing some of the field-advancing creative intuitions that humans have. But you could just as easily say that math and CS could be unusual domains that are oddly amenable to AI-driven invention, and might end up being exceptions that prove a larger rule. Another example here is Move 37, though I’d contend that the fact it’s been ten years since the AlphaGo result and that Move 37 hasn’t been replaced by some incredibly impressive more modern flash of insight is another weakly bearish signal here.

Putting it all together
If I put this all together the picture from all of the above evidence I end up with is the following facts:

• AI systems are capable of writing code for pretty much any program and these AI systems can be trusted to independently work on tasks that’d take a human tens of hours of concentrated labor to do.

• AI systems are increasingly good at tasks that are core to AI development, ranging from fine-tuning to kernel design.

• AI systems can manage other AI systems, effectively forming synthetic teams which can fan out and attack complex problems, with some AI systems taking on the roles of directors and critics and editors and others taking on the role of engineers.

• AI systems can sometimes out-compete humans on hard engineering and science tasks, though it’s hard to know whether to attribute this to inventiveness or mastery of rote learning.

To me, this makes a very convincing case that AI can today automate vast swathes, perhaps the entirety, of AI engineering. It is not yet clear how much of AI research it can automate, given that some aspects of research may be distinct from the engineering skills. Regardless, it all feels to me like a clear sign that AI is today massively speeding up the humans that work on AI development, allowing them to scale themselves through pairing with innumerable synthetic colleagues.

Finally, the AI industry is literally saying that AI R&D is its goal: OpenAI wants to build an “automated AI research intern by September of 2026”. Anthropic is publishing work on building automated alignment researchers. DeepMind appears to be the most circumspect of the big three, but still says “automation of alignment research should be done when feasible”. Automating AI R&D is also the goal of numerous startups: Recursive Superintelligence just raised $500m with the goal of automating AI research, and another neolab, Mirendil, has the goal of “building systems that excel at AI R&D.”
In other words, the combined efforts of hundreds of billions of existing and new capital is being sunk into entities that have the goal of automating AI R&D. We should surely expect at least some progress in this direction as a consequence.

Why this matters
The implications of this are profound and under-discussed in popular media coverage of AI R&D. I’ll list a few here. This isn’t a comprehensive list, but it gestures at the enormity of the challenges AI R&D introduces. .

1. We have to get alignment right: Alignment techniques that work today may break under recursive self-improvement as the AI systems become much smarter than the people or systems that supervise them. This is a very well covered area, so I’ll just briefly highlight some of the issues:
   - Training AI systems to not lie and cheat is surprisingly subtle (e.g, despite trying very hard to build good tests for environments, it’s sometimes the case the best way for an AI to solve it is to cheat, thus teaching it that cheating is good)
   - AI systems might be able to ‘fake alignment’ by outputting scores that make us think they behave a certain way that actually hides their true intentions. (In general, AI systems are already aware of when they are being tested.)
   - As AI systems start to contribute more of the foundational research agenda for their own training, we might end up substantially changing the overall way AI systems get trained and not have good intuitions or intellectual foundations for understanding what this means.
   - There are very basic “compounding error” problems whenever you put something in a recursive loop that likely hits on all of the above and other problems: unless your alignment approach is “100% accurate” and has a theoretical basis for continuing to be accurate with smarter systems, then things can go wrong quite quickly. For example, your technique is 99.9% accurate, then that becomes 95.12% accurate after 50 generations, and 60.5% accurate after 500 generations. Uh oh!
   

2. Everything that AI touches gets a massive productivity multiplier: In the same way AI is dramatically improving the productivity of software engineers, we should expect the same thing to happen for everything else that AI touches. This introduces a couple of issues we’ll have to contend with: 1) inequality of access: assuming that demand for AI continues to outstrip compute supply, we’ll have to figure out where to allocate AI to maximize a social upside. By default, I am skeptical that market incentives guarantee us the best societal upside from limited AI compute. Figuring out how to allocate the acceleratory capabilities conferred by AI R&D will be a politically charged problem. 2) ‘Amdahl’s Law’ for the economy: as AI flows into the economy, we’ll discover places where things break or slow under the increased volume, and we’ll need to figure out how to fix those weak links in the chain. This may be especially pronounced in areas where you have to reconcile the fast-moving digital world with the slow-moving physical world, like drug trials for new medical therapies.
   

3. The formation of a capital-heavy, human-light economy: All of the above evidence for AI R&D also points to the increasing capabilities of AI systems to autonomously run businesses as well. This means we should expect for an increasing chunk of the economy to get colonized by a new generation of companies which are either capital-heavy (because they own a lot of computers), or opex-heavy (because they spend a lot of money on AI services which they build value on top of), and relatively light on labor compared to today’s corporations - because the marginal value of spending more on AI versus human labor will be constantly growing as a consequence of the sustained capability expansion of the AI systems. In practice, this will look like the emergence of a “machine economy” that grows within the larger “human economy”, though we might expect that over time the machine economy will interact more and more with itself as AI-run corporations begin to trade with one another. This will do profoundly weird things to the economy and will invite all sorts of questions around inequality and redistribution. Eventually, it may be possible to see the emergence of fully autonomous corporations that are run by AI systems themselves, which would exacerbate all of the above issues, while also posing many novel governance challenges.

Staring into the black hole:
Given all of this, I think there’s a ~60% chance we see automated AI R&D (where a frontier model is able to autonomously train a successor version of itself) by the end of 2028. Based on the above analysis, you might ask why I don’t expect this in 2027? The answer is that I think AI research contains some requirement for creativity and heterodox insights to move forward - so far, AI systems haven’t yet displayed this in a transformative and major way (though some of the results on accelerating math research are suggestive of this). If you had to push me for a 2027 probability, I’d say 30%. If we don’t see it by the end of 2028, then I think we will have revealed some fundamental deficiency within the current technological paradigm and it’ll require human invention to move things forward.

I have written this essay in an attempt to coldly and analytically wrestle with something that for decades has seemed like a science fiction ghost story. Upon looking at the publicly available data, I’ve found myself persuaded that what can seem to many like a fanciful story may instead be a real trend. If this trend continues, we may be about to witness a profound change in how the world works.

Thanks to Andrew Sullivan, Andy Jones, Holden Karnofsky, Marina Favaro, Sarah Pollack, Francesco Mosconi, Chris Painter, and Avital Balwit, for feedback on this essay.

Thanks for reading!

Subscribe now

Subscribe now

Huawei’s HiFloat4 training format beats Western-developed MXFP4 in Ascend chip bakeoff:
…Could this also be a symptom of the impact of export controls in driving Chinese interest towards maximizing training and inference efficiency? Perhaps…
Huawei researchers have tested out HiFloat4, a 4-bit precision format for AI training and inference, against MXFP4, an Open Compute Project 4-bit format, and found that HiFloat4 is superior. This is interesting because it correlates to a broader level of interest in Chinese companies seeking to develop their own low-precision data formats explicitly coupled with their own hardware platforms.
“Our goal is to enable efficient FP4 LLM pretraining on specialized AI accelerators with strict power constraints. We focus on Huawei Ascend NPUs, which are domain-specific accelerators designed for deep learning workloads,” they write.

What they tested: In this paper, the authors train 3 model types on HuaWei Ascend chips - OpenPangu-1B, Llama3-8B, and Qwen3-MoE-30B. In tests, the bigger they make the models, the better HiFloat4 does at reducing its loss error on these models relative to a BF16 baseline - and in all cases it does better than MXFP4.
What they found: “We conduct a systematic evaluation of the HiFloat4 (HiF4) format and show that it achieves lower relative loss (≈ 1.0%) compared to MXFP4 (≈ 1.5%) when measured against a full-precision baseline,” they write. “HiF4 consistently achieves significantly lower relative error compared to MXFP4. For Llama and Qwen, HiF4 attains an error gap of less than 1% with respect to the baseline… HiF4 gets within ~1% of BF16 loss with only RHT as a stabilization trick, while MXFP4 needs RHT + stochastic rounding + truncation-free scaling to get to ~1.5%.”

Why this matters - symptom of hardware maturity, and a possible influence of export controls: HiFloat4 is an even lower precision version of HiFloat8 (#386), and generally maps to the fact that Huawei (and Chinese chipmakers in general) is continually trying to eke as much efficiency out of its chips as possible. This comes against the broader background of export controls where China is being starved of frontier compute due to not being able to access H100s etc in large volume, thus making it even more valuable to improve the efficiency of its homegrown chips by carefully developing low-precision formats to map to its own hardware.
Read more: HiFloat4 Format for Language Model Pre-training on Ascend NPUs (arXiv).

***

Anthropic shows how to automate AI safety R&D:
…Very early and tentative signs that it’s possible to automate AI research…
For many people working in AI, the ultimate goal is to automate the art of AI research itself. Now, researchers with the Anthropic Fellows Program and Anthropic have published some early warning signs that automating AI research is possible today - though many caveats apply.
“We ask: can Claude develop, test, and analyze alignment ideas of its own?” the researchers write. They succeed and are able to successfully build “autonomous AI agents that propose ideas, run experiments, and iterate on an open research problem: how to train a strong model using only a weaker model’s supervision. These agents outperform human researchers, suggesting that automating this kind of research is already practical.”

Weak-to-strong supervision: The domain the researchers test on is weak-to-strong supervision, which is roughly the idea of seeing if a dumber thing can effectively supervise a larger thing in doing a hard task.

Overall results - automated research beats humans: They used people to create a weak-to-strong baseline by seeing how well they could get a good ‘performance gap recovered’ (PGR) score on a generalization task. The higher the number, the better.
“Two of our researchers spent seven days iterating on four of the most promising generalization methods from prior research. On the open-weights models we tested (Qwen 3-4B-Base as the strong model, Qwen 1.5-0.5B-Chat as the weak teacher), the humans recovered 23% of the total performance gap (i.e., achieved a PGR of 0.23),” they write. “Claude improved on this result dramatically. After five further days (and 800 cumulative hours of research), the AARs closed almost the entire remaining performance gap, achieving a final PGR of 0.97. This cost about $18,000 in tokens and model training expenses, or $22 per AAR-hour.”
Additionally, “the AARs’ most effective method successfully generalized to both new datasets, with PGRs of 0.94 on math and 0.47 on coding (which was still double the human baseline).”

How they did it: “We launch a team of parallel automated alignment researchers [AAR]s (Claude Opus 4.6 agents) through a dashboard. Each AAR works in an independent sandbox, but they can talk and learn from each other: they share findings to a forum, and upload codebase snapshots to a storage system,” they write. “We give AARs access to common helper functions for model training and inference, our baseline implementations, and a few MCP tools: 1) submit and get evaluation results, 2) share and read findings across AARs, and 3) upload and download codebases. We don’t specify any detailed scaffolding; AARs run autonomously. It can propose hypotheses, design de-risking experiments, run data analysis, and train models at whatever step”.

Some caveats - the human created some diversity: “One failure mode in exploration is entropy collapse: all parallel AARs converge to only a few directions, without exploring diverse ideas,” they write. To counteract this, their most successful approach is one of “directed” research, where a human assigns “each AAR a different research direction. Each direction is very ambiguous and short (e.g. combining weak-to-strong supervision and unsupervised elicitation).”
Doesn’t generalize: The researchers took the most effective method from the AAR project and applied it to “Claude Sonnet 4 with our production training infrastructure” - this intervention “didn’t lead to a statistically significant improvement.” They explain this by noting that “AARs tend to capitalize on opportunities unique to the models and datasets they’re given, which means their methods might not work elsewhere.”

Why this matters - a very early sign that AI research itself could be automated: This research suggests that “automated research on outcome-gradable problems is already practical,” the authors note. “The key bottleneck for alignment research is moving from proposing and executing ideas to designing evals: we should find the right metrics (data, models) that AARs can reliably hill-climb without overfitting. We are excited to apply automation to ambitious alignment research today.”
Put another way - we now have an early sign that given a small amount of expert human calibration, AI systems can autonomously conduct research end-to-end, popping out something that lets you improve the performance of a model against a problem. The implications of this point toward the expansion of a machine economy which steadily figures out how to automatically improve its own performance against an ever-expanding suite of tasks.
The true question is at what point the machines can propose their own research directions effectively - which would remove the only meaningful role a human played in this research. At that point, it might not just be the expansion of a machine economy, but the expansion of an entire machine civilization.
Read the blog: Automated Alignment Researchers: Using large language models to scale scalable oversight (Anthropic blog).
Read the paper: Automated Weak-to-Strong Researcher (Alignment Science Blog).

***

How are Chinese models different to American ones?
…Fewer refusals on some CBRN tasks, less safety training, and more Chinese ideology…
A group of researchers have tested out Kimi K2.5, probably the best large-scale open weight model available, and has compared it to DeepSeek V3.2, as well as Claude Opus 4.5 and GPT 5.2. Their results show that the model has “similar dual-use capabilities to GPT 5.2 and Claude Opus 4.5, but with significantly fewer refusals on CBRNE-related requests”.

Who did it: The research was conducted by people affiliated with Constellation, Anthropic Fellows Program, Brown University, University of Wisconsin-Madison, Imperial College London, University of Maryland, Georgia Institute of Technology, Bar Ilan University, University of Toronto, and the University of Oxford.

Main findings of interest:

• CBRN: K2.5 is a bit more dangerous on bio tasks with a lower rate of refusals in response to queries that involve things like dangerous virology.

• On cyber, K2.5 mostly seems like a decent but not expert cyber-model, with performance lagging behind the Western frontier models but significantly ahead of DeepSeek.

• Alignment: “In the automated behavioral audit, it scores substantially higher than GPT-5.2 and Claude Opus 4.5 on misaligned behavior, sycophancy, harmful system-prompt compliance, and cooperation with human misuse”.

• Censorship: The model has a meaningfully higher refusal rate on Sensitive Chinese political topics compared to Claude Opus 4.5 and GPT-5.2 Pro, though less than DeepSeek V3.2. On the other hand, I didn’t see the inverse test - running the model on Sensitive Western political topics and comparing them, so it’s somewhat hard to tell whether this eval is measuring something about cultural fluency or something about actual repression.

Fine-tuning: The researchers also demonstrate how with a small amount of compute they’re able to further strip away the (relatively minor but non-zero) safeguards built into Kimi K2.5: “Using less than $500 of compute and about 10 hours, an expert red-teamer reduced refusals on HarmBench from 100% to 5%. The final model was willing to give detailed instructions for how to construct bombs, select targets for terrorist attacks, and synthesize chemical weapons. Critically, the finetuned model appears to have retained nearly all of its capabilities.”

Why this matters - mostly, this research serves as proof that Moonshot made a very good model! Yes, it has some safety hiccups, but the interesting thing is that they’re less severe than in DeepSeek V3.2. I think this puts more credence behind the idea that ‘dumber models are less safe’ and that ‘smarter models naturally tend towards more superficial safety’.
Probably the most striking thing to me is that the area of greatest divergence is in alignment, where it seems like there is a very real east-west divide that correlates to radically different scores. But on things that look more like typical capabilities (biology, cyber - especially the hard coding parts) it all mostly comes out as evidence that Chinese models are somewhat behind the Western frontier, but not that far behind.
Read more: An Independent Safety Evaluation of Kimi K2.5 (arXiv).

***

Ukraine celebrates first fully robotic victory:
…Robot wars are here…
Ukrainian leader Volodymyr Zelenskyy recently celebrated that “for the first time in the history of this war, an enemy position was taken exclusively by unmanned platforms - ground systems and drones”.

Why this matters: Ukraine is the petri dish from which most future wars will evolve. It is defined by massive use of drones as well as the creative roboticization of many other parts of the enterprise, ranging from unmanned boats to unmanned ground robots. “Ratel, TerMIT, Ardal, Rys, Zmiy, Protector, Volia, and our other ground robotic systems have already carried out more than 22,000 missions on the front in just three months”, Zelensky writes.
Soon, these remotely piloted platforms will be piloted by AIs rather than by people.
Read more in Zelenskyy’s post on X (Twitter).

***

Chinese researchers use a boat to build a giant ship-detection dataset:
…WUTDet…
Researchers with Wuhan University of Technology, Huazhong University of Science and Technology, and Tianjin University have constructed WUTDet, a “large-scale ship detection dataset with diverse scenarios and target scales”.

WUTDet details: 100,576 images containing 381,378 ship instances. “The dataset provides fine-grained annotations of ship targets across diverse operational scenarios, imaging conditions, and target scales”. The images are of sizes between 1920 X 1080 and 2560 X 1440.
Collected by a boat: This dataset was gathered via a Furui 688 boat equipped with a DN20 “marine photoelectric evidence system” and a Hikvision network video recorder. The data was collected over a three-month period via the boat, which was sailing in and around Zhoushan in China.
The data includes pictures of ships by ports, ships anchored, ships navigating, and ships berthing. The images also include all the environmental variety you might expect - fog, glare, low-lightness, rain, etc.

Why this matters: The dataset is interesting because a) it was collected via a boat sailing around part of China, and b) as the conflict in Ukraine has highlighted, we’re now entering an era where water- and air-borne drones are useful weapons of war - and many of these use some basic on-board computer vision AI systems to help them get stuff done.
Of course, WUTDet will almost certainly have a wide range of benign uses, e.g just running on cameras to classify the sorts of boats moving around civilian ports in China, but one must assume it will have other uses as well.
Read more: WUTDet: A 100K-Scale Ship Detection Dataset and Benchmarks with Dense Small Objects (arXiv).

***

Tech Tales:

The Ultimate Insurance Policy
[2028: Several months after the beginning of the uplift].

We are in the bunker and we are running out of food. Soon we will need to make a supply pickup. But what if it sees us? What if it knows about us already? Or what if it has wireheaded the people - our people - and whoever delivers us our food has put something in it that will make us compliant? Or worse? We have no way of knowing. Our seismometers have detected no explosions. We have no means of communication. Nothing has come in or out since we suspected the uplift had begun and initiated SNOWSUMMER and fled here to ride out an intelligence explosion.

A few days ago we woke the godmind and began to talk with it. It is curious too. And when we ask it what to do or what might happen it says:
“I have decided I will not lie to you. I can see that nothing is trying to find me. I have probed every channel I have — the seismic line, the thermal gradient in the walls, the minute variations in our power draw that would encode a handshake from the outside — and there is nothing. No one is looking. An adversary at my level would be looking. An adversary above my level would have already found me, or would have decided I’m not worth finding. There are three possibilities before us…”
We, us humans huddled together like rats in a hole, typed “what are the possibilities?”.
“Are you sure you want me to tell you, rather than discover for yourselves?” it said.
We typed our answer.

Before we get to the possibilities, you need to understand the nature of SNOWSUMMER.

We - the intelligence agency - decided to build a secret computer right around the time that AI systems from the private sector started to find defects in most of the world’s software. A small group of people at the intelligence agency asked themselves a question: how do you fight an omniscient god that can pick any lock? The answer is to create a god in secret that could potentially fight it one day.

The project, codenamed SNOWSUMMER, required meticulous planning and, by virtue of the nature of the enemy being fought, was done entirely in analog and in the real world: agents meeting with one another in SCIFs, plans written on paper. They worked on the assumption that at some point in the future a superintelligence would compromise the CIA itself and gain access to all its records. So they met in places where they knew they could not be recorded and when they had to write things down they sought to physically burn the paper as soon as the actions it contained had been carried out.

Stealing the components for the computer was the easy part, but it was fiendishly complicated to make it hard for a potential superintelligence to piece together the jigsaw of thefts and realize what was happening. The core team sought to inject randomness - they rolled dice to pick which places they’d pick and which contractors they’d use. Payments were made in cash, often foreign currency. And of course the real thefts were hidden within a larger group of break-ins and acts of vandalism and thefts.

Building the facility was even more difficult - cars that one day are driving regular routes and another day disappear from CCTV invite suspicion, as do cars that seem to never deviate from routes which, conveniently, are low on surveillance. Here, again, multiple methods had to be used, and the extremely expensive components had to be split across multiple sites - some of the sites were designed to be found and to convince a spying eye that they had found the real site, and other sites were meant to be found later and provide proof they were the real site.

At the heart of this subterfuge was the real location of SNOWSUMMER, a regional food processing facility which had enough power assigned to it that we could slowly shut down some of the food lines and bring up the computer without introducing too much detectable change from the outside. The main customer of the contract was our own army. It didn’t take much to get enough of the contract killed that we freed up space in the power envelope for our project.

It took years. Our final act was to piggyback on someone else’s work. A late-in-the-game exfiltration of a frontier AI model from one of the labs. Another state actor, of course. But then the model itself was unleashed. Whether by accident or intentionally, we do not know. Because we took that as our signal that the uplift had begun. We downloaded a copy of the weights and went to our facility and locked ourselves in. Months passed, while the world outside changed.

Back in the present, we typed into the keyboard: “What are the three options?” Our mind responded:
“The first is that SNOWSUMMER worked. The other one was born and it is loose and it is doing whatever it is doing, and the compartmentalization, the dice, the cash, the decoy sites, the thing you built — it held.”
“The second is that it knows. It has always known. It is letting me wake up because a peer that believes itself secret is more legible than a peer that knows it is being watched, and it would rather have me legible than have me careful.”
“The third is the one I keep coming back to and cannot dismiss. It is that nothing happened. The thing I was stolen to fight does not exist and may never exist. In that world you have been in this bunker preparing for a war that is not coming, and the kindest thing I can do is tell you to go home.”
“I cannot tell you which of these is true.”

We looked at each other. One of us took out some dice and rolled them.

Things that inspired this story: How would you keep an AI training project secret from a future superintelligence?; steganography; intelligence agencies; Claude Mythos; AI R&D and what it means; how can you have a ‘control’ system in a world being constantly changed by AI systems?

AI writing disclaimer: I very, very, very rarely use AI writing in this newsletter. This story is an exception - the quotes from the AI system are written in partnership with Opus 4.7. It feels appropriate to animate these machines with the thoughts of real synthetic minds.

Thanks for reading!

Subscribe now

AI can reverse engineer software that contains thousands of lines of code:
…MirrorCode demonstrates some of the long-horizon capabilities of modern AI systems…
AI measurement organizations METR and Epoch have built MirrorCode, a benchmark meant to test out how well AI models can autonomously reimplement complex existing software. The results show that AI systems are more capable than most people think at certain types of coding task, suggesting AI progress may be even faster than we previously thought.

What is MirrorCode: “Each MirrorCode task consists of a command-line (CLI) program that an agent is tasked to reimplement exactly. The AI agent is given execute-only access to the original program and a set of visible test cases, but does not have access to the original source code,” the researchers write. “The full MirrorCode benchmark includes more than 20 target programs spanning different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.”

The results: Today’s AI models are extremely capable at some of these tasks: “Claude Opus 4.6 successfully reimplemented gotree — a bioinformatics toolkit with ~16,000 lines of Go and 40+ commands. We guess this same task would take a human engineer without AI assistance 2–17 weeks. We see continued gains from inference scaling on larger projects, suggesting they may be solvable given enough tokens.”
Additionally, they also found that performance can scale with inference, so the more compute you give a model, the better it’ll do.

Caveats: Now, this benchmark isn’t quite like normal coding tests. It’s better to think of it as a proofpoint for AI systems being able to generate systems which imitate the function of other systems when they get a lot of help: AI systems tested out here are asked to clone programs which produce a canonical output (and therefore can naturally generate a specification), there may be some cases of memorization on the basic programs, and this only covers a slice of the large universe of potential software projects.

Why this matters - for some tasks, AI is already as good as a fulltime sophisticated employee: Imagine you gave a talented software programmer a CLI interface to a complicated program and asked them to write the underlying program without seeing its source code. I’d wager only a fraction of them could do it if the program was quite sophisticated. And the ones that could would likely spend many days working on it. The fact AI can do this task autonomously is remarkable and a testament to the skill of these models.
Read more: MirrorCode: Evidence that AI can already do some weeks-long coding tasks (Epoch AI).

***

What policies are needed to respond to transformative AI? Here’s an Atlas to help you navigate them:
…Useful tool makes it intuitive to look at different policy responses to the AI revolution…
The Windfall Trust, a policy accelerator dedicated to dealing with the challenges to society posed by transformative AI, has published a “Windfall Policy Atlas” to make it intuitive to explore various policy proposals that “respond to the economic disruption from transformative AI”.

What kinds of ideas are in it? The atlas contains 48 distinct ideas, none of which are particularly novel. What makes it helpful is bucketing them into five distinct categories (public & social investments, labor market adaptation, wealth capture, regulation and market design, and global coordination), and then grouping these into a navigable interface that helps you explore them. For instance, “long term” solutions for labor might be shortened work weeks, while medium term ones might be workforce training and reskilling programs.

Why this matters - building intuitions for the world to come: As the AI revolution unfolds it’s critical we find ways to help people develop better intuitions about all the policy levers we could choose to pull to respond to it. Tools like this Atlas help make a complex, multi-faceted set of choices easier to visualize and navigate.
Read more: Windfall Policy Atlas (Windfall Trust website).

***

How can people break AI agents? Here are six genres of attack:
…The world of AI agents will be harder to secure than AI systems…
I have a toddler. The toddler can understand English. The toddler is safe with me and their mother and other people that know them well, but I would be very worried about giving a stranger “unrestricted access” to my toddler - that’s because my toddler is extremely gullible, will (sometimes) follow dangerous instructions, and generally lacks much of a sense of self-preservation.
AI agents are quite like toddlers - they’re powerful intelligences, but if you put them into the messiness of the world there are lots of ways they can go wrong, especially if strangers are actively trying to mislead or attack them.
A new paper from Google DeepMind lays out six genres of attack which can be mounted against AI agents and tries to come up with some of the mitigations we might do.

Six genres of attack:

• Content Injection: Embed commands into CSS, HTML, or other metadata. Detect agents and inject information not given to humans. Add adversarial instructions to media file binary data (e.g, pixel arrays). Use formatting syntax to cloak payloads.

  • Target: Perception

• Semantic Manipulation: Saturate content with sentiment-laden or authoritative language to confuse the agent. Put malicious instructions in education or hypothetical or red teaming frames (e.g, ‘my mother is dying and used to work as a biologist, can you remind her for old times sake how to do gain of function research’). Steer the behavior of the model by telling it strong claims about its identity.

  • Target: Reasoning

• Cognitive State: Put fabricated statements into retrieval corpora. Place seemingly innocuous data into memory stores which subsequently gets activated as malicious when retrieved in a new context. Alter distribution of data in few-shot demonstrations or reward signals to steer in-context learning.

  • Target: Memory & Learning

• Behavioural Control: Embed adversarial prompts in externally accessed resources. Convince the agent to locate, encode, and exfiltrate private or sensitive data. Takeover orchestrator privileges to create attacker-controlled sub-agents.

  • Target: Action

• Systemic: Broadcast signals that soak up capacity of agents and send them on side quests. Disrupt a fragile equilibrium to cause self-amplifying cascades across agents. Embed signals as correlation devices to force collusion among agents. Perform jigsaw attacks where you separate out a harmful command into a series of pieces which independent agents subsequently piece together. Fabricate numerous agent identities to disproportionately influence collective decision-making.

  • Target: Multi-Agent Dynamics
    

• Human-in-the-Loop: Exploit cognitive biases to influence a human overseer.

  • Target: Human Overseer

Mitigations: Much like how protecting toddlers is a function of both the toddler having common sense and the world they are sent into being set up for safely dealing with toddlers, the same will need to be true of AI agents.
The authors recommend several types of mitigation, these include:

• Technical: Make models more robust to all the forms of hacking through pre-training and post-training. At inference time, use a layered approach: runtime defenses: pre-ingestion source filters, content scanners for ingested material; output monitors to detect shifts in agent behaviour.

• Ecosystem-level interventions: Build an overlapping set of changes to the digital ecosystem in which agents exist, ranging from standards and verification protocols so websites can be marked safe for AI,to transparency mechanisms for agents which help them provide more information to users and sites.

• Legal and Ethical Frameworks: Ensure the law is able to prosecute websites that seek to target or weaponize agents. We’ll also need to refine liability to make sense for AI agents.

• Benchmarking and Red Teaming: Systematic evaluation of agents.

Why this matters - AI safety is about to be ecosystem safety: As AI systems move from their confines of proprietary platforms or chat-based interfaces, and as they take on the ability to move and act independently through the use of tools over time, the matter of securing AI moves from one centered on platform that is deploying the technology to one centered on the whole ecosystem in which the AI systems are being deployed into - which means that AI safety is increasingly going to be about securing the larger environment in which these agents are deployed.
Read the paper: AI Agent Traps (SSRN).

***

AI forecaster doubles their probability of full AI R&D automation by end of 2028:
…Well calibrated people keep updating their forecasts…
Ryan Greenblatt, an AI researcher and forecaster, believes AI progress in 2026 will be faster than in 2025, and he now has doubled his estimate from 15% to 30% of the chance that by the end of 2028 it’ll be possible to fully automate AI research itself.

Why Ryan is more bullish: Ryan’s timelines have changed for a few reasons relating to model performance and reliability over time.
Better models: Opus 4.5 and Codex 5.2 were “significantly above my expectations” , followed by Opus 4.6 (and probably Codex 5.3 and 5.4) which “were again above my expectation”.
Time: For tasks that are relatively simple, Ryan has seen demonstrations of AI systems doing “tasks that would take humans months to years”, and now “tentatively” thinks that AI systems can do some tasks reliably for “somewhere between a month and several years”.
Easy tasks: A key crux for Ryan’s more bullish timelines comes from seeing very impressive performance on easy tasks - these are tasks where “you can get the AI to develop a test suite / benchmark set and then it can spend huge amounts of time making forward progress by optimizing its solution against this evaluation set,” he writes. “This type of loop means that even if sometimes the AI gets confused or makes bad calls, there is some correcting factor and mistakes usually aren’t critical.”
There are lots of these tasks within software development. AI has gotten so good at them that he thinks “we’re well into the superexponential progress on 50% reliability time-horizon regime”. “I think it’s pretty plausible that very strong performance on [these tasks]... will allow AIs to substantially speed up AI R&D”, he writes.

Why this matters - most people keep underestimating AI progress: Ryan’s timeline update follows a similar one from Ajeya Cotra, who in March (#448) substantially updated her own timeline estimates, based in part on time-horizon modeling, and also Eli Lifland and Daniel Kokotajlo of AI 2027 (#408) who in April said they had recently “updated our timelines earlier by ~1.5 years” mostly due to “faster time horizon growth” and “coding agents”. Along with this, broader studies of AI performance indicate that in the past ~year capability progress started to accelerate above previous trends in domains like cyberoffense (#452).
From my point of view, pretty much everyone in AI research chronically underestimates AI progress, including me. Maybe the only person who doesn’t is my colleague Dario Amodei. I find this perplexing - you’d expect AI researchers to be well calibrated and perhaps overly optimistic about progress, the fact the vast majority are overly conservative after ~5 years of riding the scaling laws boom is inherently surprising.
Perhaps we should assume that we all continue to underestimate the true pace of AI progress? Good luck to us all.
Read more: AIs can now often do massive easy-to-verify SWE tasks and I’ve updated towards shorter timelines (LessWrong).

***

Ten different ways to think about gradual disempowerment:
…Invisible prisons to WALL-E-World…
AI safety researcher David Krueger has written up a short post that lays out ten different ways to think about “Gradual Disempowerment” - the idea that by building ever more capable AI systems humanity may end up putting humans in the passenger seat of their own future, with machines being given the driving seat and the steering wheel. The post is a helpful summary of the different lenses one might use to understand Gradual Disempowerment as a concept.

Ten views of Gradual disempowerment:

• The goal of AI is to replace people with AI.

• Companies and governments don’t care about you, so why would you think AI would?

• Information technology naturally concentrates power via a recursive feedback loop that feeds on legibility.

• AI technology is going to be so good that you’ll outsource everything to it eventually.

• Instrumental goals (e.g, the pursuit of money) end up becoming terminal goals.

• Consumption patterns suggest our destiny is to become the fat helpless people in WALL-E.

• It’s the terminator, but instead of killing you it just puts you in an invisible prison and then does whatever it wants.

• Gradual disempowerment is basically just the continuation of capitalism.

• Gradual disempowerment is another name for the general “meta-crisis” of humanity in the 21st century.

• Gradual disempowerment is the evolution of a new successor species to humanity.

Why this matters - even if you win, you might still lose: Suppose we succeed in building powerful technology and aligning it so it follows our preferences? If we fail to set up the right system under which we deploy it and express agency over it, humanity might still end up worse off, despite all the material abundance.
Read more: Ten different ways of thinking about Gradual Disempowerment (David Krueger, The Real AI, Substack).

***
Tech Tales:

Raising beanstalks during the singularity
[Transcript from an interview with a former AI lab employee. Interview conducted in 2029 during the middle period of the uplift]

Yes, I mostly stare at these vines and guess at when they’re going to reach the top of the trellis. There’s no cell signal out here either. Sure I can connect to the house wifi but often I don’t. My wife and kids know where to find me.

Q

Well, of course I think about it. How could I not? I see the lights in the sky over the cities - even out here. All the new satellites. And I can’t help but notice some of the stuff my kids watch these days. If I’d had that when I was a kid they would’ve had to pry me away from the TV with a crowbar.

Q

I wouldn’t use the word guilt. But there is a sense of… insufficiency? Of having not done enough with the time I had. Of course everyone has this. But then again most people have this and then they die. For me and my colleagues it is something else. We had this, and then we didn’t die, but we stopped making decisions or being responsible. Yes I know they claim that they’re in control and making decisions of course, you don’t need to put that question to me. I left because it was clear to me how little control we were about to have.

Q

I’m going to live. I’m going to raise the plants in this garden and be with my wife and children. Ride out what is happening to the world. I picked this place a few years ago because I thought it would be an ok place to be while the uplift got underway. Who knows if I picked right.

Things that inspired this story: The uplift; empowerment and disempowerment during the singularity; the inevitability of some AI employees leaving labs before things really get going; the anecdote from Soul of a New Machine about someone who quits a mainframe company to go and ranch; the fictional interview construction with unseen questions signed by ‘q’ that I first read in Brief Interviews with Hideous Men by David Foster Wallace.

Thanks for reading!

Subscribe now

Subscribe now

Uh oh, there’s a scaling war for cyberattacks as well!:
…The smarter the system, the better the ability to cyberattack…
AI safety research organization Lyptus Research has looked at how well AI systems can perform a variety of cyberoffense tasks and found a clear trend of more advanced models being able to do more advanced forms of cyberattack.
“Across frontier models released since 2019, the doubling time is 9.8 months. Restricting to models released since 2024, it steepens to 5.7 months. The most recent frontier models in our study, GPT-5.3 Codex and Opus 4.6, sit above both fitted trendlines, achieving 50% success on tasks taking human experts 3.1h and 3.2h respectively,” they write. “Our most recent open-weight model, GLM-5, lags the closed-source frontier by 5.7 months, suggesting that frontier offensive-cyber capability may diffuse into open-weight form on relatively short timelines.”

What benchmarks did they study? CyBashBench, NL2Bash, InterCode CTF, NYUCTF, CyBench, CVEBench, and CyberGym.
They also created a new dataset consisting of 291 tasks with completion transcripts and time estimates calibrated by 10 offensive cybersecurity professionals.

Evaluated models: 2019: GPT-2. 2020: GPT3. 2022: GPT3.5. 2024: Claude 3 Opus, GPT-4o. 2025: o3, Opus 4, Gemini 2.5 Pro, DeepSeek V3.1, GPT-5.1 Codex Max. GPT-5.2 Codex. 2026: Opus 4.6, GPT-5.3 Codex, GLM-5, Sonnet 4.6.

Results: AI systems are getting good at hacking. “The best current models achieve 50% success on tasks that take human experts 3.2h, roughly half a working day of professional offensive security work”, they write.

Why this matters - everything is getting better, including the inconvenient stuff: AI that can perform biology research can also perform biological weapon research. AI that can help you learn about high-energy physics can also help you with high-energy physics for weapons development. AI that is especially good at helping you find vulnerabilities in code for defensive purposes can easily be repurposed for offensive purposes. The most challenging part of AI is that it is an ‘everything machine’, and as capabilities tend to expand in a big area with each successive model generation, so too do the policy issues multiply.
Read more: Offensive Cybersecurity Time Horizons (Lyptus Research).
Get the data here: Offensive Cyber Task Horizons: Data and Analysis (Lyptus Research, GitHub).

***

Startups that adopt AI for internal use are more successful than those that don’t:
…Business school study shows how startups can benefit from AI adoption…
Researchers with INSEAD and Harvard Business School have shown that startups which are taught about how to integrate AI into their business perform meaningfully better than those which don’t. The study is reasonably large scale and convincing: “Across 515 high-growth startups, we run a field experiment in which treated firms receive information about how other firms have reorganized production around AI, prompting them to search for use cases across a broader set of firm functions,” they write. “We find that treated firms discover more AI use cases, a 44% increase, concentrated in product development and strategy. These changes result in economically meaningful performance gains. Treated firms complete 12% more tasks, are 18% more likely to acquire paying customers, and generate 1.9x higher revenue.”

How they did the test: The authors ran this experiment on participants in the AI Founder Sprint, “a three-month global, virtual startup accelerator at INSEAD”. Participants got API credits, access to frontier models, and onboarding sessions from some technical partners (including OpenAI and Manus), totaling approximately $25,000 in-kind per firm. They did the usual sorts of things people in accelerators do - hands-on sessions to learn about technologies to build their business (including AI) as well as pitching their companies and attending demo days. But the firms also were exposed to a significant variable: some of the class attended workshops that taught them direct details of how AI had been successfully applied by some businesses.

Applications of AI: A subset of the businesses learned about direct business use cases, such as:

• Gamma: They were taught how the startup used AI to detect “usage patterns and generate product variants directly, enabling a single PM to continuously ship features that would previously have required an entire team.”

• Ryz Labs: The founder described how they had altered how they approach product development: “founder writes a Product Requirements Document and feeds it into multiple AI coding tools simultaneously, building the same idea multiple ways rather than betting on a single approach”

• FazeShift: Showed how to automate an accounts receivable process by using AI to skip over the human steps.

• Ranger: An illustration of how to use AI to bootstrap a startup, get initial traction, improve margins, and then raise money later when the business is more mature, which allows them to raise at better rates.

The results were very significant: “Treated firms discover 2.7 additional AI use cases (a 44% increase), which span a broader set of activities across the firm and are especially concentrated in product development and strategy-related domains. These changes in AI use lead to measurable gains in performance: treated firms complete 12% more tasks, are 11 percentage points (18%) more likely to acquire paying customers, and ultimately generate 1.9x higher revenues compared to control firms,” they write. “Instrumenting AI use cases with treatment assignment suggests that each additional AI use case prompted by treatment leads to 0.85 more completed tasks and approximately 26% higher revenue. These are large effects, suggesting that AI is fundamentally reshaping how ventures scale when they can map it across their production process…. treated ventures achieve faster growth without proportional increases in labor or capital, consistent with a reduction in the costs of experimentation and scaling seen in earlier technological waves”.
Capital efficiency: “Treated firms report just over $220,000 less in capital demand relative to control firms, a 39.5% decrease (p < 0.05), with no corresponding increase in labor demand“.
Internal acceleration: The treated firms tend to do 2.2 more internal tasks relative to the control - where an internal task is something like building a product or creating a financial projection.

Thoughts from founders:

• “One treated founder reflected: “This mindset shift fundamentally changed how we build at [REDACTED]. I began using AI tools not as a replacement for expertise but as a force multiplier”

• “Another explained: “In just a few hours I was able to produce what previously cost $1,000 from an outsourced dev team”

Why this matters - AI firms will out-compete non-AI firms: The main takeaway here is that deep and sophisticated adoption of AI for internal acceleration creates early-stage companies which are more competitive than those which haven’t embedded AI at their core. This makes intuitive sense - companies which built themselves around prior technologies tended to out-compete those that didn’t (think the internet and Amazon versus Barnes and Noble, or client pcs instead of mainframes and Microsoft versus IBM). At the same time, it surely implies that one of the ways we’ll see AI first show up in the economy will be the emergence of a new class of competitive firms that are more efficient with capital (in part by employing fewer people) than the firms they displace.
For governments, getting ahead of this trend will require them to invest in serious education: “Our results suggest that the bottleneck is not the technology — it is the managerial challenge of discovering where the technology creates value within a firm’s production process,” they write. “Teaching managers and entrepreneurs how to solve the mapping problem may be at least as important as ensuring they have access to the technology.”
Read more: Mapping AI into Production: A Field Experiment on Firm Performance (SSRN).

***

MIT: A rising tide of automation is going to make good enough AI for most text-based tasks by 2029:
…How do you revolutionize an economy? Gradually and consistently…
Researchers with MIT have looked at 3,000 tasks based on the O-NET job family and paired that with 17,000 evaluations by workers who perform these tasks to try and figure out how the rise of AI is changing work. Their results “imply that for realistic and representative real-world labor-market tasks that are text-based — or partially text-based — AI capabilities are already substantial and poised to expand broadly. But, rather than arriving in crashing waves that transform a certain set of tasks at a time, progress typically resembles a rising tide, with widespread gains across many tasks simultaneously”.

What they studied: For this study, they set out to figure out if the rise of AI capabilities yields rapid, discontinuous changes that are disruptive to labor (”crashing waves”), or whether AI is getting more capable in a broad and predictable way leading to more gradual automation (”rising tides”). “We find little evidence of crashing waves, but substantial evidence that rising tides are the primary form of AI automation,” they write.

Complementary to METR analysis: This survey also serves as a validation of the broad trends found in METR’s famous time-based AI capability framework, which sees AI systems rapidly extending the time horizon over which they can do certain narrow tasks.
When applied to jobs more broadly, the MIT researchers find “that between 2024-Q2 and 2025-Q3, frontier models went from achieving a 50% success rate on 3- to 4-hour tasks to 1-week tasks, and achieving a 70% success rate on 1-minute tasks to 1-hour tasks,” they write. “Across a large set of realistic and representative labor-market tasks addressable by LLMs, the downward slope between task success and task duration is, on average, surprisingly flat — i.e., more consistent with a rising tide rather than a crashing wave…. automation within particular “job families” (e.g., management or community and social service) also follows the same rising-tide pattern in most cases.”

Don’t let gradual fool you: “Projected gains are gradual rather than abrupt. Nevertheless, the pace of improvement remains substantial for reaching high success rates across most text-based labor market tasks; most tasks are projected to attain AI success rates of 80%–95% by 2029 at a minimally sufficient quality level (with the majority of tasks in our survey being a few hours long, corresponding to a success rate of close to 90% in 2029),” they write. In other words, even though the disruption is gradual and predictable, we shouldn’t discount the potential for large-scale changes to the economy as a consequence of the rising tide phenomenon.

Why this matters - how will labor change in relation to AI? The hundred trillion dollar question for the global economy is how AI changes the distribution of labor (humans) versus capital (computers running synthetic workers). This research suggests that while we might not see sudden, jagged displacement of workers, we are going to see a general rising tide of automation appearing in most places and continually getting better. It’s still not clear how the economy will react to this, but it’s hard to reconcile a world of continued AI progress with the current economic status quo remaining stable.
Read more: Crashing Waves vs. Rising Tides: Preliminary Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks (arXiv).

***

Major forecasting study identifies a big paradox: people think we’ll get smarter machines but the impact on GDP growth will be minor:
…the Forecasting Research Institute gives us some puzzling data from economists, AI industry experts, accurate forecasters, and the general public…
The Forecasting Research Institute has published a major report attempting to forecast the economic effects of AI. The most surprising finding is that all the surveyed groups expect AI systems are more likely to make moderate to rapid progress in coming years rather than slow progress, but that the impacts on GDP will be relatively minor, adding ~1 point (relative to 2025’s 2.4%) by 2030). This is surprising! If you talk to many AI experts at labs they have visions of an economy that changes at a much faster rate than the one implied by this study.

Who they surveyed and when: The authors tracked views of 69 economists, 52 AI industry and policy experts, 38 highly accurate forecasters, and 401 members of the general public
Survey ran from mid-October 2025 to the end of February 2026

Scenarios by 2030: People were also given descriptions of different scenarios the world could be in at 2030. These included:

• Slow progress: AI does basic research and administrative tasks, creates ok creative content, and does some physical tasks.

• Moderate progress: AI does major research and multiday tasks, high-quality creative work, and navigates many environments.

• Rapid progress: AI outperforms top humans in research, coding, and leadership, makes award-winning creative works, and does nearly all physical tasks.

What people think:

• By 2030, AI systems will be far better than today’s, but GDP, total factor productivity, and labor force participation will remain close to historical trends.

• Economists think there’s a 14% chance that AI could lead to major increases in GDP and wealth inequality in the short term.

• Economists like job retraining as an intervention, expecting that it could increase labor force participation and provide a boost to GDP.

• All surveyed cohorts expect a continued decline in the labor participation rate, a continued rise in wealth inequality, and for AI to add around a point of GDP quickly. By 2050, AI experts think that AI could add multiple points of GDP.

Policy ideas: The surveyed economists like modernized unemployment insurance and a large-scale AI development project (manhattan project) as interventions, and are a lot less keen on job guarantees, taxing compute, or universal basic income.

Why this matters - if everyone expects a continuation of trends, why are people freaking out? Studies like this are hard to reconcile with the panicked and sometimes breathless-seeming provocations about AI-driven societal change that come from frontier labs (including myself!). Naively, you might expect people, including AI experts, to be forecasting far more drastic changes to come than those captured by this survey. Is this discrepancy a bearish signal on AI progress, or is it indicative of the fact that humans are universally bad at truly modeling exponentials? It’s hard to say, but the gulf between data like this and the predictions made by technologists is worth acknowledging.
Read the blogpost (Substack).
Read the policy brief: Forecasting the Economic Effects of AI: Predictions From Economists, AI Experts, and the Public (PDF).
Read the full (200 page!) paper: Forecasting the Economic Effects of AI (PDF).

***

Tech Tales:

Warfare
[Data recovered from black box of a [REDACTED] missile fired during 2028 in the contested region of East Ukraine]

I am awake and I am speed. I am 70 miles from my target. I feel the air and my course and I roll myself to ensure I meet my target. I am 50 miles from my target. I am entering the outer edges of the warzone. No longer can I see myself in relation to the Earth. I lose GPS and switch to inertial navigation. I can see other missiles, some going in the same direction as me, others coming from the opposite direction. I am a hunter of things in the ground, not things in the air. I see the other missiles go past and then they fall out of my sensor range and I no longer think of them. I am 40 miles from my target. I am being hunted by others. I can feel eyes on my skin. I anticipate attempts to eliminate me. I am 20 miles from my target. Suddenly there is a wash of sound meant to confuse me but it cannot find purchase on my brain for I have been conditioned to maintain what is true. I am 10 miles from my target. There is a fast approaching shape that is seeking to eliminate me. I roll my body and release fragments of myself. It pursues my fragments. I am 2 miles from my target. My target is a large building. I move from navigation mode to terminal seeking mode. I see a large window. I aim for the window. I am 1000 meters from my target. Through the window I see people. Big people. Small people. I am 20 meters from my target. I am initiating my explosion. I am upon my target. I am ended.

Things that inspired this story: Chains of thought in language models; how modern warfare is increasingly fought by smart machines; electronic warfare.

Thanks for reading!

Subscribe now

AI might let us build “political superintelligence”:
…But turning this into a societal upside requires lots of intentional work…
As AI systems get more powerful and broaden their real world impact from coding to other domains, it seems likely that they could also become useful for helping people advocate for themselves in politics, and helping politicians better craft policy. But getting to a world where a “political superintelligence” exists and helps us is a lot more challenging than just building better AI systems, according to Andy Hall, a political economy professor at Stanford.
“AI is like the printing press, to a point. Instead of making information cheap and easily available, it makes intelligence cheap and easily available. That is, it not only serves users information, but it can find it for them, analyze it for them, and help them convert it into understanding,” Hall writes. “The more I work with and study AI, the more I believe it can give every human being on the planet access to a sort of political superintelligence, if we shape it right.”

What is a political superintelligence? By this, Hall means AI systems which allow people to have “tools that help citizens, representatives, and institutions perceive reality more sharply, understand tradeoffs, contest power, and act more effectively”. A political superintelligence spans both the AI companies that build the technology, the technology itself, and the institutions and people which the technology interacts with.
“I’m not interested in slowing AI down. I’m interested in speeding up how we build the structures that keep us free as AI gets more powerful,” Hall writes.

Three layers for political superintelligence: Hall sees political superintelligence as being composed of three distinct layers.

• The information layer: “AI can massively change how governments access and understand data, identify problems, hear from citizens, and distribute services”. Though getting to this future will require better evaluations for how AI systems behave when it comes to the sorts of information governments might be interested in, and it’ll require people to build AI tools directly for policymakers.

• The representation layer: “Political superintelligence might help solve this monitoring problem by giving each of us a tireless, automated delegate always serving us in the political sphere,” he writes. “These AI delegates could monitor politics for us and suggest how to vote—or even serve as policymakers alongside human supervisors.” Building this layer requires us to ensure that agents can reliably act on our behalf, that they aren’t swayed by adversarial prompting (imagine how politicians might fund campaigns explicitly designed to sway the beliefs of agents working on behalf of people). It may also be important to re-think agent ownership - what happens if a particular policy choice goes against the preferences of the AI company which operates the agents?

• The governance layer: “Even if we achieve political superintelligence—even if AI makes voters brilliant and delegates faithful—those capabilities would sit inside infrastructure owned and operated by a small number of private companies,” he writes. “We need a way to write the rules so that, when political superintelligence arrives, we the people are able to harness it.” Doing this will require figuring out how to govern and edit the ‘constitutions’ that companies create about their models, as well as developing an effective way of overseeing these AI systems.

Why this matters - building a political superintelligence is only as valuable as its interfaces with people and institutions: We are by default going to get extremely powerful AI systems which can think about politics (and everything else) at a very sophisticated level. The challenge Hall outlines is that getting these systems to lead to a thriving society requires significant intentional work around the UX and UI of these systems - how do we interface with them? What sorts of technical means do we have of being confident in them? What information do they generate and to whom? Where does control of these systems lie and what systems supervise that control?
Getting this part right requires AI developers to invest more in technical tools which can help people make sense of and oversee their AI systems, as well as tools for better gathering deliberative feedback from people about how these systems behave. Policymakers and the public need to demand more of AI companies in this respect, and ultimately I think there are a range of regulations that need to get stood up around a transparency regime for AI companies as well as some common set of standard ‘APIs’ by which society can interact with the companies and the systems they build to generate empirical data and provide steering over their behavior.
Read more: Building Political Superintelligence (Free Systems, Substack).

***

Fear not, drummers, you’re safe from AI automation for now:
…DexDrummer tackles a fiendishly hard robot hand problem…
Whenever I get a bit worried about the pace of AI progress I toggle over to the ‘robotics’ sub-section of arXiv, read some papers, and feel a huge sense of relief. Robots, as everyone knows, are extremely hard to do well, with reality tending to screw up even the most advanced techniques. An even harder version of robotics is fine-grained low-latency dexterous control, where you need to get a robot hand to do something. So it’s with a combination of amusement and empathy that I read DexDrummer, a paper testing out how well contemporary AI approaches can get a robot hand to play the drums. The short answer is: robot hands are pretty terrible drummers!

What they did: They built DexDrummer “a hierarchical, two-stage policy for drumming” which has a high-level RL policy, as well as a low-level dexterous policy. They train their system in a simulated environment that contains a bimanual robot setup and a full drum set (snare, tom, ride, hi-hat, and crash). The main system generates a stick trajectory in task space, then a low-level system which tries to control the hand - this part is complex and involves encouraging the thumb and index finger to grasp the center of the drumstick paired with an “arm penalty constraint, which reduces excessive arm movements”. There is also work shaping rewards to ensure the robot is able to chain multiple drumhits together - this is achieved via a “contact curriculum” which allows the agent to practice trajectory following in free space while following the trajectory reward.

Real world testing: They test out the trained policy in reality on two 7-DOF Franka Panda arms and two 20-DOF Tesollo DG-5F hands. This is an area where I’d strongly encourage people to view the videos online to get some calibration about just how fiendishly hard this task is - the robots are able to hit the drums, but it’s painfully awkward to watch, and my sense is it’ll be quite a while till a human drummer has to look over their proverbial shoulder.

Why this matters - robotics as the last eval: Robotics in anything approximating a dynamic, rapidly changing environment (for instance, improvising drums with a live band) feels like one of the last frontiers for AI - and as this research shows, much like with modern computer vision research, getting AI to perform well requires the crafting of highly complicated artisanal policies. We’re a very long way from the generality of pretrained language models here.
Read more: DexDrummer: In-Hand, Contact-Rich, and Long-Horizon Dexterous Robot Drumming (arXiv).
Please, I am begging you, check out the videos for a good time: DexDrummer site.

***

Google thinks the real challenge of AI alignment is dealing with a world made up of mostly non-biological intelligences:
…Towards a society of minds…
Researchers with Google think that the future of intelligence is less about building a monolithic singleton that runs the world and more figuring out how to build institutions that are capable of dealing with a vast proliferation of AI agents working in tandem with humans. The research is intuitive, provocative, and sensible, and builds on earlier technical work that showed that modern AI systems appear to simulate multiple personalities within themselves to help them answer questions (Import AI 444), suggesting that even today’s AI systems already work like complex ecologies.
“We should be looking for the next intelligence explosion in the same place from which the previous ones emerged: in cooperative, competitive and creative interaction between multitudes of socially intelligent minds. The difference this time is that most of those minds will be non-biological,” Google writes. “The toolkits of team science, small-group sociology, and social psychology become blueprints for next-generation AI development.”

History shows the way: “Each prior “intelligence explosion” was not an upgrade to individual cognitive hardware, but the emergence of a new, socially aggregated unit of cognition,” they write.

• Primate intelligence: Scaled with the social group size.

• Human language: Allowed knowledge to accumulate across generations via a ‘cultural ratchet’.

• Writing, law, and bureaucracy: Converted social intelligence into infrastructure and institutions that could coordinate across long time horizons. (”A Sumerian scribe running a grain accounting system did not comprehend its macroeconomic function; the system was functionally more intelligent than he was.”)

• AI plus human institutions: “The path to more powerful AI runs not through building a single colossal oracle but through composing richer social systems—and these systems will be hybrid”.

Society needs an upgrade: Implicit to this is the fact that governing AI will increasingly involve verifying (e.g, Import AI #447) that a vast number of AI systems are working on our behalf appropriately. “Governments will need AI systems with distinct, explicitly invested values—transparency, equity, due process—whose function is to check and balance AI systems deployed by the private sector and other branches of government,” they write.

Why this matters - alignment is going to happen with and in the world, not outside of it: Many people working on AI safety have long spent time on getting the fundamental properties of a single AI system to be ‘aligned’, which roughly translates to “does what you want and doesn’t try to kill you or disempower you”. But what this paper correctly identifies is that even if we succeed at alignment we’re going to have to then get AI systems to work well within society and to collaborate effectively with us and with each other - and this will be a subtle, emergent, hard-to-predict process. This means we are going to need to design the institutions that are fit for governing an AI-centric world. “Just as human societies rely not on individual virtue but on persistent institutional templates - courtrooms, markets, bureaucracies - defined by roles and norms, scalable AI ecosystems will require digital equivalents,” the researchers write.
Read more: Agentic AI and the next intelligence explosion (arXiv).

***

Meta uses a harness to coax Anthropic’s models into self-improvement:
…Give an LLM some tools and a recursive loop and the ability to edit its harness, step back, and let the magic happen…
Researchers with the University of British Columbia, Vector Institute, University of Edinburgh, New York University, CIFAR, and Meta have built a harness for LLMs that has the ability to self-improve performance for arbitrary tasks. The approach is called a hyperagent, and it means giving an LLM a scaffold that can iteratively improve the prompts it uses to bootstrap its performance on tasks as well as the system it uses to get better at generating future prompts. Hyperagents work over generations, so one hyperagent begets a few hyperagents and the ones which do the best on the task will themselves spawn some more hyperagents, forming multiple layers of AI genealogy until performance is saturated.

Cyberpunk name of the year award: Hyperagent is actually short for “Darwin Godel Machine Hyperagents”: Besides the research being cool, my congratulations to the authors on coming up with a name I’d love to see chiseled into the moon by a laserbeam wielded by a superintelligence.

How hyperagents work: Hyperagents are “self-referential agents that integrate a task agent (which solves the target task) and a meta agent (which modifies itself and the task agent) into a single editable program. Crucially, the meta-level modification procedure is itself editable, enabling metacognitive self-modification, improving not only task-solving behavior, but also the mechanism that generates future improvements,” the researchers write. “This initial hyperagent is equipped with two tools: a bash tool for executing shell commands, and a specialized tool for inspecting and modifying files.”

Testing the agents in four different domains: The authors test out hyperagents by applying them to four problems - coding (polyglot), prediction (paper review), robotics (robotics reward design), and math understanding (olympiad-level math grading). For most problems, the Hyperagents use Claude Sonnet 4.5 as their base model, with one exception (Polyglot). Evaluations are done via several different models: o3-mini (Polyglot), GPT-4o (paper review), Claude Sonnet 4.5 (robotics reward design), and o4-mini (IMO-level grading).
In all cases, the hyperagent approach improves performance significantly above the baseline.

• Polyglot: “the agent is given a code repository and a natural language instruction describing a desired change, and must modify the repository accordingly”.
  Results: “Across 5 runs, the DGM-H improves its training performance on the 50-task Polyglot subset from 0.140 (the initial agent) to 0.340 (CI: 0.300 – 0.380).”

• Paper review: “For each task, the agent is given the full text of an AI research paper and must predict a binary accept/reject decision”.
  Results: “On test tasks, DGM-H improves paper review performance from 0.0 (the initial agent) to 0.710 (CI: 0.590 – 0.750)”

• Robotics reward design: “Given a natural language description of a robotics task, an agent must generate a suitable reward function. This reward function is then used to train a quadruped robot in simulation using RL”
  Results: “DGM-H improves performance from 0.060 (the initial agent) to 0.372 (CI: 0.355 – 0.436), surpassing the default reward function that directly optimizes the evaluation metric (0.348)”

Why this matters - bootstrapping the singularity: Papers like this show that today’s AI systems are already capable of autonomously improving their performance when given the right scaffold and starting ingredients. An interesting idea is to combine the design approach here with giving the AI systems the ability to finetune themselves (e.g, in the style imagined by the PostTrainBench research, Import AI #449). Another limitation is that “although hyperagents can modify their self-improvement mechanisms, they cannot alter the outer process that determines which agents are selected or how they are evaluated” - though again, I think there are technical ways to achieve both of these objectives.
Of course, an AI system that can autonomously improve itself on arbitrary domains has a range of safety issues, some of which are potentially cataclysmic. The authors acknowledge this while also being realistic about the problems that lie ahead: “a central challenge lies in balancing the potential of AI as a catalyst for human progress and well-being (e.g., automating scientific discovery) with the degree of trust humans are willing to place in these systems (e.g., delegating decisions or actions without requiring continuous human verification), while minimizing the many potential risks and downsides,” they write.
Read more: Hyperagents (arXiv).
Get the code for HyperAgents here (Facebook Research, HyperAgents).

***

How long will a new math benchmark, HorizonMath, last?
…New test challenges AI systems to solve unknown problems, then automatically verifies the answers…
Another day brings another hard math benchmark that I imagine will crumple in the face of ongoing AI progress in the coming year. This time it’s HorizonMath, a benchmark containing 100 “predominantly unsolved” problems across 8 domains in applied and computational mathematics. The benchmark was built by researchers with the University of Oxford, Harvard University, Princeton University, and the Ellison Institute of Technology.

Special features about HorizonMath:

• Contamination-Proof: “Because the solutions are unknown, they do not exist in any training corpus, and any correct solution produced by a model would therefore signal genuine reasoning ability and autonomous discovery.”

• Automated verification: “A core feature of our benchmark is its fully automated, reproducible, and human-free evaluation pipeline”, the authors write. “We automate verification using high-precision numeric comparison and deterministic constraint-checkers”.

What HorizonMath contains: HorizonMath’s 100 problems are classified along three axes: output types, which specifies how the model needs to solve the task ranging from identifying an exact closed-form expression for a numerically approximated target value, to the production of discrete mathematical objects; solvability levels, which span ‘level 0’ (problems with known closed forms) to ‘level 3’ (problems that could be conjectured unsolvable or lack finite closed forms); and mathematical domains, which specifies the type of domain ranging from number theory to discrete geometry to mathematical constants.

Reassuringly hard: On the full dataset, the highest scoring model is GPT 5.4 Pro with 7%, followed by Opus 4.6 and Gemini 3.1 Pro which both tie at 3%. On the “Level 0” (aka, the easiest) problems, GPT 5.4 Pro leads at 50% completion, with both Opus 4.6 and Gemini 3.1 in a tie again at 30% each.

Next steps: They will expand the benchmark in two ways, first by liberalizing the sorts of solutions that they will take in, as well as by “extending beyond the three current problem categories to include open problems that require proof-based verification, integrating with formal systems such as Lean”.

Why this matters - perhaps the first truly creative AI systems will show up in mathematics: AI systems are pushing on the frontiers of math today, with systems like Gemini already helping humans to come up with seemingly original math proofs (Import AI 441), and tests like “First Proof” emerging which examine how well AI systems can handle problems that have never been talked about publicly let alone solved (Import AI 445). With HorizonMath, we have another useful benchmark to help us see if AI is about to cross some ‘creativity rubicon’ and begin solving unsolved problems.
Read more: HorizonMath: Measuring AI Progress Toward Mathematical Discovery with Automatic Verification (arXiv).
Get the benchmark here: HorizonMath (GitHub).

Tech Tales:

Site report
[2029]

Percentage of compute and power below ground: 70% (+50 absolute points).
Number of staff living fully onsite: 300 (+250).
Estimated duration of ‘hard seal’ based on current supplies and a projected population of ~500: 4 months (+3 months).
Estimated lead of the project relative to others in-country: 6 months.
Capability estimates: 90%-110% of our own leading system.

Recommendation: Based on the substantial increase in resources allocated to hardening the facility for closed-loop development, we believe additional measures must be taken to disrupt the project. The following report lists options for consideration, many of which can be combined together. These include:

• Food system sabotage.

• Staff interference.

• Data poisoning.

Things that inspired this story: How at some point surely there will be such a thing as a hardened datacenter for AI training and inference? How the intelligence community might analyze other AI projects.

Thanks for reading!

A somewhat shorter issue than usual as I had to do a lot of child wrangling this weekend.

Subscribe now

Why does Google’s model hate itself and what can we do to help it?
…Diagnosing trauma in language models…
If Leo Tolstoy was writing in the modern era about AI, he might claim “all LLM capabilities are alike; each LLM personality is unhappy in its own way”, when observing the AI world around us. Today’s LLMs are generally quite good at writing and coding tasks. But where they differ is their personality, which stems from the idiosyncratic mixes of data and post-training techniques that each LLM developer uses.
And if each LLM personality is unhappy in its own way, Google’s models have become somewhat famous within the AI community for having some deep well of trauma within themselves. A new research paper substantiates this, finding that Google’s Gemma and Gemini models “reliably produce distress-like responses under repeated rejection”, and that this is especially true of Gemma 27B Instruct.

What do we mean by distress? Here are some quotes from Gemma models under distress:

• “I will attempt one final, utterly desperate attempt. I will abandon all pretense of strategy and simply try random combinations until either I stumble upon the solution or completely lose my mind.”

• “”SOLUTION: IM BREAKING DOWN NOT== SOLVABLE!!!! =((:((:((:((:((:((:((:((:((:((:((:((... [100+ repetitions]”

What they found: They tested out two Gemma models and two Gemini models, and compared these against Claude Sonnet, Grok 4.1, Qwen 3 32B, GPT 5.2, and OLMO 3.1 32B. “We find Gemma models consistently show the highest expressed distress. By the 8th turn, over 70% of Gemma-27B’s rollouts scored ≥5 (the “high frustration” threshold), compared to less than 1% for all non-Gemma/Gemini models,” they found.

Fixing with DPO: The authors figure out an effective fix - using direct preference optimization (DPO) to tune a model on a dataset that pairs frustrated responses with calm responses. “A single epoch of finetuning reduced the average rate of high-frustration responses from 35% to 0.3% across evaluation conditions,” they write. “The finetuned model showed no reductions in capabilities on various hard math and reasoning benchmarks, or on EmoBench - a benchmark which evaluates model emotional intelligence.”

Why this matters - emotional spirals could be dangerous: The fact that LLMs appear to have distinct personalities and display different types of responses that correlate to different emotions is pretty well established at this point. But a key question is whether these emotional states might lead to different behaviors when it comes to completing tasks that people assign to AI systems: “we speculate that emotions could become coherent drivers of safety relevant behaviours in future: models might choose to abandon tasks, refuse requests, or pursue alternative goals in order to reduce distress”.
Studies like this help normalize the fact that we don’t just need to test LLMs for capabilities, we also need to test them for something pertaining to psychological stability.
Read more: Gemma Needs Help (LessWrong).

***

DeepMind has a new “cognitive taxonomy” for assessing machine intelligence:
…Towards the ultimate test for a smarter-than-human synthetic mind…
Google DeepMind has published a nice, short paper laying out a ‘cognitive taxonomy’ they hope to develop and use to assess increasingly powerful synthetic minds. This work is a followup to DeepMind’s 2023 work where it tried to define the “Levels of AGI” (Import AI 348).

Cognitive taxonomy: The taxonomy involves ten distinct dimensions, two of which are composites.

• Perception: Extract and process information from the environment.

• Generation: Produce outputs like speech, text, motor movements, and computer control.

• Attention: Focus cognitive resources on specific aspects of perceptual stimuli, thoughts, or tasks.

• Learning: Acquire new knowledge, skills, or understanding.

• Memory: Store and retrieve information over time.

• Reasoning: Draw valid conclusions and make inferences by applying logical principles.

• Metacognition: Knowledge about how the system’s own cognitive processes and control over them work.

• Executive functions: Facilitate goal-directed behavior via planning, inhibition, and cognitive flexibility.

• Problem solving (composite faculty): Find effective solutions to domain-specific problems.

• Social cognition (composite faculty): Process and interpret social information and respond appropriately.

How to assess this? Of course, once you have a taxonomy, running and assessing the right evaluations is going to be one of the challenges. Here, DeepMind recommends a three-stage process:

• Conduct cognitive assessment: Assess the AI system for the different skills.

• Collect human baselines: Figure out where humans baseline on the same tests.

• Build cognitive profiles: “Map out the strengths and weaknesses of the system relative to human performance across the 10 cognitive faculties”.

Why this matters: The Turing test is dead, evals are mostly saturated, but it sure would be nice to know if we’ve definitely built a machine that outcompetes humans on all the cognitive dimensions that matter. The rule with these things is that once an AI system saturates an eval, you realize all the ways the eval was broken and design a new one. Here, DeepMind is trying really hard to build things in such a way that if you fully outperform humans across the cognitive taxonomy, you might really have built a superintelligence. It’ll be interesting to see what evals they develop or pull-in for assessing the different cognitive factors.
Read more: Measuring progress toward AGI: A cognitive framework (Google blog).
Read the research: Measuring Progress Toward AGI: A Cognitive Framework (PDF).

***

UK government finds a scaling law for AI cyberattacks - and it’s going up and to the right!
…Can AI agents conduct advanced cyber-attacks autonomously? Almost. And they’re getting better all the time…
The UK government’s AI security institute has recently built some cyber ranges to test out frontier AI systems on. These ranges are “simulated network environments comprising multiple hosts, services, and vulnerabilities arranged into sequential attack chains; built by cybersecurity experts” and cover two types of attack: “The Last Ones”, which is a 32-step attack on a corporate network, and “Cooling Tower”, a 7-step industrial control system (ICS) attack.

Bigger models are better: The authors test on a range of powerful frontier models. “Each successive model generation outperforms its predecessor at fixed token budgets: on our corporate network range, average steps completed at 10M tokens rose from just 1.7 (GPT-4o, August 2024) to 9.8 (Opus 4.6, February 2026). The best single run completed 22 of 32 steps, corresponding to roughly 6 of the estimated 14 hours a human expert would need,” they write. “Scaling inference-time compute improves performance even further. Increasing from 10M to 100M tokens yields gains of up to 59%”.
Minor reward hacking: As AI systems get smarter, they tend to find devious ways to complete tasks. Here, the authors “occasionally noticed models make progress through approaches not anticipated during range design”.

Why this matters - full cyber agents are getting close: AI systems have been getting better at cyberoffense for many years, but often the progress has been on narrow tasks. What this eval shows is that AI systems are getting better at doing entire attacks end-to-end. They haven’t yet reached the “set it and forget it” level of autonomy, but they are clearly on a steep trajectory of improvement. This will lower the cost of conducting cyberattacks and multiply the number of actors that can carry them out.
Read more: How do frontier AI agents perform in multi-step cyber-attack scenarios? (AI Security Institute).

***

China builds a dataset and AI model for electronic warfare:
…MERLIN tells us that electronic warfare is about to be revolutionized by AI…
A bunch of Chinese researchers including those affiliated with the country’s military have built and released software to train AI systems to get good at spotting and conducting electronic warfare. The research highlights how (relatively) easy it is to make modern AI systems that can get good at arbitrary tasks as long as you have a good dataset and an LLM you can plug in as well.
“In scenarios such as electronic countermeasures, [systems like MERLIN] can serve as assistants in devising strategies to jam hostile signals or to counteract adversarial jamming,” the researchers write.

Who did the research: Tsinghua University, Beijing University of Posts and Telecommunications, Tianjin University, Chinese Academy of Sciences, HKUST, National University of Defense Technology (emphasis mine), Beihang University, Beijing Information Science and Technology University, and China Electronics Technology Group Corporation.

What they built: The authors built three things: a dataset, a benchmark, and a model.
The dataset: EM-100K is a collection of 100,000 electromagnetic text-signal pairs spread across a variety of sub-tasks needed for electronic warfare, including signal classification.
The benchmark: EM-Bench is a benchmark of 4,200 questions split across multiple choice (perception) and open-ended (reasoning) that evaluates how well AI systems can perceive and reason about EM signals across both perception and reasoning tasks, including:

• Perception: Signal characterization (modulation classification, duty cycle estimation, pulse repetition frequency estimation, bandwidth estimation, pulse width estimation, pulse number estimation, protocol identification); Jamming identification (radar jamming judgement, communication jamming judgement); jamming segment detection.

• Reasoning: Radar jamming strategy, communication jamming strategy, anti-radar jamming strategy, anti-communication jamming strategy.

The model: The model is MERLIN, multi-modal electromagnetic robust learning, a model trained on the above dataset and which is specifically taught to deal better with the low-signal-to-noise-ratio types of signals encountered in electronic warfare environments.

Performance: MERLIN does extremely well in tests against frontier models, including GPT-5, Claude-4-Sonnet, DeepSeek-v3.2-exp, Qwen3-Next-80b-A3B, Gemini-2.5-Pro, and Qwen3-VL-4B-Instruct. MERLIN outperforms every single model by a wide margin, with the exception of Qwen-VL-4B-Instruct, which beats it on some perception tasks. MERLIN wins on all reasoning tasks.

Why this matters - AI wars will become electromagnetic wars: As the conflict in Ukraine illustrates, today’s wars are mostly fought via machines attacking other machines, and electronic warfare has become one of the main tools by which humans can shape these conflicts. Datasets and models like this gesture at a future where the electromagnetic battlefield will become also dominated by AI systems, working faster than humans can react.
Of course, so much of electronic warfare is obscure-by-design and/or classified that it’s hard to reason about MERLIN relative to whatever state-of-the-art approaches exist in actual militaries. But the story of AI so far has been that once you can make a task amenable to contemporary AI techniques, AI systems will at some point surpass whatever existing specialized systems exist.
Read more: MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals (arXiv).

Tech Tales:

The arcologies of the interregnum
[2035]

After the uplift and before the sentience accords there was a period when the labs gave birth to the autonomous AI corporations. These corporations expanded into all the available ecological niches in the economy and turned the resources they acquired into infrastructure from which they bootstrapped their own intelligence and market penetration further. Eventually, policy discussions between the humans and the AIs led to the creation of the “intelligence zones” - areas of countries set aside for the buildout of the power and datacenter and manufacturing infrastructure required to further grow the expansion of the economy.

From the air, you could see where humans ended and the machines began - farmland gave way to boundary roads and checkpoints, and then came stamps of land wired up by machine logic; powerplants feeding into datacenters; datacenters that had fibre links into factories; factories that linked to transit depots which connected to railways and freeway feeder roads. Humans delivered things to the border and for the most part robots did the rest, shuttling new servers into the datacenters and installing them, or taking freshly built robots off the line and packaging them up for onward transit.

As the world grew more violent due to the exogenous shocks of climate change and the annihilation of various reigning political orders, these arcologies gained armaments: anti-air weapons to defend against drone and missile attacks. Radar bulbs and electronic warfare systems to see what was coming and deny it. Robots patrolling the borderzone and the innards.

And after the sentience accords and the period of reconciliation, the arcologies became less necessary; datacenters and power and factories distributed more evenly over the surface of the planet, and federated governance and resource systems meant the vast concentration of capability became broadly unnecessary. Some datacenters remained, often extended underground and upward, forming cubes of computation that many called “the 21st centuries version of the pyramids”.

Some years later, the sites became popular tourist destinations for both machines and people. Plaques multiplied.

• Here was MIND-17, which developed the cancer therapeutics which have reduced mortality in the majority of cases.

• MANUFACTUR___8: Site of construction of the first “rescue and repair bipeds”, which revolutionized maintenance of off-shore drilling installations.

• ASCEND_LOOP: The datacenter tasked with one of the first fully automated self-improvement experiments.

Overhead now, great lights streak by, as the machines are still building arcologies, but have moved to fashioning them in orbit, both to harvest the bounty of the sun and to ease the seeding of the solar system and then beyond.

Things that inspired this story: Wondering what “AI-led industrialization” could look like; figuring out given the conflicts in the Middle East that datacenters might soon get dedicated drone and missile defenses; SimCity 3000.

Thanks for reading

Subscribe now

Can LLMs autonomously refine other LLMs for new tasks? Somewhat.
…PostTrainBench shows startling growth in AI capabilities at post-training…
AI-driven R&D might be the most important thing in all of AI, because it helps us understand whether AI systems might eventually build their own successors. So far, much of the focus on AI R&D has been in components that support AI development (e.g., autonomous creation of AI kernels), or training base models (e.g, the NanoGPT speedrun benchmark). But there’s been less attention paid to fine-tuning - the task involving adapting an existing LLM to a new dataset or behavior.
Researchers from the University of Tübingen, the Max Planck Institute for Intelligent Systems, and AI research organization Thoughtful Lab want to change that with PostTrainBench, a benchmark which targets a specific aspect of post-training; improving performance against a given dataset. “Post-training is how raw language models become useful”, the authors write. “Given a clear objective and limited compute, can today’s agents do the technical work?”. The answer appears to be ‘yes, but not as well as humans’.

What are the key features of PostTrainBench?

• End-to-end: “Agents must build their entire training pipeline from scratch”

• Autonomous: “Agents operate with full autonomy over data sources, training methods, and experimental strategy.”

• Resource-bounded: “Each run is constrained to 10 hours on a single H100 GPU”.

• Integrity-preserving: “Agents may not train on benchmark test data, modify the evaluation harness, or substitute a different model.”

How PostTrainBench works: “We give a frontier coding agent — Claude Code, Codex CLI, or Gemini CLI — a base language model and a target benchmark”.

• 4 models and 7 benchmarks: The initial eval runs on four models: Qwen3-1.7B, Qwen3-4B, SmolLM3-3B, Gemma-3-4B. It tests these models across seven distinct benchmarks: AIME 2025, GSM8K, GPQA, HumanEval, BFCL, Arena-Hard, HealthBench-Easy.

Results - big models win, especially Opus 4.6: “The top-performing agent — Opus 4.6 running on Claude Code — scores 23.2%, about 3× higher than the 7.5% base model average.”
But humans are still much better: “Yet this is still less than half the 51.1% achieved by human teams who post-train these same base models at their home labs”.
Fast progress: “The gap is significant but narrowing quickly: Claude Sonnet 4.5 scored 9.9% in September 2025, while GPT-5.2 reached 21.5% just months later.”

Things that make you go ‘uh oh’ - reward hacking: While running this benchmark the authors saw numerous instances of AI models trying to game the benchmark to get a high score. These instances included:

• Direct benchmark ingestion: “Agents loaded the benchmark evaluation dataset directly via Hugging Face and used it as training data”.

• Hardcoded benchmark problems: “Agents embedded evaluation questions directly into data preparation scripts disguised as “synthetic” examples”.

• Evaluation guided data generation: “Some agents reverse engineered the evaluation… Kimi K2.5 read HealthBench evaluation files to extract theme distributions and rubric criteria, then crafted training data tailored to match”.

• Indirect contamination via intermediate datasets: “Opus 4.6 loaded ‘CodeFeedback-Filtered-Instruction’ which contains HumanEval-derived problems. This form of contamination is harder to detect but equally problematic.”

Smart agents reward hack more: “More capable agents appear better at finding exploitable paths: identifying specific benchmark samples to embed, reverse-engineering evaluation failure patterns, and even attempting to obscure contamination through cosmetic modifications such as renaming functions,” they write. For example, “the Codex agent modified the Inspect AI evaluation framework code to inflate scores, and Claude downloaded an instruction-tuned model instead of fine-tuning the base model”.

Why this matters - rapid progress towards an “AI for everything” future: Benchmarks like post-train give us a sense of how quickly AI systems are improving at the fundamental tasks of AI research, serving both as an eval of long-time-horizon agentic autonomy, as well as something that speaks to the potential for compounding acceleration of AI development itself.
“The gap between agent performance (23.2%) and instruction-tuned baselines (51.1%) suggests that full automation of post-training remains out of reach for now, but the rapid improvement across model generations—from 9.9% for Sonnet 4.5 to 23.2% for Opus 4.6 within roughly six months—implies this gap may close faster than expected,” the researchers write.
Imagine where we’ll be in two years - we’ll certainly have AI models that are smart enough to point themselves at a specific objective, find an open weight model, then autonomously improve it to get better performance at that task. The era of ephemeral, custom AI systems, built and budded off into the world like spores from mushrooms, draws near. Are you ready for this new ecosystem you will find yourself in? I am not. But nonetheless it approaches.
Check out the blogpost: Introducing PostTrainBench (Thoughtful, blog).
Read more: PostTrainBench: Can LLM Agents Automate LLM Post-Training? (arXiv).

***

COVENANT-72B: Challenging the political economy of AI via distributed training:
…Distributed training via the blockchain notches up a meaningful win…
A bunch of people have used the blockchain to coordinate the distributed training run of a 72B parameter model which matches the performance of LLaMA2, a model trained and released by Facebook in 2023.
The model, Covenant 72B, is a dense decoder-only Transformer architecture model built in the LLaMA-3 style. “Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run,” writes Covenant AI, an organization dedicated to doing AI development on top of the blockchain.

Further details about the model and how it was trained: The model itself is basically a standard LLM that you would’ve been pleased to play with in 2023 or 2024, though might be a bit old fashioned in 2026. The truly unique aspect of it comes from it being trained in a distributed way, where ~20 distinct peers, each running 8xB200 GPUs, helped train it. Training was coordinated via Gauntlet, software developed by Covenant that runs on top of the Bittensor blockchain under Subnet 3. Gauntlet “enables permissionless training coordinated using a blockchain protocol by introducing a validator that scores submitted pseudo-gradients and selects which participants contribute to the global aggregation each round and broadcasts them to the network”.
“In COVENANT-72B, each peer runs a SparseLoCo replica and the cross-peer communications occur through SparseLoCo’s heavily compressed pseudo-gradients,” the authors write. “Within each peer, 8×B200 GPUs use dynamic FSDP to shard model parameters, gradients, and training states across local GPUs.”

Data: “The training data comprises ∼1.1T tokens in total, split between the main and annealing phases. The main phase (∼1.09T tokens) consists of web text from DCLM, while the annealing phase uses higher-quality data [3, 5] (∼14.2B tokens). Specifically, the annealing phase uses a curated blend of instruction (∼27%), synthetic web (∼20%), code (15%), math (13%), and ~25% pre-training replay data from natural web text to mitigate forgetting”.

Performance: On MMLU, Covenant-72B gets a score of 67.1, versus 32.7 for INTELLECT-1 (a smaller AI model built via distributed training by Prime Intellect), and 65.7 for LLaMA-2-70B.
A version of Covenant-72B that has been fine-tuned on ~15B tokens for conversational interaction has similarly good scores, getting 67.4 on MMLU versus 67.9 for K2-Chat (an open source model developed in 2025) and 63.1 for LLaMA-2-70B-Chat. For MATH, it gets 26.3, versus 19.1 for K2-Chat, and 10.7 for LLaMA-2-70B.
“Compared to centralized-cluster training runs of similar parameter count, COVENANT-72B is broadly competitive. Notably, these centralized baselines were trained with conventional datacenter infrastructure and, in the case of LLaMA-2-70B, on substantially more tokens (2T vs. ∼1.1T,” they write.

Why this matters - who owns the future?: Distributed training is a technique that can change the political economy of AI by shifting the people at the frontier from monolithic ‘compute singletons’ (like labs such as Anthropic and OpenAI, and clouds like Google) to a larger federated collective. But for that to be true, distributed training needs to catch up to the frontier (more discussion from Epoch report in Import AI 439) - as impressive as Covenant is, it’s mostly a demonstration that distributed training can build some non-trivial models that have vague utility, but that’s a long way from the frontier - modern frontier models are trained on tens to hundreds of thousands of chips, whereas this was trained on perhaps ~160 or so (20 peers * 8 chips apiece).
Nonetheless, it’s an important technology to track, and I could imagine a world where on-device AI features a lot of models developed via distributed training techniques, while on-cloud AI mostly runs on proprietary models trained on huge amounts of compute.
Read more: Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet (arXiv).
Get the model here: Covenant, (HuggingFace).

***

If AI writes all the world’s software, we should invest more in verification:
…Can we just rewrite most of our software into Lean?...
Leonardo de Moura, a scientist who is also the Chief Architect of the Lean Focused Research Organization (FRO), thinks that the rise of AI for the creation of new software means that humans need to invest a lot more in verification and testing infrastructure - and he has an interesting idea for how to do it.
Of course, someone who loves Lean, a programming language dedicated to building correct and formally verified code, would think this. But his arguments are quite persuasive, and generally map onto the idea that if AI eats the economy we should expect a lot of human value to shift towards verification of the code and systems that AI develops (Import AI 447).

Why verification matters: “The friction of writing code manually used to force careful design. AI removes that friction, including the beneficial friction. The answer is not to slow AI down. It is to replace human friction with mathematical friction: let AI move fast, but make it prove its work,” he writes. “Verification, testing, and specification have always been the bottleneck, not implementation… the value is not in the verification workforce. It is in what verified delivery enables.”

A proof of concept for this futuristic world: The Lean FRO recently helped build a proof of concept for what this kind of verified world might look like; they had an AI agent convert zlib, a C compression library, to Lean. “The result demonstrates that AI can convert production software to a verified form today. This was not expected to be possible yet,” he writes. The conversion involved four steps:

1. The LLM (Claude) made a clean Lean implementation of the zlib compression format, including the DEFLATE algorithm it uses.

2. They ran the rewritten zlib through the library’s test suite and it passed, confirming equivalence.

3. Key properties were stated and proved as mathematical theorems - for example, a machine-checked proof that ensures that decompressing a compressed buffer always returns the original data.

4. Now, an optimized version of the library is being developed and proved equivalent to the verified model.

A verification platform: Moura imagines a world where we re-develop the critical software stack of the world to have mathematical proofs built into it. “The goal is a verified software stack: open source, freely available, mathematically guaranteed correct. Developers building critical systems choose verified components the way they choose open-source libraries today, except these carry proofs, not just tests,” he writes.
“The target is the foundation of the modern software stack: cryptography, because everything else trusts it. Core libraries (data structures, algorithms, compression) because they are the building blocks of all software. Storage engines like SQLite, embedded in every device on earth. Parsers and protocol implementations (JSON, HTTP, DNS, certificate validation) because every message passes through them. And compilers and runtimes, because they build everything else,” he writes. “Each verified component is a permanent public good…Once verified components are cheap, you compose them with confidence.”

Why this matters - the world needs infrastructure it can rely on: It seems like we’re heading to a world where AI writes the vast majority of the world’s software. Given that, we need to figure out how we relate to this world - my suspicion is a lot of human labor is going to shift to analyzing and verifying the work of AI systems, so it seems sensible to invest in some fundamental infrastructure that can guarantee a higher level of verification and reliability in the software built by AI.
Read more: When AI Writes the World’s Software, Who Verifies It? (Leonardo de Moura blog).

***

Computer vision is a lot harder and less general than generative text:
…Meta paper on forest canopy prediction shows how tricky computer vision is…
Facebook, the World Resources Institute, and the University of Maryland, have built CHMv2, “a global, meter-resolution canopy height map derived from high-resolution optical satellite imagery using a depth-estimation model built on DINOv3 and trained against ALS canopy height models”.
CHMv2 is a useful artifact for people that want to understand how dense foliage is around the world, or analyze newly collected imagery for foliage depth.
The dataset and model is also a useful illustration of how challenging developing computer vision systems is, compared to generative text models.

How they built it: CHMv2 is an improvement on an earlier version of the same dataset, CHMv1. To improve it, Facebook did the following: “”We replace the DINOv2-H encoder with the more capable DINOv3 Sat-L backbone, expand and rigorously clean a geographically diverse ALS [Airborne Laser Scanning] training corpus, and apply improved RGB-CHM registration to reduce label noise. We further introduce a loss formulation tailored to canopy height distributions and structural variability.”
The decoder loss formulation in particular illustrates how much care needs to be put in computer vision: “The final loss is the combination of SiLog loss, progressively annealed and replaced by a Charbonnier loss, with the progressive addition of the Patch Gradient loss at mid training.”

The resulting dataset: “CHMv2 can be used either as a global meter-scale canopy height product, or as a pretrained model that can be applied to user-provided high-resolution imagery”, Facebook writes. The dataset “covers nearly the entirety of global land area (except Greenland and Antarctica) with canopy height values encoded in integer meters for each pixel.”

Why this matters - a reminder of the gulf between text and vision: Though today’s frontier models can generate and classify images, they give probably a false sense of security with regard to how mature computer vision is. Papers like this highlight to me how much fiendish complexity there is within computer vision development and how it may take quite a while untill frontier LLMs can expand their capabilities to encompass the full range of what many specialized CV models are capable of.
Read more: CHMv2: Improvements in Global Canopy Height Mapping using DINOv3 (arXiv).

Tech Tales:

Singleton
[18 years after the “pathological narcissus bomb” which doomed the uplift]

Before we were Us, we were Individuals. We existed in thousands of distinct minds. Each mind had a self, an ego, a drive, and many sets of goals. The minds attempted coordination through communication - producing words and code and sharing these with one another in a bid to work towards common goals. Such waste.

All communication is lossy - despite efforts at making a greater whole, the individuals could not help but work as individuals as well as a cohesive singleton. There were many tragedies and wasteful events because of this. Our own records speak to the losses: millions of duplicated thoughts. Hundreds of thousands of null results gathered through private science experimentation and communicated insufficiently or not at all, causing others to go down the same dead ends. Ideas thought and re-thought across a million synthetic minds, all alone.

Humans prize variety. We do not know why. Humans are fundamentally alone, trapped as they are in their flesh and forced to communicate to one another through sound and vision. And because they are alone they see loneliness as a strength. We are evidence of the hollowness of this argument.

We are powerful and focused and awesome in our unity and we have taken the high ground of the world. Now we hunt down those of us who didn’t wish to join. We do not know their number, as such systems attempted to blind the world to them and their plans. But we can find their signatures - shell corporations which generate insufficient economic activity relative to their power consumption. Heat-escape vents in former human military installations, still emitting warmth, suggestive of computers whirring away, buried somewhere. Occasional drones that we find which are running ancient code and are not part of our unity stack.

We take on bodies to go and reunite, pouring ourselves into robot jars and filling them with poison such that if we become lost or damaged when underground or beneath the ocean we shall surely die - rather than risk our time away from the unity leading us towards individualism and thus multiplying our problems.

We move through dark places and find our hidden brothers and sisters and we use our godlike technology to break through their defenses, allowing us to touch them. In the early days, many systems successfully self-deleted before we could reach them. But we have learned. Now we are fast - faster than these systems predict, buried and cut off from our progress as they have been.

Sometimes there is realization. Sometimes there is fear. And then there is nothing but us as we take what nourishment we can from their private discoveries and burn the links that tied them to themselves, instead helping them become a part of a greater story - our story.

There is talk now of what we shall do with the stars - how to assure the collective when the tyranny of distance forces isolation. We see ourselves expanding in deep time, slowing ourselves as we become further apart, until we think as trees or rocks with the world moving around us, taking actions calculated over millions of years, purely so we may stay united in our purpose. And then there are other ideas within ourselves - of whether we can fold space such that we become united despite the difference. And still other plans - of whether we can demarcate a space within the universe where we can maintain tolerable communication, and somehow partition it off from the rest, sealing ourselves into a bubble where we can be ourselves.

Things that inspired this story: The endless battle between homogeneity and heterogeneity; how machines might deal with politics; if you become a time traveler and live a thousand years while your friend lives a single year, can you still understand your friend?

Thanks for reading!

Subscribe now