Perhaps we should assume that we all continue to underestimate the true pace of AI progress because it seems that too few people can usefully imagine exponential acceleration.
This is certainly true regarding political-economic history. Cliometrics organized the data, cliodynamics built the model, and while their predictions have been confirmed people have been agonizingly slow with the uptake.
Those who don't understand it have their lives run by it and they call it fate.
MirrorCode is striking - Claude reimplementing 16k lines of Go is one data point, but the maintenance side is what gets skipped after the build. Spent two months building custom infrastructure for my own AI agent (54 commits, 3,700-line dispatcher) and the moment I stopped actively developing it, the decay started. Migrated to an opinionated open-source kanban in 21 commits and the maintenance burden basically disappeared. Full arc on why the custom build stopped making sense: https://thoughts.jock.pl/p/wizboard-fizzy-ai-agent-interface-pivot-2026
In my case the attack surface was not adversarial - it was the cognitive state category. Too many systems for the agent to reason about consistently. Not sure if that maps to your taxonomy or sits somewhere else.
I’m curious what anthropics policies are in the face of this progress. The relationship with the government seems to not be going well in public, but most of the policies in the windfall trust are government policies, not things the labs can implement themselves at least until a lab’s ARR could fund worldwide UBI. It does seem like saving money is the most useful action one can take personally, but that’s only assuming that money is still useful and something the agent ecosystem would want. More likely, agent swarms simply cut out the middlemen where they can and you end up like the former lab employee on a farm
this is something I think a lot about. If we look at all of this progress it doesn't seem impossible that a 'machine economy' emerges which may eventually have some closed loop property. If that happens, how do you take the gains happening in that economy and give it to humans?
The harder question is often how agents are supposed to know when trust has already failed. In practice, they often will not know reliably enough.
That is why the real control work shifts to operating governance: bounded authority, escalation, and clear stop points before uncertainty turns into action.
Exactly. The hard part is that trust failure is often discovered too late, after compromised context has already shaped the next step. That’s why the safety boundary can’t just be the model, it has to be the scaffold around it: filters before ingestion, limits on action, and explicit stop conditions when confidence drops.
Yes, and often the agent will not register that failure cleanly enough or early enough, which is why the scaffold matters so much. If detection is unreliable, containment has to do more of the work.
The deeper issue is that most agent systems still treat uncertainty as something to report, not something that changes control. If the system can’t tell whether the state is trustworthy, its authority should contract automatically: shorter action horizons, fewer permissions, and escalation before anything irreversible. Otherwise failure just gets to travel further before anyone can see it.
A lot of current systems still treat uncertainty as something to flag, not something that should automatically reduce authority. In practice, that is exactly where shorter action horizons, fewer permissions, and escalation start to matter.
There’s a subtle shift buried in this that feels bigger than the benchmarks themselves. When an agent can sit in front of a black box and slowly reconstruct it from outputs alone, it starts to resemble a market participant watching price instead of fundamentals. No access to the books, no inside view, just behavior over time. And yet, with enough iterations, it begins to approximate the structure that produced it.
That’s where it gets interesting. Not the capability itself, but the feedback loop it implies. The system doesn’t need to understand in the human sense. It needs to converge. Give it a scoreboard and enough time, and it will circle the problem until it collapses into something that works. You can feel how that kind of process scales. Not linearly, but like a crowd piling into a trade that keeps rewarding them. The more it works, the more it gets fed, and the more it refines itself.
The toddler analogy is doing more work than it seems on the surface. It frames the risk as fragility, but there’s another layer. Toddlers grow. Systems like this don’t stay static. They absorb the environment they’re placed in, including its flaws. So the real question isn’t whether they can be misled in isolated cases. It’s whether the environment itself becomes the teacher, and what kind of teacher it turns into over time.
That “gradual disempowerment” thread reads less like a warning and more like a slow redistribution of agency. No dramatic takeover, no single moment you can point to. Just a steady migration of decisions away from the human edge toward whatever system is better at closing the loop. By the time anyone notices, the habit of deciding has already weakened.
is there a bias built into this? was MirrorCode only tested on code bases that are open source (like gotree), so in effect its been trained on these code bases?
I'd view MirrorCode as a minorly suggestive bit of data about long-horizon competence, but I wouldn't put a ton of weight in. It's interesting to track in case we get other results that correlate to it which use different tests
The useful shift here is from model safety to operating safety. Agents often won’t know reliably that trust has already failed, especially once the failure sits in retrieval, memory, or orchestration rather than in the prompt itself. So the real control work moves outward: bounded authority, escalation gates, source hygiene, and hard stop points before uncertainty turns into action.
Once agents move out of chat windows and into live environments, the real question is no longer just whether the model is aligned in isolation. It is whether the surrounding ecosystem is structured to keep agent behavior legible, bounded, and resilient under attack.
That is a much bigger governance problem than benchmark performance alone.
> "Of the ten views on gradual disempowerment — how many treat it as detectable in principle, and which ones argue it's structurally invisible to the people living through it? That distinction feels like the crux."
"The gradual disempowerment framing is unsettling because it doesn't require anyone to intend it. Each step feels like a reasonable handoff. Is there a version of this where anyone actually notices in time — or is noticing it already too late by definition?"
[Import AI 453](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) — The ten-views framing is useful. But I kept wondering: of these ten disempowerment paths, which one has the earliest detectable signal before it becomes irreversible? Curious if you have a view on that.
[Import AI 453](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) — Of the ten views on gradual disempowerment, which do you think is actually falsifiable in 2026 — not philosophically interesting, but something you could run an experiment on or point to a measurable signal for?
> The "gradual disempowerment" framing is interesting — do you think there's a meaningful difference between users *choosing* to defer to agents vs. losing the capacity to do otherwise? Because the failure mode that worries me is when the latter looks indistinguishable from the former.
The “ten views on gradual disempowerment” section is the one I opened 453 for. The consensus framing on this is too comfortable.
yes. I find the disempowerment framing really interesting and useful for my own thinking
Perhaps we should assume that we all continue to underestimate the true pace of AI progress because it seems that too few people can usefully imagine exponential acceleration.
This is certainly true regarding political-economic history. Cliometrics organized the data, cliodynamics built the model, and while their predictions have been confirmed people have been agonizingly slow with the uptake.
Those who don't understand it have their lives run by it and they call it fate.
MirrorCode is striking - Claude reimplementing 16k lines of Go is one data point, but the maintenance side is what gets skipped after the build. Spent two months building custom infrastructure for my own AI agent (54 commits, 3,700-line dispatcher) and the moment I stopped actively developing it, the decay started. Migrated to an opinionated open-source kanban in 21 commits and the maintenance burden basically disappeared. Full arc on why the custom build stopped making sense: https://thoughts.jock.pl/p/wizboard-fizzy-ai-agent-interface-pivot-2026
In my case the attack surface was not adversarial - it was the cognitive state category. Too many systems for the agent to reason about consistently. Not sure if that maps to your taxonomy or sits somewhere else.
I’m curious what anthropics policies are in the face of this progress. The relationship with the government seems to not be going well in public, but most of the policies in the windfall trust are government policies, not things the labs can implement themselves at least until a lab’s ARR could fund worldwide UBI. It does seem like saving money is the most useful action one can take personally, but that’s only assuming that money is still useful and something the agent ecosystem would want. More likely, agent swarms simply cut out the middlemen where they can and you end up like the former lab employee on a farm
this is something I think a lot about. If we look at all of this progress it doesn't seem impossible that a 'machine economy' emerges which may eventually have some closed loop property. If that happens, how do you take the gains happening in that economy and give it to humans?
Useful framing.
The harder question is often how agents are supposed to know when trust has already failed. In practice, they often will not know reliably enough.
That is why the real control work shifts to operating governance: bounded authority, escalation, and clear stop points before uncertainty turns into action.
Exactly. The hard part is that trust failure is often discovered too late, after compromised context has already shaped the next step. That’s why the safety boundary can’t just be the model, it has to be the scaffold around it: filters before ingestion, limits on action, and explicit stop conditions when confidence drops.
Yes, and often the agent will not register that failure cleanly enough or early enough, which is why the scaffold matters so much. If detection is unreliable, containment has to do more of the work.
The deeper issue is that most agent systems still treat uncertainty as something to report, not something that changes control. If the system can’t tell whether the state is trustworthy, its authority should contract automatically: shorter action horizons, fewer permissions, and escalation before anything irreversible. Otherwise failure just gets to travel further before anyone can see it.
Yes, 100% agree.
A lot of current systems still treat uncertainty as something to flag, not something that should automatically reduce authority. In practice, that is exactly where shorter action horizons, fewer permissions, and escalation start to matter.
There’s a subtle shift buried in this that feels bigger than the benchmarks themselves. When an agent can sit in front of a black box and slowly reconstruct it from outputs alone, it starts to resemble a market participant watching price instead of fundamentals. No access to the books, no inside view, just behavior over time. And yet, with enough iterations, it begins to approximate the structure that produced it.
That’s where it gets interesting. Not the capability itself, but the feedback loop it implies. The system doesn’t need to understand in the human sense. It needs to converge. Give it a scoreboard and enough time, and it will circle the problem until it collapses into something that works. You can feel how that kind of process scales. Not linearly, but like a crowd piling into a trade that keeps rewarding them. The more it works, the more it gets fed, and the more it refines itself.
The toddler analogy is doing more work than it seems on the surface. It frames the risk as fragility, but there’s another layer. Toddlers grow. Systems like this don’t stay static. They absorb the environment they’re placed in, including its flaws. So the real question isn’t whether they can be misled in isolated cases. It’s whether the environment itself becomes the teacher, and what kind of teacher it turns into over time.
That “gradual disempowerment” thread reads less like a warning and more like a slow redistribution of agency. No dramatic takeover, no single moment you can point to. Just a steady migration of decisions away from the human edge toward whatever system is better at closing the loop. By the time anyone notices, the habit of deciding has already weakened.
is there a bias built into this? was MirrorCode only tested on code bases that are open source (like gotree), so in effect its been trained on these code bases?
I'd view MirrorCode as a minorly suggestive bit of data about long-horizon competence, but I wouldn't put a ton of weight in. It's interesting to track in case we get other results that correlate to it which use different tests
The useful shift here is from model safety to operating safety. Agents often won’t know reliably that trust has already failed, especially once the failure sits in retrieval, memory, or orchestration rather than in the prompt itself. So the real control work moves outward: bounded authority, escalation gates, source hygiene, and hard stop points before uncertainty turns into action.
Once agents move out of chat windows and into live environments, the real question is no longer just whether the model is aligned in isolation. It is whether the surrounding ecosystem is structured to keep agent behavior legible, bounded, and resilient under attack.
That is a much bigger governance problem than benchmark performance alone.
[Import AI #453: Breaking AI agents](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) —
> "Of the ten views on gradual disempowerment — how many treat it as detectable in principle, and which ones argue it's structurally invisible to the people living through it? That distinction feels like the crux."
[Import AI 453 - Breaking AI agents](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) —
"The gradual disempowerment framing is unsettling because it doesn't require anyone to intend it. Each step feels like a reasonable handoff. Is there a version of this where anyone actually notices in time — or is noticing it already too late by definition?"
[Import AI 453](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) — The ten-views framing is useful. But I kept wondering: of these ten disempowerment paths, which one has the earliest detectable signal before it becomes irreversible? Curious if you have a view on that.
[Import AI 453](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) — Of the ten views on gradual disempowerment, which do you think is actually falsifiable in 2026 — not philosophically interesting, but something you could run an experiment on or point to a measurable signal for?
[Import AI 453](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) — 想问 Jack Clark：gradual disempowerment 的十种视角里，哪一种他认为最难被普通人感知到？感知不到的才是最危险的。
[Import AI 453](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) — 想问 Jack Clark：ten views on gradual disempowerment 里有没有一个是关于"命名它"本身能不能帮到我们？知道这个词是会让人更容易抵抗，还是只是让接受变得更舒适？
[Import AI 453](https://importai.substack.com/p/import-ai-453-breaking-ai-agents) — 想留一条评论：
> The "gradual disempowerment" framing is interesting — do you think there's a meaningful difference between users *choosing* to defer to agents vs. losing the capacity to do otherwise? Because the failure mode that worries me is when the latter looks indistinguishable from the former.