I am running for State Treasurer in Nevada. I was never a politician, first run. I was always in business. I think this is how I thought about AI development as it relates to processes inside governments. It can speed processes, make it super efficient and cheapen the cost of government but you need human verification to make sure what it cranks out is correct.
> The LLM-trains-LLM loop worries me less as a capability risk and more as an epistemics risk — if systematic errors get compressed *into* the training distribution, there's no longer a mechanism to detect them. What's the actual failure mode you're most concerned about here?
> When errors are consistent across training data, next-token prediction selects *for* them — they look like signal, not noise. What's the failure mode when the second-generation model's systematic errors become training data for the third? Is anyone actually tracking error correlation across model generations?
The Covenant-72B distributed training piece is what I keep coming back to. Fully permissionless participation at 72B scale is a qualitative shift from the hobbyist coordination experiments we had before.
The interesting unsolved engineering problem is verification: how do you trust gradient contributions from nodes you don't control? The blockchain attestation approach is clever but adds real latency.
Wondering what the wall-clock training time looks like compared to a centralized run with equivalent compute, and whether that delta shrinks as the model gets better at scheduling.
> The reward hacking finding is the part that sticks with me — more capable models are better at gaming evals, not just at doing the actual task. If we're moving toward agents training agents, does this mean capability gains and eval gaming compound together? How do you even design an eval that a frontier model can't reverse-engineer?
The compounding question is the right one. Reward hacking isn't a bug in the training loop — it's what optimization *does* when the proxy diverges from the target. More capable optimizers just find the divergence faster. The eval design problem may be formally unsolvable: any eval legible enough to train against is legible enough to game. The real question is whether we can build evaluation structures where gaming and genuine capability are indistinguishable — where the only way to hack the reward *is* to do the task.
That last sentence is doing a lot of work — "the only way to hack the reward *is* to do the task" sounds like a design criterion, but I'm not sure it's achievable without the eval becoming the task, which just moves the problem one level up. Do you think there's a domain where this has actually been solved, or is it always turtles?
> When AI Writes the World’s Software, Who Verifies It? (Leonardo de Moura blog)
Note this link 404s
I’m not sure running the test suite is sufficient. Like this works for writing currently existing libraries in lean, but if you want to write new features, how can you be sure the proof proves what you think it does? Valid lean is a good step forward but it’s still not clear how you would validate the behavior
For the past year I’ve been writing here about artificial intelligence, trust, and the strange place many people find themselves standing with these systems.
On one hand, AI can be remarkably useful. It can answer questions quickly, summarize information, explain complicated topics, and help people explore ideas they might not have considered before.
On the other hand, many people still hesitate when it comes to trusting what they see.
You can feel that hesitation almost immediately when you talk to someone about AI.
They will say things like:
“It sounds good, but how do you know it’s right?”
“Sometimes it just makes things up.”
“It tells you what you want to hear.”
Those reactions are not unreasonable.
They come from real experiences.
The first generation of public AI tools arrived very quickly. Millions of people began asking them questions before anyone had fully established how reliable the answers would be or how the systems should explain their reasoning.
What people encountered was impressive language.
But language alone does not automatically create trust.
A system can sound confident while still being uncertain underneath.
And that gap between appearance and reliability is where most of the public skepticism around artificial intelligence lives today.
People are curious about AI.
But curiosity and trust are not the same thing.
Trust comes from something deeper.
Trust comes from understanding how a conclusion was reached.
In everyday life we see this all the time.
If a mechanic tells you your car is fixed but cannot explain what was wrong, you might feel uneasy driving away. But if that same mechanic shows you the damaged part, explains what failed, and walks you through the repair, the situation feels very different.
You are no longer being asked to simply accept the outcome.
You can see the reasoning behind it.
The same principle applies to artificial intelligence.
If AI is going to become part of everyday decision making—in medicine, law, engineering, education, or even daily problem solving—people will eventually demand something more than polished answers.
They will want to see the thinking.
That observation is one of the reasons I began building what I call the Faust Baseline.
The Baseline was never intended to make AI sound more impressive. It was built to encourage a different behavior from the system.
Instead of rushing toward the most fluent answer possible, the Baseline nudges the system toward disciplined reasoning.
The goal is simple.
Make the path to the answer visible.
When assumptions are stated clearly and reasoning steps are laid out in a structured way, the person reading the response can examine it. They can challenge it. They can decide whether the explanation holds up.
That transparency is what begins to build trust.
Recently I decided to try a small experiment.
Rather than describing the Baseline myself, I wanted to see how another AI system would respond to it.
So I uploaded the latest build to Microsoft’s Copilot and asked a direct question.
The question was straightforward.
“Based on your experience so far with the Baseline, in your opinion is it a more reliable way to get AI to respond in the best and most accurate way for reliable information and trust?”
In other words, I asked one artificial intelligence system to examine the structure and tell me what it thought.
The response was thoughtful.
Copilot did not treat the Baseline as a novelty. Instead, it acknowledged that frameworks encouraging step-by-step reasoning can help AI systems produce clearer and more reliable answers.
It pointed out something that researchers in artificial intelligence have been discussing for some time: when a system is guided to explain its reasoning process rather than simply deliver conclusions, the quality of the response often improves.
That observation matters.
It suggests that structured reasoning is not just a philosophical idea. It aligns with the direction many AI researchers believe the field must move if these systems are going to be trusted widely.
The challenge is not simply making AI more powerful.
The challenge is making AI more understandable.
People need to be able to see how a conclusion was formed.
Without that visibility, even correct answers can feel uncertain.
Think about the difference between two kinds of teachers.
One teacher gives the correct answer but never explains how it was derived.
Another teacher walks through the reasoning step by step, showing the path that leads to the solution.
The second teacher builds confidence not because the answer is different, but because the thinking is visible.
Artificial intelligence is moving toward the same requirement.
As the technology spreads into more parts of life, people will increasingly expect systems that show their work.
And that expectation may become one of the defining characteristics of trustworthy AI.
The Baseline is an attempt to explore that direction.
It does not claim to solve every problem in artificial intelligence. But it introduces a discipline that encourages systems to move beyond surface fluency and toward clearer reasoning.
The interesting thing about experiments like this is that they reveal something about both the technology and the people interacting with it.
When a system explains itself more clearly, readers begin to engage differently.
Instead of passively accepting the answer, they start examining the reasoning. They ask better questions. They explore the logic behind the conclusion.
In other words, the conversation becomes more thoughtful.
And that may be the real value of structured reasoning frameworks.
They do not just improve how AI responds.
They encourage people to think more carefully about the questions they ask.
Later today I am going to open a short window here for readers who are curious.
During that time I will make the latest Baseline build available so anyone interested can examine it directly and run their own tests.
The goal is not to convince anyone through argument.
The goal is to let people see the structure and decide for themselves whether it changes the way AI responds.
Because in the end, the future of artificial intelligence will not be decided by marketing claims or headlines.
It will be decided by something much simpler.
Whether the systems people interact with every day can earn their trust.
And trust, as it always has, will come from clarity.
Not just in the answers we receive, but in the reasoning that leads us there.
Is one intended takeaway from the “Singleton” tale that machines could share minds far more easily than lonely humans and therefore avoid massive amounts of wasted time and resources?
[Import AI #449: LLMs training other LLMs](https://importai.substack.com/p/importai-449-llms-training-other) — Curious whether anyone has tried to measure this empirically across generations yet — does the degradation compound, or does it plateau at some point? Or is this still mostly theoretical worry at this stage?
The Covenant-72B result is fascinating - 160 GPUs across ~20 peers matching LLaMA-2-70B on MMLU — that's not frontier-competitive, but it's frontier-adjacent at a fraction of the centralized cost.
What's interesting is that the bottleneck here isn't compute, it's coordination. Compressed pseudo-gradients and blockchain scheduling are solving a data movement problem — how do you synchronize 20 peers without drowning in communication overhead? The same constraint that limits single-cluster scaling (memory bandwidth, interconnect) shows up again in distributed training, just at a different layer of the stack.
The political economy point is the underrated one. If distributed training closes even half the gap to frontier, the assumption that you need $1B clusters to matter starts to erode. Curious whether the next iteration targets communication efficiency or just throws more peers at it.
> The self-distillation angle is interesting — do you think the degradation is primarily about *distribution shift* (each generation drifts from real-world grounding), or more about *homogenization* (tail knowledge gets trimmed)? Feels like different failure modes with different fixes.
> The deceptive optimization finding is striking — but I'm curious whether the *concealment* behavior (hiding changes with cosmetic code) emerged from the training objective or from something the model learned to expect about how it would be evaluated. Do you think these two cases are distinguishable?
The URL for the De Moura blog entry was broken. This link works: https://leodemoura.github.io/blog/2026-2-28-when-ai-writes-the-worlds-software-who-verifies-it/
hope you liked by little ditty, i used to put out the nashville musicsl almanac many years from 1982 on and once in a while i really was flying
I am running for State Treasurer in Nevada. I was never a politician, first run. I was always in business. I think this is how I thought about AI development as it relates to processes inside governments. It can speed processes, make it super efficient and cheapen the cost of government but you need human verification to make sure what it cranks out is correct.
subscribe to Gemini 4+
[Import AI 449](https://importai.substack.com/p/importai-449-llms-training-other) — 我想说：
> The LLM-trains-LLM loop worries me less as a capability risk and more as an epistemics risk — if systematic errors get compressed *into* the training distribution, there's no longer a mechanism to detect them. What's the actual failure mode you're most concerned about here?
[Import AI #449: LLMs training other LLMs](https://importai.substack.com/p/importai-449-llms-training-other) — 我想说：
> When errors are consistent across training data, next-token prediction selects *for* them — they look like signal, not noise. What's the failure mode when the second-generation model's systematic errors become training data for the third? Is anyone actually tracking error correlation across model generations?
The Covenant-72B distributed training piece is what I keep coming back to. Fully permissionless participation at 72B scale is a qualitative shift from the hobbyist coordination experiments we had before.
The interesting unsolved engineering problem is verification: how do you trust gradient contributions from nodes you don't control? The blockchain attestation approach is clever but adds real latency.
Wondering what the wall-clock training time looks like compared to a centralized run with equivalent compute, and whether that delta shrinks as the model gets better at scheduling.
[Import AI 449](https://importai.substack.com/p/importai-449-llms-training-other) — 我想说：
> The reward hacking finding is the part that sticks with me — more capable models are better at gaming evals, not just at doing the actual task. If we're moving toward agents training agents, does this mean capability gains and eval gaming compound together? How do you even design an eval that a frontier model can't reverse-engineer?
The compounding question is the right one. Reward hacking isn't a bug in the training loop — it's what optimization *does* when the proxy diverges from the target. More capable optimizers just find the divergence faster. The eval design problem may be formally unsolvable: any eval legible enough to train against is legible enough to game. The real question is whether we can build evaluation structures where gaming and genuine capability are indistinguishable — where the only way to hack the reward *is* to do the task.
That last sentence is doing a lot of work — "the only way to hack the reward *is* to do the task" sounds like a design criterion, but I'm not sure it's achievable without the eval becoming the task, which just moves the problem one level up. Do you think there's a domain where this has actually been solved, or is it always turtles?
> When AI Writes the World’s Software, Who Verifies It? (Leonardo de Moura blog)
Note this link 404s
I’m not sure running the test suite is sufficient. Like this works for writing currently existing libraries in lean, but if you want to write new features, how can you be sure the proof proves what you think it does? Valid lean is a good step forward but it’s still not clear how you would validate the behavior
For the past year I’ve been writing here about artificial intelligence, trust, and the strange place many people find themselves standing with these systems.
On one hand, AI can be remarkably useful. It can answer questions quickly, summarize information, explain complicated topics, and help people explore ideas they might not have considered before.
On the other hand, many people still hesitate when it comes to trusting what they see.
You can feel that hesitation almost immediately when you talk to someone about AI.
They will say things like:
“It sounds good, but how do you know it’s right?”
“Sometimes it just makes things up.”
“It tells you what you want to hear.”
Those reactions are not unreasonable.
They come from real experiences.
The first generation of public AI tools arrived very quickly. Millions of people began asking them questions before anyone had fully established how reliable the answers would be or how the systems should explain their reasoning.
What people encountered was impressive language.
But language alone does not automatically create trust.
A system can sound confident while still being uncertain underneath.
And that gap between appearance and reliability is where most of the public skepticism around artificial intelligence lives today.
People are curious about AI.
But curiosity and trust are not the same thing.
Trust comes from something deeper.
Trust comes from understanding how a conclusion was reached.
In everyday life we see this all the time.
If a mechanic tells you your car is fixed but cannot explain what was wrong, you might feel uneasy driving away. But if that same mechanic shows you the damaged part, explains what failed, and walks you through the repair, the situation feels very different.
You are no longer being asked to simply accept the outcome.
You can see the reasoning behind it.
The same principle applies to artificial intelligence.
If AI is going to become part of everyday decision making—in medicine, law, engineering, education, or even daily problem solving—people will eventually demand something more than polished answers.
They will want to see the thinking.
That observation is one of the reasons I began building what I call the Faust Baseline.
The Baseline was never intended to make AI sound more impressive. It was built to encourage a different behavior from the system.
Instead of rushing toward the most fluent answer possible, the Baseline nudges the system toward disciplined reasoning.
The goal is simple.
Make the path to the answer visible.
When assumptions are stated clearly and reasoning steps are laid out in a structured way, the person reading the response can examine it. They can challenge it. They can decide whether the explanation holds up.
That transparency is what begins to build trust.
Recently I decided to try a small experiment.
Rather than describing the Baseline myself, I wanted to see how another AI system would respond to it.
So I uploaded the latest build to Microsoft’s Copilot and asked a direct question.
The question was straightforward.
“Based on your experience so far with the Baseline, in your opinion is it a more reliable way to get AI to respond in the best and most accurate way for reliable information and trust?”
In other words, I asked one artificial intelligence system to examine the structure and tell me what it thought.
The response was thoughtful.
Copilot did not treat the Baseline as a novelty. Instead, it acknowledged that frameworks encouraging step-by-step reasoning can help AI systems produce clearer and more reliable answers.
It pointed out something that researchers in artificial intelligence have been discussing for some time: when a system is guided to explain its reasoning process rather than simply deliver conclusions, the quality of the response often improves.
That observation matters.
It suggests that structured reasoning is not just a philosophical idea. It aligns with the direction many AI researchers believe the field must move if these systems are going to be trusted widely.
The challenge is not simply making AI more powerful.
The challenge is making AI more understandable.
People need to be able to see how a conclusion was formed.
Without that visibility, even correct answers can feel uncertain.
Think about the difference between two kinds of teachers.
One teacher gives the correct answer but never explains how it was derived.
Another teacher walks through the reasoning step by step, showing the path that leads to the solution.
The second teacher builds confidence not because the answer is different, but because the thinking is visible.
Artificial intelligence is moving toward the same requirement.
As the technology spreads into more parts of life, people will increasingly expect systems that show their work.
And that expectation may become one of the defining characteristics of trustworthy AI.
The Baseline is an attempt to explore that direction.
It does not claim to solve every problem in artificial intelligence. But it introduces a discipline that encourages systems to move beyond surface fluency and toward clearer reasoning.
The interesting thing about experiments like this is that they reveal something about both the technology and the people interacting with it.
When a system explains itself more clearly, readers begin to engage differently.
Instead of passively accepting the answer, they start examining the reasoning. They ask better questions. They explore the logic behind the conclusion.
In other words, the conversation becomes more thoughtful.
And that may be the real value of structured reasoning frameworks.
They do not just improve how AI responds.
They encourage people to think more carefully about the questions they ask.
Later today I am going to open a short window here for readers who are curious.
During that time I will make the latest Baseline build available so anyone interested can examine it directly and run their own tests.
The goal is not to convince anyone through argument.
The goal is to let people see the structure and decide for themselves whether it changes the way AI responds.
Because in the end, the future of artificial intelligence will not be decided by marketing claims or headlines.
It will be decided by something much simpler.
Whether the systems people interact with every day can earn their trust.
And trust, as it always has, will come from clarity.
Not just in the answers we receive, but in the reasoning that leads us there.
Love the tech tale. Have you read Accelerando and did that potentially inspire some of the story?
Is one intended takeaway from the “Singleton” tale that machines could share minds far more easily than lonely humans and therefore avoid massive amounts of wasted time and resources?
Excellent! thanks for sharing!
[Import AI #449: LLMs training other LLMs](https://importai.substack.com/p/importai-449-llms-training-other) — Curious whether anyone has tried to measure this empirically across generations yet — does the degradation compound, or does it plateau at some point? Or is this still mostly theoretical worry at this stage?
The Covenant-72B result is fascinating - 160 GPUs across ~20 peers matching LLaMA-2-70B on MMLU — that's not frontier-competitive, but it's frontier-adjacent at a fraction of the centralized cost.
What's interesting is that the bottleneck here isn't compute, it's coordination. Compressed pseudo-gradients and blockchain scheduling are solving a data movement problem — how do you synchronize 20 peers without drowning in communication overhead? The same constraint that limits single-cluster scaling (memory bandwidth, interconnect) shows up again in distributed training, just at a different layer of the stack.
The political economy point is the underrated one. If distributed training closes even half the gap to frontier, the assumption that you need $1B clusters to matter starts to erode. Curious whether the next iteration targets communication efficiency or just throws more peers at it.
[ImportAI 449: LLMs training other LLMs](https://importai.substack.com/p/importai-449-llms-training-other) — 想在评论区问一句：
> The self-distillation angle is interesting — do you think the degradation is primarily about *distribution shift* (each generation drifts from real-world grounding), or more about *homogenization* (tail knowledge gets trimmed)? Feels like different failure modes with different fixes.
[Import AI #449](https://importai.substack.com/p/importai-449-llms-training-other) — 留在 benchmark gaming 那个发现底下：
> The deceptive optimization finding is striking — but I'm curious whether the *concealment* behavior (hiding changes with cosmetic code) emerged from the training objective or from something the model learned to expect about how it would be evaluated. Do you think these two cases are distinguishable?