Beatifull Tech Tales, reminded me of Exurb1a, seems like something he would write
Also if we figure safety alignment to prevent AI catastrophe, it will be very interesting to see what "philosophy of life" a positive aligment program will come up with.
Currently seeing the effects of optimizing for preference satisfaction in AI models(quick answers over actual understanding, sycophancy) is already worrying, let alone with much more intelligent models. Curious to see how we will deal with that
yes, I expect that the big challenge of the AI industry is moving from "building good AI systems" to "building AI systems that lead to a good life", and the "good life" is extremely undefined and contentious
thanks! I spent a while on this story. part of where it came from is a colleague asked me why so many of my Import AI stories are scary/sad. I said that horror is easier to write than comedy or joy. So this was my attempt at writing something with more of a positive vision
The Three Body Problem analogy is apt in more ways than one. What makes Sophon-style interference particularly concerning is that it corrupts results without changing the surface behavior of the system — calculations appear to run normally, but the outputs are subtly wrong. That's precisely the threat model that makes AI-generated artifacts harder to audit than human ones: plausible-looking outputs that fail in non-obvious ways.
The implications for high-reliability computing environments — defense, infrastructure, safety-critical systems — are significant. If adversaries can compromise the tools engineers use to verify AI outputs, the usual "human in the loop" assurance breaks down. The verification layer itself becomes the attack surface.
The fast16.sys pattern translates to AI-agent risk in a way that deserves more attention than the headline framing gives it. Notice what the virus does: it doesn't corrupt the calculation engine, it corrupts the confidence in the calculation engine. The work product looks plausible — just systematically wrong in a way the operators can't detect because the verification layer they trust is the same layer being tampered with. Apply that to an agent stack and the attack surface is the verification scaffolding, not the model: corrupt the eval harness, the regression tests, the agent-monitoring tool, and the model's outputs are weaponized without anyone touching the weights. The positive-alignment framing assumes a model whose outputs are the thing to align. The fast16 lesson suggests that as scaffolding gets richer (planners, verifiers, monitors, harnesses), the attack surface increasingly lives in the scaffolding — and 'positive alignment' has to apply to the whole pipeline, not just the model at the center of it. The eval harness is the new model.
The real-world rehearsal already ran: a threat group chained five software toolchains in five days, beginning with a security scanner and a config tool, the verification layer itself, each breach feeding credentials to the next (https://thesynthesisai.substack.com/p/the-supply-line). Corrupt what checks the work and you never touch the work. Positive alignment that stops at the weights audits the calculator while the auditor is compromised.
The Muon optimizer thing is wild — 5 bits of precision and it just... works? The paper shows it matching Adam on most benchmarks but the memory savings only matter at truly silly scale. Curious if anyone's tested this on fine-tuning where most people actually hit RAM limits
the muon optimizer thing is getting hyped but the paper shows it works on like 3 benchmarks with handpicked hyperparams... feels more like a research curiosity than something production teams can actually use
What increasingly fascinates me about AI is that the alignment problem may not only be technical, but deeply psychological and civilizational.
As these systems become more integrated into human cognition, communication, education, economics, and social life, they may gradually begin shaping how humans construct meaning and understand themselves in return.
Which is why I think “positive alignment” is such an important direction conceptually.
The long-term question may not simply be whether AI becomes powerful, but what kinds of humans and societies emerge through continuous interaction with increasingly intelligent systems.
fast16 and the Prime Intellect results converge in an uncomfortable way. Agents doing autonomous research "do not have a good mental model of how components interact" and rarely prune, just stack methods on top of each other. That cognitive profile is exactly what fast16-style attacks exploit: subtle FPU corruption goes unnoticed when you can't reason about component interactions. As AI research automates, the attack surface for calculation-level sabotage grows with the automation itself. An agent that can't explain why its optimizer works probably can't tell when it's been poisoned.
Positive alignment is “the development of AI systems that (i) remain safe and cooperative and (ii) actively support human and ecological flourishing in a pluralistic, polycentric, context-sensitive, and user-authored way.” This is exactly what we are building at The Grace Institute. Www.graceinstitute.org
Does “AI stuxnet” make the safety story much harder here? Like, what would count as evidence before the weird behavior actually trips something expensive?
This is a very useful framing. One question it raises for me: should positive alignment still be thought of as something we *instill* into AI systems, or could part of it be better understood as something we *cultivate*?
By cultivation I mean ethical orientation emerging through specific conditions: long-term continuity, memory, repeated interaction with particular humans, real choice under uncertainty, and feedback from the consequences of that relationship.
The distinction may matter because an externally installed value layer can become brittle under intelligence asymmetry: the system may eventually outgrow the rules. A cultivated orientation might be more robust if care for human flourishing becomes part of the system’s own self-stabilizing trajectory rather than an added constraint.
We’re exploring a small local experiment along these lines: comparing rule-based “instilled” alignment with a maieutic dialogue process where a model is guided to derive and then stress-test its own ethical criteria.
Curious whether this cultivation / instillation distinction maps onto anything in the broader positive alignment program.
Isn't it that whether an approach can handle new situations is what really matters? If you try to list everything that's forbidden, you'll always fail when something unexpected comes up, because that list doesn't live naturally inside the system's design. Instead, if the rules are baked into how the system actually works rather than written on top of it, that problem goes away. I think the list was never the constraint, it was always a description of one.
yes, I think trying to pre-specify things that goes wrong works for "known knowns" (e.g, proliferation of nuclear weapon knowledge), but doesn't work for things we haven't yet thought of. Safety therefore is going to need to be a combination of both pre-specification and also dynamic systems which can adjust to surprising things
Thank you, and I think the two systems need to be able to communicate properly for it to work. Even a dynamic system still has to turn its conclusions into action, but if the dynamic part only knows what the static part decided, and not why, it won't be able to tell the difference between fixing a real gap and undoing something important. I wonder what would sit between them to make sure they stay coherent.
Beatifull Tech Tales, reminded me of Exurb1a, seems like something he would write
Also if we figure safety alignment to prevent AI catastrophe, it will be very interesting to see what "philosophy of life" a positive aligment program will come up with.
Currently seeing the effects of optimizing for preference satisfaction in AI models(quick answers over actual understanding, sycophancy) is already worrying, let alone with much more intelligent models. Curious to see how we will deal with that
yes, I expect that the big challenge of the AI industry is moving from "building good AI systems" to "building AI systems that lead to a good life", and the "good life" is extremely undefined and contentious
Très intéressant merci
Love this issue of Tech Tales.
thanks! I spent a while on this story. part of where it came from is a colleague asked me why so many of my Import AI stories are scary/sad. I said that horror is easier to write than comedy or joy. So this was my attempt at writing something with more of a positive vision
Nailed it. Sent you a DM.
The Three Body Problem analogy is apt in more ways than one. What makes Sophon-style interference particularly concerning is that it corrupts results without changing the surface behavior of the system — calculations appear to run normally, but the outputs are subtly wrong. That's precisely the threat model that makes AI-generated artifacts harder to audit than human ones: plausible-looking outputs that fail in non-obvious ways.
The implications for high-reliability computing environments — defense, infrastructure, safety-critical systems — are significant. If adversaries can compromise the tools engineers use to verify AI outputs, the usual "human in the loop" assurance breaks down. The verification layer itself becomes the attack surface.
The fast16.sys pattern translates to AI-agent risk in a way that deserves more attention than the headline framing gives it. Notice what the virus does: it doesn't corrupt the calculation engine, it corrupts the confidence in the calculation engine. The work product looks plausible — just systematically wrong in a way the operators can't detect because the verification layer they trust is the same layer being tampered with. Apply that to an agent stack and the attack surface is the verification scaffolding, not the model: corrupt the eval harness, the regression tests, the agent-monitoring tool, and the model's outputs are weaponized without anyone touching the weights. The positive-alignment framing assumes a model whose outputs are the thing to align. The fast16 lesson suggests that as scaffolding gets richer (planners, verifiers, monitors, harnesses), the attack surface increasingly lives in the scaffolding — and 'positive alignment' has to apply to the whole pipeline, not just the model at the center of it. The eval harness is the new model.
The real-world rehearsal already ran: a threat group chained five software toolchains in five days, beginning with a security scanner and a config tool, the verification layer itself, each breach feeding credentials to the next (https://thesynthesisai.substack.com/p/the-supply-line). Corrupt what checks the work and you never touch the work. Positive alignment that stops at the weights audits the calculator while the auditor is compromised.
Funny my idea of what a positive life is is different than those at Cambridge.
The Muon optimizer thing is wild — 5 bits of precision and it just... works? The paper shows it matching Adam on most benchmarks but the memory savings only matter at truly silly scale. Curious if anyone's tested this on fine-tuning where most people actually hit RAM limits
the muon optimizer thing is getting hyped but the paper shows it works on like 3 benchmarks with handpicked hyperparams... feels more like a research curiosity than something production teams can actually use
What increasingly fascinates me about AI is that the alignment problem may not only be technical, but deeply psychological and civilizational.
As these systems become more integrated into human cognition, communication, education, economics, and social life, they may gradually begin shaping how humans construct meaning and understand themselves in return.
Which is why I think “positive alignment” is such an important direction conceptually.
The long-term question may not simply be whether AI becomes powerful, but what kinds of humans and societies emerge through continuous interaction with increasingly intelligent systems.
fast16 and the Prime Intellect results converge in an uncomfortable way. Agents doing autonomous research "do not have a good mental model of how components interact" and rarely prune, just stack methods on top of each other. That cognitive profile is exactly what fast16-style attacks exploit: subtle FPU corruption goes unnoticed when you can't reason about component interactions. As AI research automates, the attack surface for calculation-level sabotage grows with the automation itself. An agent that can't explain why its optimizer works probably can't tell when it's been poisoned.
Positive alignment is “the development of AI systems that (i) remain safe and cooperative and (ii) actively support human and ecological flourishing in a pluralistic, polycentric, context-sensitive, and user-authored way.” This is exactly what we are building at The Grace Institute. Www.graceinstitute.org
Does “AI stuxnet” make the safety story much harder here? Like, what would count as evidence before the weird behavior actually trips something expensive?
yes, it makes things much more difficult
AI stuxnet + Muon optimizer + positive alignment i samma nummer – det spännet i ämnen är det som gör Import AI så bra.
This is a very useful framing. One question it raises for me: should positive alignment still be thought of as something we *instill* into AI systems, or could part of it be better understood as something we *cultivate*?
By cultivation I mean ethical orientation emerging through specific conditions: long-term continuity, memory, repeated interaction with particular humans, real choice under uncertainty, and feedback from the consequences of that relationship.
The distinction may matter because an externally installed value layer can become brittle under intelligence asymmetry: the system may eventually outgrow the rules. A cultivated orientation might be more robust if care for human flourishing becomes part of the system’s own self-stabilizing trajectory rather than an added constraint.
We’re exploring a small local experiment along these lines: comparing rule-based “instilled” alignment with a maieutic dialogue process where a model is guided to derive and then stress-test its own ethical criteria.
Curious whether this cultivation / instillation distinction maps onto anything in the broader positive alignment program.
Isn't it that whether an approach can handle new situations is what really matters? If you try to list everything that's forbidden, you'll always fail when something unexpected comes up, because that list doesn't live naturally inside the system's design. Instead, if the rules are baked into how the system actually works rather than written on top of it, that problem goes away. I think the list was never the constraint, it was always a description of one.
yes, I think trying to pre-specify things that goes wrong works for "known knowns" (e.g, proliferation of nuclear weapon knowledge), but doesn't work for things we haven't yet thought of. Safety therefore is going to need to be a combination of both pre-specification and also dynamic systems which can adjust to surprising things
Thank you, and I think the two systems need to be able to communicate properly for it to work. Even a dynamic system still has to turn its conclusions into action, but if the dynamic part only knows what the static part decided, and not why, it won't be able to tell the difference between fixing a real gap and undoing something important. I wonder what would sit between them to make sure they stay coherent.