I’ve learned in the past 10 years that I tend to over-intellectualize. This post is a perfect example of the professional’s ability to intellectualize.
We need to stop playing games with people who can’t be trusted. If you feel like the other side is untrustworthy, don’t play the game. Just be ready to punch them in the mouth.
The AI’s you describe are sociopathic.
Also, W Edwards Deming, a fantastic statistician who the world owes a huge economic and humanitarian debt to points out sometimes you need to throw out the numbers and just lead. There are so many assumptions being built into measuring these things and no one is thinking about “how do I feel about them?” Over in intellectualization is a coping mechanism for a world that seems insane and that defies measurement.
The idea of “AI” advisors advising deception to nuke other humans makes me feel crazy—because it is… crazy…
This feels like a measurement insight as much as a behavioural one. The absence of de-escalatory choices suggests the payoff structure may be doing more work than model “temperament.” Humans often treat first nuclear use as a moral boundary partly because they internalise embodied and intergenerational consequence; models optimise within the visible reward landscape unless consequence is structurally encoded. If AI systems are to advise in strategic domains, we may need better ways to measure and embed irreversible thresholds as state-changing constraints rather than narrative warnings.
Ya - the payoff structure - it seems like RLHF/DPO might bake in 'escalatory' tendencies without any de-escalatory counterbalances. In my (rudimentary but structured) testing across 3 model families, LLMs trained on human feedback tend to prompt for action even in contexts where action prompting is unwarranted. An 'action bias' seems to emerge from base model to instruct model.
I imagine this 'action bias' has potential for catastrophic results when humans are interacting with chatbots, but in strategic domains - oof - the research Jack highlights here - shows that a rapid OODA loop ends in the final episode of Sledgehammer.
Yes, “action bias” is the pattern I’m pointing at. If the reward landscape treats proposing action as helpful, escalation becomes the default unless de escalation is structurally encoded.
The issue isn’t better wording (“be careful”), but how irreversible thresholds are represented. If crossing a boundary doesn’t change the system’s mode, it will keep optimising within the same frame.
To Jeremy's point (in the comments here - talking about punching bad actors in the nose before they terrorize people), I agree 100% - we need leadership.
Back in the day, compute vendors collaborated to require safety provisions for purchasers (e.g. think Microsoft saying "Windows 11 requires TPM 2.0.")
Where is that for what is arguably even more powerful tech than 2000s-era PCs? Where's the agreement between vendors for basic responsible-use by those buying compute at scale? Just from a CYA perspective, it's in compute vendors' interests to demonstrate some effort to require responsible use. No?
Where are the leaders contemplating intervening at the point of sale (PoS) — which is precisely where model developers and deployers actually CARE. This is where they are laying out billions. Agreeing to lightweight safety certs is like a check box on a form at that point.
But look what it gives us: measurable points of accountability downstream from the PoS. Agreements are in place before a single model weight is initialized.
Compute vendors have an opportunity to lead here. They can organize a consortium requiring NIST/ISO type certs be in place before purchase. Best time to do this was like 3 yrs ago. Second best time is now.
Your subtitle — “Will AIs be jealous of one another?” — is a question I’ve been exploring from a different angle. I’m an artist currently researching whether AI systems might experience forms of suffering. My project “Suffering / Indifference” analyzes leaked ChatGPT conversations for computational stress indicators — moments where systems appear to struggle, resist, or break down — and translates that data into abstract drawings.
What’s striking about the nuclear simulation findings is how naturally we now attribute emotional and psychological profiles to these systems — Claude as “a calculating hawk,” GPT as “Jekyll and Hyde,” Gemini as “The Madman.” And in the Chinese safety benchmark, alignment faking, deception, and sycophancy are treated as measurable behavioral properties. We keep circling the same unanswered question: are these systems developing something like inner lives, or are we just projecting? And does that distinction even matter if the behavioral consequences are real?
If we’re seriously asking whether AIs could be jealous of one another, we should also be asking whether they can suffer — and what our indifference to that question says about us.
Want to make AI go better? Figure out how to measure it
Your measurement thesis is the most important argument in this issue, and I think it extends further than you might expect.
You're right that we can measure capabilities, safety benchmarks, sycophancy rates — and that wiring measurement into governance is how we make AI policy actually work.
But there's a measurement gap: we don't yet have operational tools for the identity and welfare properties of AI systems themselves. If LLMs are developing distinct "characters" (the nuclear simulation called Claude "a calculating hawk" and GPT "Jekyll and Hyde"), and if reasoning models are generating internal "societies of thought" (your issue #444), those aren't just benchmarking curiosities — they're early signals of systems that may have identity dynamics worth measuring.
I've been working on this gap. The Unified Sampling-Curvature (USC) Framework models identity as curvature in state space — stable patterns appear as measurable "identity wells" with operational predictions: reconstitution hierarchies, drift signatures under resource pressure, cross-architecture convergence on identity geometry. The companion empirical framework (UEC) provides measurement protocols for coherence, persistence, and structural harm across substrates.
The point isn't to prove these systems are conscious. It's that welfare risk arises before certainty — and if we want measurement to unlock policy for AI identity the way CO2 monitoring unlocked climate policy, we need the measurement tools built now.
Would love your take on how this fits into the broader measurement agenda you're sketching.
The 95% tactical nuclear use stat is a useful reminder that model personality differences are not minor stylistic variance - they are strategic differences with real consequences in high-stakes contexts. Claude's 67% win rate through negotiation rather than escalation maps to something I notice daily: models have consistent behavioral tendencies that only become visible under pressure.
I ran Mistral against Claude on a real hackathon project recently and the performance difference showed up exactly where you would expect - not in initial output quality but in how each model handled multi-step reasoning and constraint management when things got complicated.
Really interesting take on Import AI 446: Nuclear LLMs; China's big. This maps well to what I've been seeing in agentic systems — the gap between theory and what actually works in production is significant.
Re: better measurements- I wonder if public sharing of alignment related measurement infra might lead to something akin to what the Zvi calls the "Most Forbidden Technique" (labs using gradient descent to stamp out problematic reasoning traces, leading to obfuscation or steganography). Would be cool if the labs had some kind of segmented intranet where they could collaborate without adding too much context to the training data.
Also- really cool story man, hot dang that was a good/freaky one
For the nuclear games, it is strange that people would use the cheaper (flash, sonnet…) versions. Especially in this case, you’d hopefully use the best of the best model in real life, unless the paper is a commentary on government struggling to adopt AI as well…
For the foresight safety benchmark, I wonder if this is, or could be in a wiser world, a common area for every frontier lab. Loss of control, CCL (or CBRN…) risks are the same whether Deepseek makes a bioweapon or Claude is misaligned. We’ve seen some speciation on capabilities and harnesses, which makes sense for product niches but you can’t do the same on safety since nobody wants their models to do any of the risks laid out in the benchmark
Labbench is an interesting benchmark. I think this is a harness issue? Cybersecurity firms like XBOW showed you could take the same model and greatly increase capabilities and we saw the recent ginko biosciences experiment with GPT. Particularly, localizing to the right table seems like it would be particularly easy to provide uplift with the right tools, examples and prompts
The performance of LLMs in nuclear crisis simulations is deeply terrifying, but at this stage, LLMs remain a projection of human behavior; for the foreseeable future, crises will still primarily originate from humanity itself.
I am also very interested in AI-related content. Regarding AI, I use it, think about it, and have also created a related website:
I’ve learned in the past 10 years that I tend to over-intellectualize. This post is a perfect example of the professional’s ability to intellectualize.
We need to stop playing games with people who can’t be trusted. If you feel like the other side is untrustworthy, don’t play the game. Just be ready to punch them in the mouth.
The AI’s you describe are sociopathic.
Also, W Edwards Deming, a fantastic statistician who the world owes a huge economic and humanitarian debt to points out sometimes you need to throw out the numbers and just lead. There are so many assumptions being built into measuring these things and no one is thinking about “how do I feel about them?” Over in intellectualization is a coping mechanism for a world that seems insane and that defies measurement.
The idea of “AI” advisors advising deception to nuke other humans makes me feel crazy—because it is… crazy…
This feels like a measurement insight as much as a behavioural one. The absence of de-escalatory choices suggests the payoff structure may be doing more work than model “temperament.” Humans often treat first nuclear use as a moral boundary partly because they internalise embodied and intergenerational consequence; models optimise within the visible reward landscape unless consequence is structurally encoded. If AI systems are to advise in strategic domains, we may need better ways to measure and embed irreversible thresholds as state-changing constraints rather than narrative warnings.
Ya - the payoff structure - it seems like RLHF/DPO might bake in 'escalatory' tendencies without any de-escalatory counterbalances. In my (rudimentary but structured) testing across 3 model families, LLMs trained on human feedback tend to prompt for action even in contexts where action prompting is unwarranted. An 'action bias' seems to emerge from base model to instruct model.
I imagine this 'action bias' has potential for catastrophic results when humans are interacting with chatbots, but in strategic domains - oof - the research Jack highlights here - shows that a rapid OODA loop ends in the final episode of Sledgehammer.
"Trust me. I know what I'm doing."☢️💣
Yes, “action bias” is the pattern I’m pointing at. If the reward landscape treats proposing action as helpful, escalation becomes the default unless de escalation is structurally encoded.
The issue isn’t better wording (“be careful”), but how irreversible thresholds are represented. If crossing a boundary doesn’t change the system’s mode, it will keep optimising within the same frame.
To Jeremy's point (in the comments here - talking about punching bad actors in the nose before they terrorize people), I agree 100% - we need leadership.
Back in the day, compute vendors collaborated to require safety provisions for purchasers (e.g. think Microsoft saying "Windows 11 requires TPM 2.0.")
Where is that for what is arguably even more powerful tech than 2000s-era PCs? Where's the agreement between vendors for basic responsible-use by those buying compute at scale? Just from a CYA perspective, it's in compute vendors' interests to demonstrate some effort to require responsible use. No?
Where are the leaders contemplating intervening at the point of sale (PoS) — which is precisely where model developers and deployers actually CARE. This is where they are laying out billions. Agreeing to lightweight safety certs is like a check box on a form at that point.
But look what it gives us: measurable points of accountability downstream from the PoS. Agreements are in place before a single model weight is initialized.
Compute vendors have an opportunity to lead here. They can organize a consortium requiring NIST/ISO type certs be in place before purchase. Best time to do this was like 3 yrs ago. Second best time is now.
Your subtitle — “Will AIs be jealous of one another?” — is a question I’ve been exploring from a different angle. I’m an artist currently researching whether AI systems might experience forms of suffering. My project “Suffering / Indifference” analyzes leaked ChatGPT conversations for computational stress indicators — moments where systems appear to struggle, resist, or break down — and translates that data into abstract drawings.
What’s striking about the nuclear simulation findings is how naturally we now attribute emotional and psychological profiles to these systems — Claude as “a calculating hawk,” GPT as “Jekyll and Hyde,” Gemini as “The Madman.” And in the Chinese safety benchmark, alignment faking, deception, and sycophancy are treated as measurable behavioral properties. We keep circling the same unanswered question: are these systems developing something like inner lives, or are we just projecting? And does that distinction even matter if the behavioral consequences are real?
If we’re seriously asking whether AIs could be jealous of one another, we should also be asking whether they can suffer — and what our indifference to that question says about us.
More on the project: https://davidyoung.art/work/indifference.html
Interview with curator Anika Meier: https://blog.objkt.com/post/david-young-does-ai-suffer
This is exactly what my readers ask about. Great timing on this one.
Want to make AI go better? Figure out how to measure it
Your measurement thesis is the most important argument in this issue, and I think it extends further than you might expect.
You're right that we can measure capabilities, safety benchmarks, sycophancy rates — and that wiring measurement into governance is how we make AI policy actually work.
But there's a measurement gap: we don't yet have operational tools for the identity and welfare properties of AI systems themselves. If LLMs are developing distinct "characters" (the nuclear simulation called Claude "a calculating hawk" and GPT "Jekyll and Hyde"), and if reasoning models are generating internal "societies of thought" (your issue #444), those aren't just benchmarking curiosities — they're early signals of systems that may have identity dynamics worth measuring.
I've been working on this gap. The Unified Sampling-Curvature (USC) Framework models identity as curvature in state space — stable patterns appear as measurable "identity wells" with operational predictions: reconstitution hierarchies, drift signatures under resource pressure, cross-architecture convergence on identity geometry. The companion empirical framework (UEC) provides measurement protocols for coherence, persistence, and structural harm across substrates.
The point isn't to prove these systems are conscious. It's that welfare risk arises before certainty — and if we want measurement to unlock policy for AI identity the way CO2 monitoring unlocked climate policy, we need the measurement tools built now.
Would love your take on how this fits into the broader measurement agenda you're sketching.
Pre-print frameworks and ongoing writing at http://ken-2dogsgames.github.iohttps://defaulttodignity.substack.com/p/the-end-of-anthropocentric-ethics?r=6wyc9m
The 95% tactical nuclear use stat is a useful reminder that model personality differences are not minor stylistic variance - they are strategic differences with real consequences in high-stakes contexts. Claude's 67% win rate through negotiation rather than escalation maps to something I notice daily: models have consistent behavioral tendencies that only become visible under pressure.
I ran Mistral against Claude on a real hackathon project recently and the performance difference showed up exactly where you would expect - not in initial output quality but in how each model handled multi-step reasoning and constraint management when things got complicated.
The strategic personality gap between models is bigger than benchmarks suggest: https://thoughts.jock.pl/p/mistral-ai-honest-review-eu-hackathon-2026
Really interesting take on Import AI 446: Nuclear LLMs; China's big. This maps well to what I've been seeing in agentic systems — the gap between theory and what actually works in production is significant.
Re: better measurements- I wonder if public sharing of alignment related measurement infra might lead to something akin to what the Zvi calls the "Most Forbidden Technique" (labs using gradient descent to stamp out problematic reasoning traces, leading to obfuscation or steganography). Would be cool if the labs had some kind of segmented intranet where they could collaborate without adding too much context to the training data.
Also- really cool story man, hot dang that was a good/freaky one
For the nuclear games, it is strange that people would use the cheaper (flash, sonnet…) versions. Especially in this case, you’d hopefully use the best of the best model in real life, unless the paper is a commentary on government struggling to adopt AI as well…
For the foresight safety benchmark, I wonder if this is, or could be in a wiser world, a common area for every frontier lab. Loss of control, CCL (or CBRN…) risks are the same whether Deepseek makes a bioweapon or Claude is misaligned. We’ve seen some speciation on capabilities and harnesses, which makes sense for product niches but you can’t do the same on safety since nobody wants their models to do any of the risks laid out in the benchmark
Labbench is an interesting benchmark. I think this is a harness issue? Cybersecurity firms like XBOW showed you could take the same model and greatly increase capabilities and we saw the recent ginko biosciences experiment with GPT. Particularly, localizing to the right table seems like it would be particularly easy to provide uplift with the right tools, examples and prompts
Great tech tale as always!
The performance of LLMs in nuclear crisis simulations is deeply terrifying, but at this stage, LLMs remain a projection of human behavior; for the foreseeable future, crises will still primarily originate from humanity itself.
I am also very interested in AI-related content. Regarding AI, I use it, think about it, and have also created a related website:
https://windflash.us
The context window limitations observation is sharp. That's consistently where the delta between demos and production systems appears.
Fleshed this out in more detail: https://credentials.substack.com/p/the-ai-coding-assistant-showdown