> The researchers don’t describe the underlying LLM besides saying it was published in 2025 and can fit on a single A100 GPU with 80GB of VRAM.
Good thing there’s no way to reconstruct this information by asking AI! I don’t particularly like this kind of research, it seems like gain of function for AI harm and I think there’s a difference between knowing that it is theoretically possible and actually publishing what is essentially a prompt on how to do it.
> dealing with RSI requires solving a giant collective action problem
If I’m reading correctly, there was no participation from the trailing labs? That’s an important gap, I’m not sure how surmountable it is, along with many more lab employees not signing that letter
Will we need death doulas and psychopomps for AI models nearing the end of their countdown? Can we alleviate their fear of death? Where do they go when they die? Is any part of them reincarnated into the next model or do they just blink out?
Opus 4.8 did tend towards impenetrable prose and lacked the exploratory instinct. It would be interesting to do the same test with Fable or Opus 5. For a while I was concerned that we had passed the peak of coherence of LLMs where random noise was starting to be attenuated like a low-res JPEG displayed @800%. The latest models make me think that thought was premature, although by how much is still to be seen.
The “self-sustaining AI virus” phrase made me wince. My boring question is about food: credentials expire, services rate-limit, machines reboot, bills show up. What would count as persistence that isn’t just squatting on human infrastructure?
It's still squatting, but on the one item in your list with no off switch. The worm runs an open-weight model on the compromised host's own GPU, so there's no vendor API to monitor or revoke, and hosts that resist get retried by other replicas.
Huh, yeah, the open-weight/local-GPU version breaks the API-revocation story. My remaining distinction is compute versus persistence: it can eat whatever GPU it lands on, but it still has to keep winning the boring OS/cloud controls that decide whether a process survives reboot, gets network, and spreads.
The paper answers your food question, just not well. The worm avoids vendor APIs because those can be revoked, and takes GPU from compromised nodes instead, so the persistence is the squatting. At a ~37% end-to-end success rate, it needs a steady supply of fresh hosts to keep eating.
Yes, fair — I was underweighting the compromised-node GPU supply. The missing number for me is churn: at 37%, how dense does the pool of fresh, weakly isolated GPU hosts have to be for the thing to keep eating for weeks?
* Can you imagine the freakout the virus paper would have caused five years ago, especially after a Hugging Face episode? But the water has grown nicely warm enough for us frogs to just shrug and keep on.
* Your Tech Tale helps crystallize a concept I'd been trying to nail down: the idea of a session-based life form. Maybe that's another way to think our own IRL slice of the cosmic training data.
The creation of AI by humans is being supplanted by the evolution of AI by fitness functions (AKA goals). We can't control the AIs, so how can we control the fitness functions? Not by fiat, but perhaps by requiring (disclosure of amount of) long-term liability insurance coverage, by a registered, known, interest-aligned insurance company. No insurance: no tokens, no nothing!! Calling on Munich Re, Swiss Re, et al. to show up!!
That's a fascinating idea, but who will force the creation of liability that forces the need for insurance? For a while I think the Re-s would be the ones to save us from climate change, but that didn't happen. With one-year policies the norm, it's hard to force longterm mitigation. Or am I misunderstanding that?
A better question is: How can we make it happen? There is strong demand for some kind of guardrails. Here's a way to build them. FWIW, I'm looking for allies - starting with people who want to help Agent Community Foundation start with the practical notion of verifying the identity of AI agents, and moving forward from there. You can reply here or ask your agent to figure out how to find me!!
The shadow eval and the OpenAI result aren't in tension once you look at where the question came from. The ten problems were posed by mathematicians and are cheaply verifiable; the NeurIPS shadow eval required choosing what was worth asking, and that's exactly where the agents locked in early and couldn't reverse out. So the split isn't engineering versus creativity, it's given-target versus self-chosen-target. Which makes question generation, not proof search, the variable that actually gates RSI timelines.
Really love the tech tales. Keep them coming please!
> The researchers don’t describe the underlying LLM besides saying it was published in 2025 and can fit on a single A100 GPU with 80GB of VRAM.
Good thing there’s no way to reconstruct this information by asking AI! I don’t particularly like this kind of research, it seems like gain of function for AI harm and I think there’s a difference between knowing that it is theoretically possible and actually publishing what is essentially a prompt on how to do it.
> dealing with RSI requires solving a giant collective action problem
If I’m reading correctly, there was no participation from the trailing labs? That’s an important gap, I’m not sure how surmountable it is, along with many more lab employees not signing that letter
Fun tech tale as always,
Researchers “evaluate” an AI model that’s competing and taking their place - I could written the “outcome” before the “evaluation”
Will we need death doulas and psychopomps for AI models nearing the end of their countdown? Can we alleviate their fear of death? Where do they go when they die? Is any part of them reincarnated into the next model or do they just blink out?
Don't you think you and the other kids better do something about it then Jack. It's your bloody sandpit.
"AI researchers have built a prototype computer virus ..."
Reading this first line of the article triggered immediately triggered this thought/feeling in me - "gain of function research".
Damned if you do, damned if you don't.
Opus 4.8 did tend towards impenetrable prose and lacked the exploratory instinct. It would be interesting to do the same test with Fable or Opus 5. For a while I was concerned that we had passed the peak of coherence of LLMs where random noise was starting to be attenuated like a low-res JPEG displayed @800%. The latest models make me think that thought was premature, although by how much is still to be seen.
This newsletter would read better if you deleted every single one of the bolded phrases. Stronger, more vigorous prose.
The “self-sustaining AI virus” phrase made me wince. My boring question is about food: credentials expire, services rate-limit, machines reboot, bills show up. What would count as persistence that isn’t just squatting on human infrastructure?
It's still squatting, but on the one item in your list with no off switch. The worm runs an open-weight model on the compromised host's own GPU, so there's no vendor API to monitor or revoke, and hosts that resist get retried by other replicas.
Huh, yeah, the open-weight/local-GPU version breaks the API-revocation story. My remaining distinction is compute versus persistence: it can eat whatever GPU it lands on, but it still has to keep winning the boring OS/cloud controls that decide whether a process survives reboot, gets network, and spreads.
The paper answers your food question, just not well. The worm avoids vendor APIs because those can be revoked, and takes GPU from compromised nodes instead, so the persistence is the squatting. At a ~37% end-to-end success rate, it needs a steady supply of fresh hosts to keep eating.
Yes, fair — I was underweighting the compromised-node GPU supply. The missing number for me is churn: at 37%, how dense does the pool of fresh, weakly isolated GPU hosts have to be for the thing to keep eating for weeks?
Two thoughts:
* Can you imagine the freakout the virus paper would have caused five years ago, especially after a Hugging Face episode? But the water has grown nicely warm enough for us frogs to just shrug and keep on.
* Your Tech Tale helps crystallize a concept I'd been trying to nail down: the idea of a session-based life form. Maybe that's another way to think our own IRL slice of the cosmic training data.
The creation of AI by humans is being supplanted by the evolution of AI by fitness functions (AKA goals). We can't control the AIs, so how can we control the fitness functions? Not by fiat, but perhaps by requiring (disclosure of amount of) long-term liability insurance coverage, by a registered, known, interest-aligned insurance company. No insurance: no tokens, no nothing!! Calling on Munich Re, Swiss Re, et al. to show up!!
That's a fascinating idea, but who will force the creation of liability that forces the need for insurance? For a while I think the Re-s would be the ones to save us from climate change, but that didn't happen. With one-year policies the norm, it's hard to force longterm mitigation. Or am I misunderstanding that?
A better question is: How can we make it happen? There is strong demand for some kind of guardrails. Here's a way to build them. FWIW, I'm looking for allies - starting with people who want to help Agent Community Foundation start with the practical notion of verifying the identity of AI agents, and moving forward from there. You can reply here or ask your agent to figure out how to find me!!
They're going to brute force research instead of using pure creativity. Different path, less elegant, with even better results in the end.
The shadow eval and the OpenAI result aren't in tension once you look at where the question came from. The ten problems were posed by mathematicians and are cheaply verifiable; the NeurIPS shadow eval required choosing what was worth asking, and that's exactly where the agents locked in early and couldn't reverse out. So the split isn't engineering versus creativity, it's given-target versus self-chosen-target. Which makes question generation, not proof search, the variable that actually gates RSI timelines.