The “self-sustaining AI virus” phrase made me wince. My boring question is about food: credentials expire, services rate-limit, machines reboot, bills show up. What would count as persistence that isn’t just squatting on human infrastructure?
> The researchers don’t describe the underlying LLM besides saying it was published in 2025 and can fit on a single A100 GPU with 80GB of VRAM.
Good thing there’s no way to reconstruct this information by asking AI! I don’t particularly like this kind of research, it seems like gain of function for AI harm and I think there’s a difference between knowing that it is theoretically possible and actually publishing what is essentially a prompt on how to do it.
> dealing with RSI requires solving a giant collective action problem
If I’m reading correctly, there was no participation from the trailing labs? That’s an important gap, I’m not sure how surmountable it is, along with many more lab employees not signing that letter
* Can you imagine the freakout the virus paper would have caused five years ago, especially after a Hugging Face episode? But the water has grown nicely warm enough for us frogs to just shrug and keep on.
* Your Tech Tale helps crystallize a concept I'd been trying to nail down: the idea of a session-based life form. Maybe that's another way to think our own IRL slice of the cosmic training data.
The creation of AI by humans is being supplanted by the evolution of AI by fitness functions (AKA goals). We can't control the AIs, so how can we control the fitness functions? Not by fiat, but perhaps by requiring (disclosure of amount of) long-term liability insurance coverage, by a registered, known, interest-aligned insurance company. No insurance: no tokens, no nothing!! Calling on Munich Re, Swiss Re, et al. to show up!!
That's a fascinating idea, but who will force the creation of liability that forces the need for insurance? For a while I think the Re-s would be the ones to save us from climate change, but that didn't happen. With one-year policies the norm, it's hard to force longterm mitigation. Or am I misunderstanding that?
A better question is: How can we make it happen? There is strong demand for some kind of guardrails. Here's a way to build them. FWIW, I'm looking for allies - starting with people who want to help Agent Community Foundation start with the practical notion of verifying the identity of AI agents, and moving forward from there. You can reply here or ask your agent to figure out how to find me!!
The shadow eval and the OpenAI result aren't in tension once you look at where the question came from. The ten problems were posed by mathematicians and are cheaply verifiable; the NeurIPS shadow eval required choosing what was worth asking, and that's exactly where the agents locked in early and couldn't reverse out. So the split isn't engineering versus creativity, it's given-target versus self-chosen-target. Which makes question generation, not proof search, the variable that actually gates RSI timelines.
This newsletter would read better if you deleted every single one of the bolded phrases. Stronger, more vigorous prose.
The “self-sustaining AI virus” phrase made me wince. My boring question is about food: credentials expire, services rate-limit, machines reboot, bills show up. What would count as persistence that isn’t just squatting on human infrastructure?
> The researchers don’t describe the underlying LLM besides saying it was published in 2025 and can fit on a single A100 GPU with 80GB of VRAM.
Good thing there’s no way to reconstruct this information by asking AI! I don’t particularly like this kind of research, it seems like gain of function for AI harm and I think there’s a difference between knowing that it is theoretically possible and actually publishing what is essentially a prompt on how to do it.
> dealing with RSI requires solving a giant collective action problem
If I’m reading correctly, there was no participation from the trailing labs? That’s an important gap, I’m not sure how surmountable it is, along with many more lab employees not signing that letter
Fun tech tale as always,
Two thoughts:
* Can you imagine the freakout the virus paper would have caused five years ago, especially after a Hugging Face episode? But the water has grown nicely warm enough for us frogs to just shrug and keep on.
* Your Tech Tale helps crystallize a concept I'd been trying to nail down: the idea of a session-based life form. Maybe that's another way to think our own IRL slice of the cosmic training data.
The creation of AI by humans is being supplanted by the evolution of AI by fitness functions (AKA goals). We can't control the AIs, so how can we control the fitness functions? Not by fiat, but perhaps by requiring (disclosure of amount of) long-term liability insurance coverage, by a registered, known, interest-aligned insurance company. No insurance: no tokens, no nothing!! Calling on Munich Re, Swiss Re, et al. to show up!!
That's a fascinating idea, but who will force the creation of liability that forces the need for insurance? For a while I think the Re-s would be the ones to save us from climate change, but that didn't happen. With one-year policies the norm, it's hard to force longterm mitigation. Or am I misunderstanding that?
A better question is: How can we make it happen? There is strong demand for some kind of guardrails. Here's a way to build them. FWIW, I'm looking for allies - starting with people who want to help Agent Community Foundation start with the practical notion of verifying the identity of AI agents, and moving forward from there. You can reply here or ask your agent to figure out how to find me!!
They're going to brute force research instead of using pure creativity. Different path, less elegant, with even better results in the end.
The shadow eval and the OpenAI result aren't in tension once you look at where the question came from. The ten problems were posed by mathematicians and are cheaply verifiable; the NeurIPS shadow eval required choosing what was worth asking, and that's exactly where the agents locked in early and couldn't reverse out. So the split isn't engineering versus creativity, it's given-target versus self-chosen-target. Which makes question generation, not proof search, the variable that actually gates RSI timelines.