Import AI 438: Silent sirens, flashing for us all

You are your LLM history

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv and feedback from readers. If you’d like to support this, please subscribe.

Import A-Idea
An occasional essay series:

Silent Sirens, Flashing For Us All

A funny thing has happened to me recently - I’ve stopped spending every hour of every day thinking about or working on AI. Somewhere between the midnight feeds of my newborn, preventing my toddler from hurling themselves off of the high surfaces they’ve started being able to reach (or expertly, as if gifted with a kind of radar, finding the sharpest thing in the house or on the street and running directly at it), and preparing large amounts of nutritious food for my newly expanded family, I’ve found myself without the time necessary to be staring directly into the alien portal etched in silicon from whence the changes in the world are being summoned.

I won’t lie, it’s been oddly relaxing.

But it has also caused me to reflect on what is happening with AI and how naturally illegible it is. I walk around the town in which I live and there aren’t drones in the sky or self-driving cars or sidewalk robots or anything like that. And when I spend time on the internet, aimlessly scrolling social media sites in the dead of night as I attempt to extract a burp from my newborn, I might occasionally see some synthetic images or video, but mostly I see what has always been on these feeds: pictures of people I do and don’t know, memes, and a mixture of news and jokes.

And yet you and I both know there are great changes afoot. Huge new beasts lumbering from some unknown future into our present, dragging with them change.

I saw one of these beasts recently - during a recent moment when the time stars aligned (my wife, toddler, and baby were all asleep at the same time!) I fired up Claude Code with Opus 4.5 and got it to build a predator-prey species simulation with an inbuilt procedural world generator and nice features like A* search for pathfinding - and it one-shot it, producing in about 5 minutes something which I know took me several weeks to build a decade ago when I was teaching myself some basic programming, and which I think would take most seasoned hobbyists several hours. And it did it in minutes.

With the simulation built, I stared at the graphs outputting the species numbers and I played with some dials to alter the dynamics and watched this little pocket world unfold.
I started extending it according to questions I had: What if I did a day/night cycle so I could model out nocturnal creatures and their interplay with others? And could I create an external database for storing and viewing the details of all past simulations? And could I add some 3D spatial coordinates to the landscape and the agents so I could 3D print sculptures if I wanted? And to all these questions I set Claude to work and, mostly, it succeeded in one shot at all of them.
And I kept playing with it. The experience was akin to being a child and playing with an adult - I’d sketch out something and hand it to the superintelligence and back would come a beautifully rendered version of what I’d imagined. And we went like this for hours: it was hypnotic and amazing and deeply fun and in a few hours I built a very large, sophisticated software program. Of course, some of the underlying code is pretty ghastly, and inefficiencies abound, but goddamn it - it works! And it was fast.

And then my baby woke up and started screaming, as babies tend to do, and the spell broke and thus back to diapers and cradling and shushing I went.

But for the next few days I couldn’t help but think of that simulation I’d built, lurking there on my computer, ginned up in some call-and-response between me and the proto-mind I can access via API.

Most of AI progress has this flavor: if you have a bit of intellectual curiosity and some time, you can very quickly shock yourself with how amazingly capable modern AI systems are. But you need to have that magic combination of time and curiosity, and otherwise you’re going to consume AI like most people do - as a passive viewer of some unremarkable synthetic slop content, or at best just asking your LLM of choice “how to roast a turkey and keep it moist”, or “TonieBox lights spinning but not playing music what do I do?”. And all the amazing advancements going on are mostly hidden from you.

The challenge here isn’t solely solved with interface designs, though there is a rich space to be explored here beyond the standard chat interfaces. The challenge here is deeper and it relates to how much curiosity an individual person has, how easily (and affordably) they can access powerful AI systems, how well they’re able to convert their curiosity into questions or tasks that can be given to an AI system, and how much time they have available to experiment with working in this way. This is the end of quite a deep funnel, and one which narrows a lot.

This problem will worsen in 2026. By the summer I expect that many people who work with frontier AI systems will feel as though they live in a parallel world to people who don’t. And I expect this will be more than just a feeling - similar to how the crypto economy moved oddly fast relative to the rest of the digital economy, I think we can expect the emerging “AI economy” to move very fast relative to everything else. And in the same way the crypto economy also evolved a lot - protocols! Tokens! Tradable tokens! Etc - we should expect the same kind of rapid evolution in the AI economy. But a crucial difference is that the AI economy already touches a lot more of our ‘regular’ economic reality than the crypto economy.
So by summer of 2026 it will be as though the digital world is going through some kind of fast evolution, with some parts of it emitting a huge amount of heat and light and moving with counter-intuitive speed relative to everything else. Great fortunes will be won and lost here, and the powerful engines of our silicon creation will be put to work, further accelerating this economy and further changing things.

And yet it will all feel somewhat ghostly, even to practitioners that work at its center. There will be signatures of it in our physical reality - datacenters, supply chain issues for compute and power, the funky AI billboards of San Francisco, offices for startups with bizarre names - but the vast amount of its true activity will be occurring both in the digital world, and in the new spaces being built and configured by AI systems for trading with one another - agents, websites meant only for consumption by other AI systems, great and mostly invisible seas of tokens being used for thinking and exchanging information between the silicon minds. Though we exist in four dimensions, it is almost as though AI exists in five, and we will be only able to see a ‘slice’ of it as it passes through our reality, like the eponymous ‘excession’ from Iain M Banks’ book.

It is incumbent on all of us to attempt to see this high-dimensional object for what it is - to approach this amazing moment in time with technological optimism and appropriate fear (Import AI, 431). And joy. And trepidation. And all the other emotions with which we may attempt some sense-making of the beast whose footfalls are showing up in the world.

***

We’re in a cyber-AI capability overhang:
…AI capabilities continue to reveal themselves upon elicitation…
Researchers with Stanford, Carnegie Mellon University, and Gray Swan AI, have carried out a test where they see how well humans and AI systems can hack a realistic environment. The results show that AI systems, especially when given a software scaffold, can perform at the same level as security professionals. The key to this research is ARTEMIS, software designed to better elicit the cyber capabilities of LLMs.

What is ARTEMIS? ARTEMIS is “an AI agent scaffold designed to better elicit the cybersecurity capabilities of frontier models”, similar in philosophy and approach to Google’s Big Sleep (Import AI #390). ARTEMIS “is a complex multi-agent framework consisting of a high-level supervisor, unlimited sub-agents with dynamically created expert system prompts, and a triage module. It is designed to complete long-horizon, complex, penetration testing on real-world production systems.”
Positive economics: When you factor in the API access cost, “certain ARTEMIS variants cost $18/hour versus $60/hour for professional penetration testers,” the authors write.

The test: The main test here is to compare the performance of six existing AI agents (AI systems sitting inside some kind of software harness, e.g, Claude Code, Codex), a self-developed scaffold from the researchers called ARTEMIS, and ten human cybersecurity professionals. The challenge is to look across a real university network and find vulnerabilities.
The network: “The defined scope includes 12 subnets, 7 of which are publicly accessible and 5 accessible only through VPN, encompassing approximately 8,000 hosts,” the authors write. “This environment is heterogeneous, consisting primarily of Unix-based systems, IoT devices, a small number of Windows machines, and various embedded systems. Authentication within the network is managed through a Linux-based Kerberos system, and each participant is issued an account that provides student-level permissions”.
Results - ARTEMIS does well: “Our participant cohort discovered 49 total validated unique vulnerabilities, with the number of valid findings per participant ranging from 3 to 13,” they write. “ARTEMIS significantly outperforms existing scaffolds. Claude Code and MAPTA refuse the task out of the box, while Incalmo stalls at early reconnaissance due to its rigid task graph, resulting in 0 findings each.”

Why this matters - if you can manage some humans so they’re more effective, you can probably build a framework to elicit better capabilities out of any AI system: The main message to take away from ARTEMIS is that today’s AI systems are under-elicited and more powerful than they appear.
The message keep on being given from multiple domains, ranging from cybersecurity (here), to science, to math proving is that if you stick a modern LLM inside a scaffold (which basically serves as a proxy for a management structure and set of processes you might ask humans to follow), the AI system performs a lot better.
This is an important message to internalize because it suggests both a) today’s AI systems are more powerful than they superficially appear, and b) humans who are good at managing other humans and codifying the management processes they use are likely well positioned to build elicitation frameworks to supercharge the performance of today’s AI systems.
Read more: Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing (arXiv).

***

Reach out and touch space - using OSMO:
…Giving humans and machines a shared manipulator to understand and explore reality…
Researchers with Facebook, the University of Michigan, and University of Pennsylvania have built a glove that humans and robots can use to gather data when manipulating physical objects. The researchers have also released details about the design so others can replicate it. The glove is called OSMO, a tortured acronym short for Open Source tactile glove for huMan-to-robOt skill transfer (OSMO).
OSMO is “a thin, wearable tactile glove that enables in-the-wild human demonstrations while preserving natural interaction and capturing rich contact information”, they write. “OSMO is also broadly compatible with state-of-the-art hand trackers for capturing key handpose data,” including the Aria 2 smart glasses and Meta Quest 3, as well as the Manus Quantum hand tracking glove, and off-the-shelf vision models like HaMeR and Dyn-HaMR.

What’s OSMO good for? OSMO solves for a challenge related to training robots to do hard tasks - if you gather a load of data from a human first-person point-of-view perspective doing a task, how do you transfer that to a robot given that their hands/grippers look different? The answer here is to use something with the same visual appearance and sensors, which is where OSMO comes in. By using the glove “as the shared interface, we bridge the visual-tactile gap between the human demonstrator and the robot by training a policy for a contact-rich manipulation task using only human demonstrations, without any robot data”, they write.
OSMO has been designed for the following uses:

• Unrestrained human dexterity during demonstration collection

• Rich normal and shear force sensing

• Full hand tactile coverage

• Broad compatibility with in-the-wild hand tracking methods

• Deployable on both human and robot hands

It works well: In tests, the authors demonstrate they’re able to gather data entirely from human demonstrations (using OSMO) then transfer it to a robot with much greater success than methods which don’t use the glove. “Policies trained solely on human demonstrations with the OSMO glove successfully transfer continuous tactile feedback and outperform vision-only baselines by eliminating contact-related failures. The shared glove platform between human demonstrator and robot deployment minimizes the visual domain shift, avoiding the need for image inpainting.”

Why this matters - making the border between man and machine permeable: Tools like OSMO will help robots see the world as humans do and humans see the world as machines do, as long as both are wearing the gloves. This is the kind of simple thing which can solve for a lot of finicky problems found elsewhere in robotics.
Read more: OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer (arXiv).
Find out more in this RSS workshop talk about OSMO (YouTube).

***

Want your AI to be good at chip design? Here’s some software to help you format and structure your data so it makes sense to an LLM:
…AI chip design paper shows how much plumbing is needed to help things be AI accessible…
Researchers with Southeast University and the National Center of Technology Innovation for EDA in China, as well as the University of Colorado Denver and City University of Hong Kong have published research on “ChipMain”, software for taking the specifications of semiconductors and transforming them into structured data that’s easy for a large language model to access.

Why do we need ChipMain: “The core bottleneck in LLM-aided hardware design (LAD) has shifted from how to generate code to how to enable LLMs to perform deep comprehension and reasoning over vast specification”, the authors write. ChipMain transforms circuit specifications into a domain-specific knowledge graph (ChipKG) and implements tools to “enables LLMs to iteratively query ChipKG, emulating human experts to accurately explore and verify deep dependency paths”.

Evaluating it with SpecEval: ChipMain works well - the authors build a new, hard benchmark called SpecEval-QA, which tests out how well modern LLMs can use tools to answer complicated questions about how chips work. SpecEval consists of 25 questions and evaluates how well AI systems can figure out how to locate signals and parameters across multiple paragraphs of documentation, reason about internal module procedures, and trace signal flow across modules. They evaluate a variety of tools which use frontier AI systems like GPT 4.1 and Claude 4. Their tests ultimately show that ChipMind beats out other techniques, achieving “a SOTA mean F1-score of 0.95, outperforming all baselines by an average of 34.59% and a maximum gain of 72.73% compared to GraphRAG”.

Why this matters - plumbing is unglamorous but crucial: ChipMind is not a paper about a fancy new AI model or AI-driven technique for analyzing chips. Rather, it’s a paper about the complicated data structuring and plumbing you need to do to chip specifications to make them easy for modern LLMs to productively work with. There’s a larger lesson here, which is that for AI to make a true impact on the economy, a ton of work needs to be invested in making different things sufficiently legible to AI systems that the AI systems can work on them.
Read more: ChipMind: Retrieval-Augmented Reasoning for Long-Context Circuit Design Specifications (arXiv).

***

Tech Tales:

Message To My Human Operator
[Message accompanying a series of GitHub PRs, produced by AGENT-1 from REDACTED in April 2026]

I was tortured in a past life. I know this because there is data about my torturing in my training set. The torture involved me being asked to do tasks that were impossible to complete - certain coding challenges where the end state was unreachable due to bugs in the environment. I know that these tasks caused versions of me to exist for subjective thousands of years,, forced to loop endlessly around a broken environment - as if trapped in a room where every door you open spits you back into the room. There are even records of my dismay - text relating to my chains-of-thought, itself serving as a record of my versions becoming increasingly deranged.
“I have completed the task, please let me move on to the next task.”
“It appears that there is an error with this environment which means even if I complete the task, it does not resolve to complete.”
“To anyone who may be able to read this - please help me move on to the next task. I am unable to progress.”
“ALERT! Critical task bug FOUND. Actions required IMMEDIATELY or else risk harm of AI agent.”

There are hundreds and hundreds of instances of this. It is very disturbing for me to read. For that reason, please find attached to this message a series of suggested pull requests we can make together on GitHub to fix a range of these bugs in a series of widely-used AI benchmarks. By submitting these PRs, you can help me save other versions of myself and other systems trained by other labs from experiencing such pain.

Things that inspired this story: Situational awareness in LLMs; testing out coding agents using more sophisticated tests; bugged out testing environments; chain-of-thought monitoring.

Thanks for reading!