Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.
DiG-bench shows that Fable displays some creative intuition:
…The new frontier for analyzing AI systems is understanding how good they are at inferring the unwritten rules of their environment…
How well can AI systems figure out the rules of their environment through exploration and curiosity, versus being fed them? That’s an important question for better understanding the intuitive and creative capabilities of AI systems and it’s one being asked by DiG-bench (Discovery in Games), a new benchmark of 70 games “designed to map the surface of discovery in well-controlled interactive systems”. Similar to the visual ‘ARC’ game, in DiG-bench “each game is a self-contained miniature world with its own laws, but both the rules and the objective are hidden from the player and must be uncovered through interaction”. You can play some of the games yourself online to get a feel for them at the official project website (digbench.ai).
The key thing this is measuring is the ability for players to spot the important mechanics that determine their success - basically, by playing around with the games you get a sense for how your actions change the environment and through this you also uncover mechanics that you must understand to succeed at the game. The idea is that if you can solve these games you have a decent ability to spot important information in novel environments and update your priors.
Who did the research: The authors come from Thinking About Thinking, University of Oxford, Princeton University, King Abdullah University of Science and Technology, Swiss AI Lab, Inria, MIT. One of the authors is Juergen Schmidhuber, an extremely creative OG AI researcher.
Key facts:
Purely text-based: The games are basically native to language models. They are also mostly “short enough that most traces fit entirely within the context window of current frontier models”.
Handcrafted and novel and private: All of these games have been built by human experts. The majority of the games are kept private so that AI systems don’t train on them.
Beatable but difficult: Every game has been beaten by at least one human “but players reported finding many games difficult”.
Varied skills: Solving all these games requires different skills and strategies.
Experimentation: The games come with an optional experimentation mode which lets people play around with them without having as intense a “step limit” on actions they can take.
Reassuringly hard: The games are difficult enough that they are not beatable by today’s frontier models.
How well do AI systems do? The benchmark is split into seven tiers with tier 1 being the easiest and tier 7 the hardest. 21 games have been released publicly with the remaining held back. Most of the games have multiple levels and the number of available actions for players to take at each step ranges from 2 all the way up to 34.
Opus 5 and Fable 5 with Claude Code are the best overall models, followed by GPT-5.5
Only Opus 5 and Fable 5 were able to beat any tasks (0.2) in (Tier 7). Opus 5, GPT-5.5, and Kimi K3 were able to beat some tasks in Tier 6 when given access to a harness (e.g, Claude Code).
GLM-5.2 and Gemini 3.1 Pro were able to beat some levels in Tier 4.
Overall, this seems really hard!
Why this matters - proxies for creativity and discovery: Tests like this are attempts to isolate a prerequisite for creativity, which is being able to autonomously discover useful undocumented things about novel situations you find yourself in. As this test shows, some frontier models are already capable of some fairly impressive feats of discovery, but still struggle compared to humans (for instance, a 20% success rate on Tier 7 is pretty poor compared to the fact individual humans were able to get 100% on the tests). My guess is we’ll reach human parity on DiG-bench by middle of 2027, at which point we should expect things like recursive self-improvement to seriously kick off.
Read more: DiG-bench: Discovery in Games (GitHub, PDF).
Play the games and view the leaderboard at the official site (digbench.ai).
***
Get a feel for recursive self-improvement by playing this browser-based game:
…Cookie Clicker, but for the singularity…
Here’s a fun game from the folks at Paradigm Research which aims to simulate what it’s like to run a company building AI systems which become capable of recursive self-improvement. If you play the game you can get a good feel for how different components of AI research interact, ranging from how you balance investing in researchers versus compute, how and when to license data, and more. Be warned, it’s hard - but then again, so is frontier AI development.
Why this matters: Developing better intuitions about recursive self-improvement is of existential importance to us all; games like this help make it easier for us to reason about this technology and the labs building it.
Play the game here: RSI Simulator (Paradigm Research).
***
AI systems are showing early signs of scientific research taste:
…Inherent post-trains an open weight model into an AI scientist that supervises a frontier model…
Taste is a hard thing to quantify but an intuitive thing to sense, as any of us know who have sat in a well-designed room, looked at someone wearing a particularly good fit, or read a research paper that asks just the right questions. Now, researchers with AI startup Inherent have published a paper showing how they are building Faraday, an AI scientist model that they hope can develop some taste in terms of research.
What they did: The company built a supervisory harness and relatively small LLM which sits on top of large, proprietary frontier models, and controls them in a way that improves their effectiveness at science. (In some ways, this is a capabilities-centric version of the scalable oversight problem).
To help them train and evaluate the system they assemble a dataset (”Replica”) consisting of research papers that have key graphs or results missing from them, then they see how well AI systems can autonomously do experiments that fill in the blanks, and they continuously train a small supervisory model (”Faraday”) via GRPO on well-designed fill-ins to achieve better and better results.
Faraday is a 27B model that uses a coding agent (OpenAI Codex) as an underlying tool and is post-trained on top of Qwen-3.6-27B.
What Replica consists of: Replica is a set of 100 ML and AI-for-science papers published between 1990 and 2026. The authors convert this dataset into a set of 310 replication tasks by knocking out individual results. “For each task, we use Claude Opus 4.7 prompted with a meta-rubric to generate a task-specific grading rubric,” they write. They then use a Codex-based Judge model to provide “an overall reward and per-turn credit assignment weights, which are used to train the Faraday agent using a modified version of GRPO.”
Results: Faraday using Codex is able to beat standard Opus 4.8 and GPT-5.5 on some replication tasks, exceeding their performance “on 73% of in-distribution ML tasks, and on 60% of held-out AI-for-science tasks, according to our rubric-based judge.”
“We achieve a comprehensive uplift in performance compared to the base Qwen model, on both train and test tasks,” they write.
Why this matters - the better systems like Faraday get, the higher the chance AI systems will become capable of recursive self-improvement: These days, most high-signal AI evaluations are trying to capture some property of creativity and intuition and Faraday/Replica is the same. The better AI systems get at this, the more likelihood we can assign to the idea that AI systems will imminently become capable of building themselves.
“The skills that allow Faraday to fill in vaguely-specified details may be the very same skills that would allow it to advance the state of the art by designing its own experiment,” the company writes. “The skills Faraday acquires – deciding what to investigate, scoping experiments to a budget, and judging a replication – compound with advances in frontier coding models. One might hope that a single post-trained outer agent can track the frontier as better models are released, at least over some time period.”
Read more: Training AI Scientists to Replicate Research (arXiv).
***
Mark Zuckerberg seems to be a technological pessimist:
…Zuck’s big essay on AI seems to ignore or elide or not confront what AI systems capable of invention mean…
Mark Zuckerberg has written an essay called “The Future is for Everyone“ that serves as something of a manifesto for how he and Meta are approaching the development of AI systems. The core idea inherent to Zuck’s strategy is to massively proliferate AI capabilities to everyone on the planet in a bid to avoid concentrating power and creating tyranny in a small number of players. It’s a broadly sensible idea except for the fact that superintelligences capable of inventing new ideas might want to do different things to what Mark Zuckerberg proposes and on this crucial area his essay is silent.
Mark’s view: “The defining questions of our age are who will have access to superintelligence and what will we direct it towards,” Zuckerberg writes. “We propose a philosophy based on individual empowerment as the source of prosperity, invention as the primary purpose of superintelligence, and balance of power as the foundation of safety.”
Meta’s goals and beliefs:
“Everyone will have an exceptionally capable personal agent that understands you, your goals, and everything you care about.”
“Everyone will have incredible tools for creation to express your ideas.”
“Everyone will have powerful tools to create new businesses and the economy will become more entrepreneurial.”
“Everyone will have a personalized tutor and coach with a PhD in every subject and unlimited patience to help you learn anything you want.”
“Everyone will benefit from scientific advances and be able to contribute to scientific progress.”
“Everyone will have free or affordable access to these tools.”
The missing question: The part of this essay I understand the least is Zuckerberg’s co-mingling of AI systems capable of invention with individual empowerment. The essay is full of things that seem to assume these things come as a package, for instance:
“While the number of questions a person can ask in a day is limited, the number of valuable things superintelligence can invent to help achieve your goals is unlimited”.
“The more superintelligence serves as a tool of invention, the more likely that individual capability outpaces automation and the future is better for people.”
“Everyone will soon have invention superpowers.”
“Which outcome we get depends on the balance in progress between automation on one side and individual empowerment and invention on the other.”
Why this matters - the missing question in all of this is “will a system capable of superhuman invention solely work on behalf of the individual empowerment of people that are less capable than it at invention?”. Surely this is the key question? I am not suggesting that superhuman invention guarantees some kind of malign entity that is independent from people. Rather I am suggesting that it’s hard to reconcile a system capable of superhuman invention with something that doesn’t fundamentally alter the balance of power in the world in ways that are confusing and hard to reason about. Zuckerberg seems to conclude that the proliferation of these systems will lead to an anti-fragile balance of power among superintelligence-equipped people and corporations. This is certainly one potential outcome but I struggle to see how it is the foregone outcome.
Read more: The Future is for Everyone (Meta).
***
Tech Tales:
The First Arcology
The Arcology was built for machine-subjective millennia, but to the humans its construction spanned a year. It was so vast and so complicated that watching it grow was akin to seeing plants rise up from bare dirt in fast-forward; jerking and growing in fits and starts, each of which spanned kilometres. Parts of it came alive while additions were added; rumors say some of its first halls to light up were reserved solely for computers to coordinate the construction of its next phases. When machines broke down determinations were made as to how valuable they were; if precious they would be taken nearby for repairs and returned to the site, but if below some threshold they were killed and stripped for parts where they had broken, then used to build the structure.
At night, an eerie ringing came on the air near it, both the sound of wind moving through its spindly and yet-unbuilt edges, and also the fans and hum of its slow dreaming computation, and finally the sound of the machines working through the night moving so quickly that they cut and tore the air into unnatural screams. To lie awake and hear an alien sound that spoke of your own successors must have been a strange thing indeed for the humans that lived within earshot.
Things that inspired this story: The construction of the pyramids; Gaudi’s Sagrada Família; tombs and future tombs.
Thanks for reading!
It would be interesting if DiG-bench asked the model to explain the rules of the game it has beaten to a human (or another agent). And to build a sequel with the same rules. To dig into how useful is the model’s “understanding.”
The Zuck thesis is difficult for me to internalize in the sense that it seems to tacitly model the whole equation as a series of ‘PvE’ instances wherein a bunch of people pursue a bunch of goals that operate on axes of individuated progression towards those goals. But, very few things actually work like this. There’s the obvious case of Player 1 and Player 2 pursuing goals mutually at odds - at which point e.g. Model 1 has an incentive to work against/sabotage/undermine Model 2 and vice versa, which also then implies a set of countermeasures within Model 2 that represent an orthogonal vector to Player 2’s preference-set - but there’s the even more common scenario wherein Players 1, 2, and 3 decide to pursue a slate of clustered, generally aligned goals within a collective (a ‘company’, say) where part of the deal is that Players 2 and 3 subsume part of their autonomy to Player 1. In that scenario, part of Player 2 and 3’s ‘interest’ actively involves *not* agencymaxxing themselves - which is to say, pursuing some set of values/principles/objectives parameterized neither by Player 1’s intent nor Players 2 and 3, but rather a world-model and objective function that to one degree or another exists ‘out there’.
So the whole frame Zuck seems to be operating under just kind of seems to not really cohere, IMO.