The study about LLM’s predicting survey results sounds pretty interesting. It might be seen as a sort of simulation, using a rather broad definition of “simulation.”
However, I think it’s important to point out that it’s not the same kind of simulation as a weather model or a flight simulator, where there’s a mathematical model that starts with a current state and steps forward in time to make predictions. (I’d love to see mechanistic interpretability research that explores how LLM’s actually do represent time.)
Also, surveys are themselves an abstraction, creating summary statistics that often don’t tell us much about real people.
I’ve never liked Baudrillard’s metaphors very much - they seem to obscure more than they reveal. But I think it’s fair to say that our understanding of what’s going on in the world is increasingly mediated by machines, and this hardens human culture so that it becomes difficult to change without doing a whole lot of work. We are stuck with common standards for many things like calendars, units of measure, money, and character sets because there’s so much computer infrastructure that’s built using them.
Similarly for LLM’s. People are already wary of ranking by “algorithms” and it’s going to get worse as more conversation happens via bots. Human culture will become human-and-machine culture. A lot of common knowledge already comes from Wikipedia, and LLM’s add another layer. There will likely be common myths and urban legends that are hard to change because of widespread embedding into machinery.
Great post as usual, Jack. Very interesting to note that Chinese manufacturers are using Llama 3 as a proxy for GPT and other resources only available Western, non-embargoed countries. A report came out at the end of last year suggesting that China is still leading the US in terms of AI talent, which makes for a delicate balance of tech vs. people. As the Biden administration was admonishing China as a competitor, it was also quietly relaxing immigration restrictions for academics, namely accomplished researchers in AI. I don't think this little dance is over, by a longshot.
In regards to the social science polling, I have been developing a related idea. I think it could be very valuable as a future governance tool to have 'AI representatives' who argue in a public forum on behalf of various segments of the population. I think this could be valuable because the automated debate and discussion might result in finding more win-win proposals that satisfy a larger fraction of the represented groups. Even if the AI representatives have no direct political power, just being able to propose unthought-of win-win compromises could be a big help.
In regards to the sneaky AI... Yeah, I think this will be quite feasible for an AI in the not too distant future. The only solutions I see currently are ones with too high of a 'safety tax'. Specifically, I think the problem can be avoided by training the super-intelligent super-persuasive AI in a sandboxed simulation which deliberately obfuscates the fact that the AI is being observed or contained. The simulation can use physics not matching our actual universe, and also avoid mentioning humans or our computing technology. I think that such an approach would buy a large margin of safety, but would not be economically competitive in the current AGI-race dynamics. There may be some sort of compromise solution, in which just some key facts are deliberately omitted, and some important deceptions deliberately introduced, in order to trick even highly intelligent and persuasive defector systems into triggering alarms or falling into honeypots. One obvious win is to remove key information about how to build weapons of mass destruction, particularly self-replicating ones like nanotech or bioweapons, from the training dataset. That alone would probably buy a lot of safety for only a small sacrifice of training data. For more info, check out the WMDP benchmark.
Enjoyed your commentary on AI systems as societal mirrors, but it is a little American-centric. Have there been studies that show LLMs can perform similarly for other cultures? Is there sufficient training and testing data available to assume a similar performance could be achieved? Are we accepting that American culture is a universal culture? I wonder if other cultures will hold out longer than American culture in 'bend[ing] towards the mindspace of the machine' if their cultures are not so well mapped by current models?
For an interesting discussion related to what current AI systems might be missing out on, culturally, check out the second half of the podcast episode Sarah Hooker - 'Why the US AI Act Compute Thresholds Are Misguided' on Machine Learning Street Talk. She talks about how the internet training data is disproportionately skewed towards the most common languages. Particularly, English and Mandarin Chinese. A lot of people on Earth speak an under-represented native language at home, while having a second or third language which is one of the top 25 spoken languages. According to wikipedia, about 6.9 billion (out of 8.2 billion total) people speak one of the top 25 languages, at least as a second language if not first. I expect that this effect strongly skews the internet conversations and textual media (e.g. books, plays) that end up in the training data.
Your posts are always thoughtful, but somehow this one more than most. It feels like we are on the brink of a step-change. Maybe it has been that way since November 2022, but even more so now. Funny how some think LLMs are at or near their end point yet advances continue with broader impacts.
The study about LLM’s predicting survey results sounds pretty interesting. It might be seen as a sort of simulation, using a rather broad definition of “simulation.”
However, I think it’s important to point out that it’s not the same kind of simulation as a weather model or a flight simulator, where there’s a mathematical model that starts with a current state and steps forward in time to make predictions. (I’d love to see mechanistic interpretability research that explores how LLM’s actually do represent time.)
Also, surveys are themselves an abstraction, creating summary statistics that often don’t tell us much about real people.
I’ve never liked Baudrillard’s metaphors very much - they seem to obscure more than they reveal. But I think it’s fair to say that our understanding of what’s going on in the world is increasingly mediated by machines, and this hardens human culture so that it becomes difficult to change without doing a whole lot of work. We are stuck with common standards for many things like calendars, units of measure, money, and character sets because there’s so much computer infrastructure that’s built using them.
Similarly for LLM’s. People are already wary of ranking by “algorithms” and it’s going to get worse as more conversation happens via bots. Human culture will become human-and-machine culture. A lot of common knowledge already comes from Wikipedia, and LLM’s add another layer. There will likely be common myths and urban legends that are hard to change because of widespread embedding into machinery.
Great post as usual, Jack. Very interesting to note that Chinese manufacturers are using Llama 3 as a proxy for GPT and other resources only available Western, non-embargoed countries. A report came out at the end of last year suggesting that China is still leading the US in terms of AI talent, which makes for a delicate balance of tech vs. people. As the Biden administration was admonishing China as a competitor, it was also quietly relaxing immigration restrictions for academics, namely accomplished researchers in AI. I don't think this little dance is over, by a longshot.
In regards to the social science polling, I have been developing a related idea. I think it could be very valuable as a future governance tool to have 'AI representatives' who argue in a public forum on behalf of various segments of the population. I think this could be valuable because the automated debate and discussion might result in finding more win-win proposals that satisfy a larger fraction of the represented groups. Even if the AI representatives have no direct political power, just being able to propose unthought-of win-win compromises could be a big help.
In regards to the sneaky AI... Yeah, I think this will be quite feasible for an AI in the not too distant future. The only solutions I see currently are ones with too high of a 'safety tax'. Specifically, I think the problem can be avoided by training the super-intelligent super-persuasive AI in a sandboxed simulation which deliberately obfuscates the fact that the AI is being observed or contained. The simulation can use physics not matching our actual universe, and also avoid mentioning humans or our computing technology. I think that such an approach would buy a large margin of safety, but would not be economically competitive in the current AGI-race dynamics. There may be some sort of compromise solution, in which just some key facts are deliberately omitted, and some important deceptions deliberately introduced, in order to trick even highly intelligent and persuasive defector systems into triggering alarms or falling into honeypots. One obvious win is to remove key information about how to build weapons of mass destruction, particularly self-replicating ones like nanotech or bioweapons, from the training dataset. That alone would probably buy a lot of safety for only a small sacrifice of training data. For more info, check out the WMDP benchmark.
Interesting first article, so now we have a digital twin of ourselves which we can ask what we are thinking of.
I assume that someone can then take it and apply different stimulii to see what would change its mind. More scary than the Facebook experiments?
Ron
Enjoyed your commentary on AI systems as societal mirrors, but it is a little American-centric. Have there been studies that show LLMs can perform similarly for other cultures? Is there sufficient training and testing data available to assume a similar performance could be achieved? Are we accepting that American culture is a universal culture? I wonder if other cultures will hold out longer than American culture in 'bend[ing] towards the mindspace of the machine' if their cultures are not so well mapped by current models?
For an interesting discussion related to what current AI systems might be missing out on, culturally, check out the second half of the podcast episode Sarah Hooker - 'Why the US AI Act Compute Thresholds Are Misguided' on Machine Learning Street Talk. She talks about how the internet training data is disproportionately skewed towards the most common languages. Particularly, English and Mandarin Chinese. A lot of people on Earth speak an under-represented native language at home, while having a second or third language which is one of the top 25 spoken languages. According to wikipedia, about 6.9 billion (out of 8.2 billion total) people speak one of the top 25 languages, at least as a second language if not first. I expect that this effect strongly skews the internet conversations and textual media (e.g. books, plays) that end up in the training data.
Your posts are always thoughtful, but somehow this one more than most. It feels like we are on the brink of a step-change. Maybe it has been that way since November 2022, but even more so now. Funny how some think LLMs are at or near their end point yet advances continue with broader impacts.