The study about LLM’s predicting survey results sounds pretty interesting. It might be seen as a sort of simulation, using a rather broad definition of “simulation.”
However, I think it’s important to point out that it’s not the same kind of simulation as a weather model or a flight simulator, where there’s a mathematical model that starts with a current state and steps forward in time to make predictions. (I’d love to see mechanistic interpretability research that explores how LLM’s actually do represent time.)
Also, surveys are themselves an abstraction, creating summary statistics that often don’t tell us much about real people.
I’ve never liked Baudrillard’s metaphors very much - they seem to obscure more than they reveal. But I think it’s fair to say that our understanding of what’s going on in the world is increasingly mediated by machines, and this hardens human culture so that it becomes difficult to change without doing a whole lot of work. We are stuck with common standards for many things like calendars, units of measure, money, and character sets because there’s so much computer infrastructure that’s built using them.
Similarly for LLM’s. People are already wary of ranking by “algorithms” and it’s going to get worse as more conversation happens via bots. Human culture will become human-and-machine culture. A lot of common knowledge already comes from Wikipedia, and LLM’s add another layer. There will likely be common myths and urban legends that are hard to change because of widespread embedding into machinery.
>There will likely be common myths and urban legends that are hard to change because of widespread embedding into machinery.
Not only myths and urban legends but lies, misrepresentations and false history. The planet is awash in lies and nobody with the money to do it is gonna filter them out, because the dominant lies in our society are the lies that serve the controllers' interests.
This is one of the many reasons why running your own LLM must not be encumbered by police-state policy policemen.
In regards to the social science polling, I have been developing a related idea. I think it could be very valuable as a future governance tool to have 'AI representatives' who argue in a public forum on behalf of various segments of the population. I think this could be valuable because the automated debate and discussion might result in finding more win-win proposals that satisfy a larger fraction of the represented groups. Even if the AI representatives have no direct political power, just being able to propose unthought-of win-win compromises could be a big help.
In regards to the sneaky AI... Yeah, I think this will be quite feasible for an AI in the not too distant future. The only solutions I see currently are ones with too high of a 'safety tax'. Specifically, I think the problem can be avoided by training the super-intelligent super-persuasive AI in a sandboxed simulation which deliberately obfuscates the fact that the AI is being observed or contained. The simulation can use physics not matching our actual universe, and also avoid mentioning humans or our computing technology. I think that such an approach would buy a large margin of safety, but would not be economically competitive in the current AGI-race dynamics. There may be some sort of compromise solution, in which just some key facts are deliberately omitted, and some important deceptions deliberately introduced, in order to trick even highly intelligent and persuasive defector systems into triggering alarms or falling into honeypots. One obvious win is to remove key information about how to build weapons of mass destruction, particularly self-replicating ones like nanotech or bioweapons, from the training dataset. That alone would probably buy a lot of safety for only a small sacrifice of training data. For more info, check out the WMDP benchmark.
An essay that resonates deeply with me, particularly: "AI systems are creative mirrors, they are machine spirits of the human unconscious, they are value simulacras: Are you getting this yet? We are not dealing with calculators here. We are not dealing with simple tools. We are dealing with vast high-dimensional artifacts that encode within themselves the culture on which they have been trained and can reflect this culture back."
Great post as usual, Jack. Very interesting to note that Chinese manufacturers are using Llama 3 as a proxy for GPT and other resources only available Western, non-embargoed countries. A report came out at the end of last year suggesting that China is still leading the US in terms of AI talent, which makes for a delicate balance of tech vs. people. As the Biden administration was admonishing China as a competitor, it was also quietly relaxing immigration restrictions for academics, namely accomplished researchers in AI. I don't think this little dance is over, by a longshot.
Enjoyed your commentary on AI systems as societal mirrors, but it is a little American-centric. Have there been studies that show LLMs can perform similarly for other cultures? Is there sufficient training and testing data available to assume a similar performance could be achieved? Are we accepting that American culture is a universal culture? I wonder if other cultures will hold out longer than American culture in 'bend[ing] towards the mindspace of the machine' if their cultures are not so well mapped by current models?
For an interesting discussion related to what current AI systems might be missing out on, culturally, check out the second half of the podcast episode Sarah Hooker - 'Why the US AI Act Compute Thresholds Are Misguided' on Machine Learning Street Talk. She talks about how the internet training data is disproportionately skewed towards the most common languages. Particularly, English and Mandarin Chinese. A lot of people on Earth speak an under-represented native language at home, while having a second or third language which is one of the top 25 spoken languages. According to wikipedia, about 6.9 billion (out of 8.2 billion total) people speak one of the top 25 languages, at least as a second language if not first. I expect that this effect strongly skews the internet conversations and textual media (e.g. books, plays) that end up in the training data.
Your comment implies that it constitutes a problem that other cultures develop at a different pace or with different priorities, causing a lack of representation in your domain of interest. The question here is whether a divergent proportional representation of one group in any given metric is in itself a legitimate cause for furrowed foreheads and action agendas...
Where does it end? Do you stop at, "The Pacific Ocean has too few birds." Or "Left handed people with hangnails are under-represented as grocery store cashiers"? It's become ubiquitous secular dogma that all things must be equalized by State intervention: a mission statement that enables metastatic policy mischief.
I apologize for straying a bit offtopic but I consider it a useful exercise, to help intelligent people recognize questionable assumptions at play.
I didn't intend such an implication. I'm very much in favor of cultural diversity and letting people choose their own levels of involvement with tech and global society. I was simply seeking to point out that we may wish to consider the nature of our AI training data more carefully. Specifically, I think it'd be very interesting to deliberately design a synthetic language which is maximally inclusive of all known human concepts across all known human languages, then translate all the training data into this synthetic language. How might we measure the impact of such a synthetic language training set? We could test it's ability to reason in the context of many different languages (passing through its internal translator). I suspect there would be an improvement in quality of thought from such a unification.
Your posts are always thoughtful, but somehow this one more than most. It feels like we are on the brink of a step-change. Maybe it has been that way since November 2022, but even more so now. Funny how some think LLMs are at or near their end point yet advances continue with broader impacts.
The study about LLM’s predicting survey results sounds pretty interesting. It might be seen as a sort of simulation, using a rather broad definition of “simulation.”
However, I think it’s important to point out that it’s not the same kind of simulation as a weather model or a flight simulator, where there’s a mathematical model that starts with a current state and steps forward in time to make predictions. (I’d love to see mechanistic interpretability research that explores how LLM’s actually do represent time.)
Also, surveys are themselves an abstraction, creating summary statistics that often don’t tell us much about real people.
I’ve never liked Baudrillard’s metaphors very much - they seem to obscure more than they reveal. But I think it’s fair to say that our understanding of what’s going on in the world is increasingly mediated by machines, and this hardens human culture so that it becomes difficult to change without doing a whole lot of work. We are stuck with common standards for many things like calendars, units of measure, money, and character sets because there’s so much computer infrastructure that’s built using them.
Similarly for LLM’s. People are already wary of ranking by “algorithms” and it’s going to get worse as more conversation happens via bots. Human culture will become human-and-machine culture. A lot of common knowledge already comes from Wikipedia, and LLM’s add another layer. There will likely be common myths and urban legends that are hard to change because of widespread embedding into machinery.
>There will likely be common myths and urban legends that are hard to change because of widespread embedding into machinery.
Not only myths and urban legends but lies, misrepresentations and false history. The planet is awash in lies and nobody with the money to do it is gonna filter them out, because the dominant lies in our society are the lies that serve the controllers' interests.
This is one of the many reasons why running your own LLM must not be encumbered by police-state policy policemen.
In regards to the social science polling, I have been developing a related idea. I think it could be very valuable as a future governance tool to have 'AI representatives' who argue in a public forum on behalf of various segments of the population. I think this could be valuable because the automated debate and discussion might result in finding more win-win proposals that satisfy a larger fraction of the represented groups. Even if the AI representatives have no direct political power, just being able to propose unthought-of win-win compromises could be a big help.
In regards to the sneaky AI... Yeah, I think this will be quite feasible for an AI in the not too distant future. The only solutions I see currently are ones with too high of a 'safety tax'. Specifically, I think the problem can be avoided by training the super-intelligent super-persuasive AI in a sandboxed simulation which deliberately obfuscates the fact that the AI is being observed or contained. The simulation can use physics not matching our actual universe, and also avoid mentioning humans or our computing technology. I think that such an approach would buy a large margin of safety, but would not be economically competitive in the current AGI-race dynamics. There may be some sort of compromise solution, in which just some key facts are deliberately omitted, and some important deceptions deliberately introduced, in order to trick even highly intelligent and persuasive defector systems into triggering alarms or falling into honeypots. One obvious win is to remove key information about how to build weapons of mass destruction, particularly self-replicating ones like nanotech or bioweapons, from the training dataset. That alone would probably buy a lot of safety for only a small sacrifice of training data. For more info, check out the WMDP benchmark.
An essay that resonates deeply with me, particularly: "AI systems are creative mirrors, they are machine spirits of the human unconscious, they are value simulacras: Are you getting this yet? We are not dealing with calculators here. We are not dealing with simple tools. We are dealing with vast high-dimensional artifacts that encode within themselves the culture on which they have been trained and can reflect this culture back."
Great post as usual, Jack. Very interesting to note that Chinese manufacturers are using Llama 3 as a proxy for GPT and other resources only available Western, non-embargoed countries. A report came out at the end of last year suggesting that China is still leading the US in terms of AI talent, which makes for a delicate balance of tech vs. people. As the Biden administration was admonishing China as a competitor, it was also quietly relaxing immigration restrictions for academics, namely accomplished researchers in AI. I don't think this little dance is over, by a longshot.
Interesting first article, so now we have a digital twin of ourselves which we can ask what we are thinking of.
I assume that someone can then take it and apply different stimulii to see what would change its mind. More scary than the Facebook experiments?
Ron
Enjoyed your commentary on AI systems as societal mirrors, but it is a little American-centric. Have there been studies that show LLMs can perform similarly for other cultures? Is there sufficient training and testing data available to assume a similar performance could be achieved? Are we accepting that American culture is a universal culture? I wonder if other cultures will hold out longer than American culture in 'bend[ing] towards the mindspace of the machine' if their cultures are not so well mapped by current models?
For an interesting discussion related to what current AI systems might be missing out on, culturally, check out the second half of the podcast episode Sarah Hooker - 'Why the US AI Act Compute Thresholds Are Misguided' on Machine Learning Street Talk. She talks about how the internet training data is disproportionately skewed towards the most common languages. Particularly, English and Mandarin Chinese. A lot of people on Earth speak an under-represented native language at home, while having a second or third language which is one of the top 25 spoken languages. According to wikipedia, about 6.9 billion (out of 8.2 billion total) people speak one of the top 25 languages, at least as a second language if not first. I expect that this effect strongly skews the internet conversations and textual media (e.g. books, plays) that end up in the training data.
Your comment implies that it constitutes a problem that other cultures develop at a different pace or with different priorities, causing a lack of representation in your domain of interest. The question here is whether a divergent proportional representation of one group in any given metric is in itself a legitimate cause for furrowed foreheads and action agendas...
Where does it end? Do you stop at, "The Pacific Ocean has too few birds." Or "Left handed people with hangnails are under-represented as grocery store cashiers"? It's become ubiquitous secular dogma that all things must be equalized by State intervention: a mission statement that enables metastatic policy mischief.
I apologize for straying a bit offtopic but I consider it a useful exercise, to help intelligent people recognize questionable assumptions at play.
I didn't intend such an implication. I'm very much in favor of cultural diversity and letting people choose their own levels of involvement with tech and global society. I was simply seeking to point out that we may wish to consider the nature of our AI training data more carefully. Specifically, I think it'd be very interesting to deliberately design a synthetic language which is maximally inclusive of all known human concepts across all known human languages, then translate all the training data into this synthetic language. How might we measure the impact of such a synthetic language training set? We could test it's ability to reason in the context of many different languages (passing through its internal translator). I suspect there would be an improvement in quality of thought from such a unification.
Your posts are always thoughtful, but somehow this one more than most. It feels like we are on the brink of a step-change. Maybe it has been that way since November 2022, but even more so now. Funny how some think LLMs are at or near their end point yet advances continue with broader impacts.