>> "Maybe there are just no mysteries? Maybe everything in AI exhibits a scaling law. This is a big deal - it suggests that we've found a common technology (here, neural nets) that yield smooth and predictable performance increases in a seemingly arbitrary range of domains"
My current suspicion is that if someone figures out the right way to break down reasoning and meta-learning into subtasks, and generate sufficient synthetic data to train specialized models for that, then hook them up in a 'society of mind' sort of agent scaffold, you'd see a substantial boost in general reasoning and problem-solving capabilities. I expect this would be quite noticeable even at relatively small model model sizes and modest quantities of fine-tuning data.
The human cortex certainly seems to be organized like groups of competing experts within regions that collaborate with groups in other regions. Lots of not-so-smart not-quite-agents with specialized skills working together to form something greater than the individual parts would suggest.
The toddler: a groundbreaking advancement in adversarial training!
>> "Maybe there are just no mysteries? Maybe everything in AI exhibits a scaling law. This is a big deal - it suggests that we've found a common technology (here, neural nets) that yield smooth and predictable performance increases in a seemingly arbitrary range of domains"
My current suspicion is that if someone figures out the right way to break down reasoning and meta-learning into subtasks, and generate sufficient synthetic data to train specialized models for that, then hook them up in a 'society of mind' sort of agent scaffold, you'd see a substantial boost in general reasoning and problem-solving capabilities. I expect this would be quite noticeable even at relatively small model model sizes and modest quantities of fine-tuning data.
The human cortex certainly seems to be organized like groups of competing experts within regions that collaborate with groups in other regions. Lots of not-so-smart not-quite-agents with specialized skills working together to form something greater than the individual parts would suggest.
The pi zero robot might be better. Infinite patience to wash forks until the baby does something else