The technological struggles are in some ways beside the point. The financial bet on artificial general intelligence is so big that failure could cause a depression.
LLMs weren’t out of left field. Chatbots have been in development since the '90s at least. Probably even longer. And word prediction has been around at least a decade. People just don’t pay attention until it’s commercially available.
Most ai research has serious and obvious scaling problems. It did well at first, but scaling up the training didn’t significantly improve the results. LLMs went from more of the same to a gold rush the day it was revealed that they scaled “well” (relatively speaking). They then went through orders of magnitude improvements very quickly because they could (unlike previous ai training models which wouldn’t have benefited like this).
We’ve had chatbots for decades, but with a the same low capability ceiling that most other old techniques had, they really were a different beast to modern LLMs with their stupidly excessive training regimes.
LLMs weren’t out of left field. Chatbots have been in development since the '90s at least. Probably even longer. And word prediction has been around at least a decade. People just don’t pay attention until it’s commercially available.
Modern llms were a left field development.
Most ai research has serious and obvious scaling problems. It did well at first, but scaling up the training didn’t significantly improve the results. LLMs went from more of the same to a gold rush the day it was revealed that they scaled “well” (relatively speaking). They then went through orders of magnitude improvements very quickly because they could (unlike previous ai training models which wouldn’t have benefited like this).
We’ve had chatbots for decades, but with a the same low capability ceiling that most other old techniques had, they really were a different beast to modern LLMs with their stupidly excessive training regimes.