If you get better at that, and language models got better at predicting the next word than humans in around 2019, if you keep getting better at that, you start running out of easy words. So initially you just had to predict simple sentence structure. Later on, the model has to predict simple facts about the world, like if I complete a sentence like « the capital of France is », it has to be like, okay, that's Paris. And once it's good at that and it keeps improving, the only thing that's left is predicting the answer of some complex physics equation.
★ In the digest▶ 8:29