They are moving beyond just big transformer blob LLM text prediction. Mixture of Experts is not preassembled for example, it's something like x empty experts with an empty router and the experts and routing emerges naturally with training, modeling the brain part architecture we see the brain more. There is stuff "Integrated Gated Calculator (IGC)" in Jan 2025 which makes a premade calculator neural network and integrates it directly into the neural network and gets around the entire issue of making LLMs do basic number computation and the clunkiness of generating "run tool tokens". The model naturally learns to use the IGC built into itself because it will always beat any kind of computation memorization in the reward function very quickly.
Models are truly input multimodal now. Feeding an image, feeding audio and feeding text all go into separate input nodes, but it all feeds into the same inner layer set and outputs text. This also mirrors how brains work more as multiple parts integrated in one whole.
Humans in some sense are not empty brains, there is a lot of stuff baked in our DNA and as the brain grows it develops a baked in development program. This is why we need fewer examples and generalize way better.
Though there is info in DNA etc, you likely missed the biggest source of why we learn much faster. Search for Pim van Lommel near death research and find out how wrong the classic consciousness arises from the brain hypothesis is.
You're not likely to find much support on this forum for these ideas. For those that have interest, the book Irreducible Mind: Toward a Psychology for the 21st Century is a well-written treatise on the topic.
A gentler step in that direction is to see what Michael Levin and his lab are up to. He is looking for (one aspect of) intelligence, and finding it at the cellular level and below, even in an agential version of bubble sort. He's certainly challenging the notion that consciousness is limited to brain cells. All of his findings arise through experimental observation, so it forces some reckoning in a way that sociological research doesn't.
Models are truly input multimodal now. Feeding an image, feeding audio and feeding text all go into separate input nodes, but it all feeds into the same inner layer set and outputs text. This also mirrors how brains work more as multiple parts integrated in one whole.
Humans in some sense are not empty brains, there is a lot of stuff baked in our DNA and as the brain grows it develops a baked in development program. This is why we need fewer examples and generalize way better.