Summary: New research has identified structural and evolutionary principles that govern how both children and sensory networks acquire language. Research combines cognitive linguistics and deep learning to demonstrate the power of “repetitive learning”, a process in which a language retrains itself over many generations to become increasingly organized and structured data becomes easier to learn.
By building a deep neural network modeled after the child's progressive learning stages, the researchers proved that structural adaptations naturally arise from communication stress and transmission system errors.
Important Facts
- The Iterated Evolution Paradigm: Iterative learning emphasizes that human language is not static but an evolving system that retrains itself over successive generations to increase structural efficiency and ease the burden of cognitive learning.
- Imitating a Child's Mind: Researchers have built a deep linear neural network designed with structural learning features similar to a child's brain, which generates sequential versions of the computer's brain in data structures that mimic human language.
- Error-Driven Architecture: Children acquire language in systematic stages, sometimes making careless mistakes due to the production of too much information (eg, assuming that all birds with wings fly until they meet a penguin). In the transmission from one generation to the next, these inadvertent errors filter the data, causing highly organized, easily readable patterns of language to be preserved while the disorganized elements are systematically forgotten.
- Absolute Depth: To map the exact neural basis of this song evolution, the team deployed deep linear networks. Experiments have proven that iterative learning is only successful if the network has enough depth and multiple processing layers; shallow networks with few layers completely fail to capture the systematic principles that make language readable.
- II Intersection modern: Research proves that the structural evolution seen in major AI productivity tools is based on the same cognitive principles found in child development. The structure of the learning network and the complexity of its environment determine how effectively it can acquire and transmit language.
- The Cross of Understanding: Lead author Dr. Devon Jarvis notes that although deep linear networks and repetitive learning have existed as separate concepts in different literatures, combining them proves that language adapts directly to learning based on how children process data and favor the reuse of information.
Source: University of the Witwatersrand
New research from the University of the Witwatersrand, South Africa, has major implications for understanding both the development of human language and the behavior of large varieties of artificial intelligence languages.
Culture is key, as well as understanding “repetitive learning”, which states that language evolves over generations (in humans and in computers) to be better constructed.
“We created a computer brain with the same characteristics as a child, and compared it to the behavior we see in children's brains.” Then we fed it information with the same characteristics found in human language and watched how generations (versions) of computer brains learn.”
“It turns out that computer geniuses find the structure of data in the same way that children like certain aspects of language in learning. It also shows that the dataset (language) is highly organized across generations because it makes learning easier,” said lead author Dr. Devon Jarvis, Lecturer in the School of Computer Science and Applied Mathematics (CSAM), and Fellow in Wits and Neural Machine Intelligence (NeINDM Discovery Institute).
Their findings were recently published in a paper titled: Order and Order Emerge from Iterative Learning in Deep Linear Networks in a respected journal Proceedings of the National Academy of Sciences (PNAS).
It all starts in childhood
Jarvis explains that children have an amazing ability to learn language quickly during early development. They learn the world in stages: they start with basic concepts and gradually understand more complex ones.
Jarvis says: “First, they learn that plants and animals are different things, and then they learn that there are different kinds of animals.
Take the penguin, for example. Children learn that birds have wings so they can fly. AH! But they are confused that the penguin cannot fly. Here, they overextend, and mistakes are made, which helps them learn new information: penguins can't fly, but they can swim, AHA!. And little by little, they build a systematic understanding of the world with increasing accuracy.
“Although this continuous acquisition of knowledge has its advantages, the work focuses on the effects of generations of learners. A child learns a certain language from his parents, and eventually they pass it on to their children.
“Like the example of penguins, these errors are not automatic and are caused by an overload of information. The result is that the parts of the language that are easily learned are remembered and reused, while the parts that are less structured are forgotten. Basically, people are able to learn but only with the pressure of communication do we really see the depth of their intelligence,” explains Javisgence.
Not all neural networks are equal
Researchers used deep neural networks (mathematical models that simulate the way the brain processes information) to study the neural basis of this process. They found that recurrent learning only works well if the network has enough depth, multiple layers of processing, and a sufficiently complex language. Shallow networks, those with few layers, fail to capture the systematic principles that make language readable.
This suggests that the structure of the learning environment, whether natural or artificial, as well as the richness of the environment, play an important role in how well language structure is absorbed and transmitted. A point that will also come up in the recent development of generative AI models, which rely heavily on scale for their evolving capabilities.
Jarvis continues: “Snippets of this book have been around for a long time in a variety of books. Deep linking methods are examples of child development and repetitive learning that linguists have known for years.”
But the combination of these two ideas seems to make a useful point: that language changes to be read based on the specific situation of how children learn in stages and prefer to reuse information rather than learn new things.”
“The fact that this has been demonstrated in a very simple version of the technology that underpins the modern boom in AI tools is also encouraging and suggests that at the intersection of many fields there are fundamental principles of consciousness.”
Important Questions Answered:
A: Because those mistakes are not just; they are predictable signs of a brain trying to find order. If a child overcomplicates a rule, such as thinking that a penguin flies because it has wings, it uses a formal shortcut. Over generations of parents passing on speech to children, the messy, unstructured, and difficult parts of language are forgotten, while the simpler, rule-based parts are retained and reused.
A: Goes down to processing depth and layers. Researchers at Wits have found that shallow networks with very few layers are completely blind to the hidden regularities of complex language, which makes them fail at conveying structured information. Deep networks, which reflect a child's ability to learn the world in layers, require multiple layers of depth to successfully absorb, organize, and transmit the structures of language.
A: Proves that the fundamental principles of human cognition are the exact same forces that drive modern artificial intelligence. The modern boom in artificial intelligence tools relies heavily on massive computing scale and depth of layers to achieve its potential. This research shows that even a simple, linearly intensive version of this technology replicates the way human language evolves to be readable.
Editor's Notes:
- This article was edited by a Neuroscience News editor.
- The journal paper is fully revised.
- Additional content added by our staff.
About this AI and language learning research issues
Author: Shirona Patel
Source: University of the Witwatersrand
Contact person: Shirona Patel – University of the Witwatersrand
Image: Image posted in Neuroscience News
Actual research: Open access.
“Structure and organization emerge from recurrent learning in deep linear networks” by Devon Jarvis, Richard Klein, Benjamin Rosman, and Andrew M. Saxe. PNAS
DOI:10.1073/pnas.2509739123
Abstract
Order and organization emerge from iterative learning in deep linear networks
Humans have an amazing ability to synthesize things—to reason about new situations by combining elements of previous experiences. Language provides one of the prime examples of this ability and modern machine learning draws a lot of inspiration from language.
A recent example is iterative learning, a process where generations of networks learn from the output of previous learners. The result is the development of a network “language” or output labels for a given input to a composition structure.
Here we study theoretically the evolution of compositional language, and the ability of simple neural networks to develop this synthesis for systematic execution.
We build on previous theoretical work on linear networks, which mathematically describe systematic adaptation, by a) applying shallow and deep linear network analysis to the iterative learning process by detecting the linear dynamics of learning across generations; b) to modify the definition of organization to understand the advantages and limitations of repeated learning.
We find that iterative learning facilitates systematic generalization over standard training paradigms by revealing the underlying structure of output labels.
Our results confirm a long-held assumption: that multiple generations of repeated learning are required for a novel structure to emerge, which can outperform a single-generation network trained with a prior state.
However, in order for the network to treat the input systematically and ignore the unusual features, the network must be trained on a very large dataset. Therefore, we define “weak systematic generalization” to explain this in the order from the scale.
