Summary: A new study evaluated whether large language models (LLMs) can accurately account for human reading dynamics. The team compared eye-tracking data from 368 adult readers against predictions generated by over 400 neural network language models across syntactically complex texts, including “garden-path” sentences.
The study revealed a critical divergence between human cognitive architecture and transformer-based AI architectures. While LLM next-word prediction models successfully account for the initial processing speed of word recognition during smooth forward reading, they fail to predict the cognitive friction that occurs when humans integrate words into broader context.
Key Facts
- Initial Alignment vs. Subsequent Divergence: LLMs accurately model early visual word recognition time during linear forward reading, but fail to explain late-stage context integration.
- Underestimation of Cognitive Difficulty: Next-word predictability in AI models severely underpredicts the processing time human readers expend when resolving ambiguous or “garden-path” sentences (e.g., “The old man the boat”).
- The Role of Regressive Eye Movements: Backward eye movements account for roughly 20% of human reading fixations; standard autoregressive LLMs lack an equivalent structural mechanism for retrospective re-parsing.
- Benchmarking Scale: The researchers evaluated human eye-tracking metrics against more than 400 distinct large language model architectures to isolate where predictive algorithms fall short of cognitive reality.
- Roadmap for Cognitive AI: The findings demonstrate that while predictive autoregressive mechanisms contribute to sentence comprehension, human cognition relies on distinct multi-stage context-integration frameworks that future AI architectures must adopt to model human thought accurately.
Source: NYU
Why do we breeze through some sentences in a book or article but have to reread others to comprehend their meaning?
A team of linguists and data scientists has found a partial answer in AI—some of this processing parallels that of neural-network-based large language models (LLMs). However, other aspects of why we read this way cannot be explained by these technologies, revealing where human and AI language processing diverge and maintaining the mystery of some stages of the reading process.
The new study, by researchers from New York University and the University of Massachusetts Amherst, shows that humans and AI process language in similar ways during the earliest moments of reading: both rely on next-word predictions. However, as reading continues and passages become more complex—often requiring rereading—humans’ processing differs from that of AI, which is entirely built on next-word prediction and therefore cannot account for how we navigate these types of passages.
“Language models develop their remarkable language understanding capabilities by being trained to predict the next word in a sentence, which led us to ask whether the same predictive processes that drive these AI systems could also explain how humans comprehend sentences,” explains William Timkey, a linguistics doctoral student at NYU and the lead author of the paper, which appears in the journal Proceedings of the National Academy of Sciences (PNAS).
“We found that LLMs can explain how long it takes people to recognize words when their eyes move smoothly forward through a text, but they fail to capture the cases where people have difficulty integrating a word into the larger context of a sentence, which is often accompanied by rereading.”
The authors note that despite the remaining uncertainty on how humans read—notably the rereading of passages—the findings nonetheless offer a potential roadmap for both improving language learning and addressing reading-related afflictions.
“We now know a little bit better how humans and models are different,” says Brian Dillon, a professor of linguistics at the UMass Amherst and the paper’s senior author. “That is the first step in understanding how we can close that gap, which we want to do because that could have enormous advantages down the road.”
“Our work shows that AI can be very valuable for cognitive science, but it is not enough,” adds Tal Linzen, an associate professor of linguistics and data science at NYU and one of the paper’s authors. “The human mind does not always work like standard AI systems—for instance, 20 percent of our eye movements when reading are backward, and AI models cannot explain when we decide to do that. We now have our work cut out for us to create computational models that more closely match the human mind and that can help us understand in detail how it operates.”
When we see words on a page, we go through mental processes of taking the visual information of the letters, accessing the meaning of the word, and then integrating that with the rest of a sentence. While much of reading is driven by word prediction, less clear are its limits—a question the researchers explored in the PNAS study.
To do so, they deployed LLMs because their predictive-text feature aligns with some theories of how the brain works: the prediction process drives our ability to comprehend sentences.
“LLMs seem to capture some of the properties of language as we understand it—they can generate text fluently and they appear to react in a way that suggests they have some understanding of what’s going on,” explains Dillon, who directs the Computational Sentence Processing Lab in the Department of Linguistics at UMass Amherst.
“We build a mental representation of what we think a sentence means based on the words on the page, then use that representation to make predictions about the next words, and then update our mental representation when those predictions are wrong,” adds Timkey.
In the study, the researchers used eye-tracking technology to analyze 368 adult readers, focusing on how long participants spent reading—and rereading—each word of carefully designed sentences. These included a diverse set of syntactically challenging sentences, known as garden- path sentences—grammatically correct sentences that start in such a way that a reader’s initial interpretation will likely be incorrect.
For instance, take the sentence “The old man the boat.” Readers may initially think the sentence is about an old man, but instead, the sentence means that old people are manning a boat. Such sentences, the authors note, are good candidates for understanding how we process complex passages.
The researchers then compared those eye movements with predictions generated by more than 400 AI language models.
The results showed that AI models’ next word predictions can explain the first step of processing each word of a sentence: identifying the word from a sequence of letters. However, they can’t explain the next step of integrating that word into the larger meaning of the sentence—a process that is particularly difficult for humans in garden-path sentences, and one which still remains poorly understood.
“The predictability of a word really doesn’t even come close to explaining just how much time we spend on difficult words and garden-path sentences,” says Timkey. “LLMs were drastically underpredicting the type of difficulty that we experience when reading.”
“It’s in that second stage of processing—recognizing a word and then integrating it with other words in passages—where we find big gaps between what word predictability can explain and what we need cognitive models to explain,” adds Linzen.
Funding: The research was supported by grants from the National Science Foundation (BCS-2020914, BCS-2020945, IIS-2504953, and IIS-2504954).
Key Questions Answered:
A: Garden-path sentences are grammatically correct sentences structured so that a reader’s initial interpretation is likely incorrect (e.g., “The old man the boat”). They are ideal for psycholinguistic research because they force the brain to stop, re-evaluate initial assumptions, and reread previous words to integrate the correct meaning.
A: LLMs and human readers align during the first stage of word processing—word recognition. When eyes move smoothly forward through straightforward text, the statistical predictability of a word in an LLM accurately models how quickly a human reader will visually recognize that word.
A: Standard LLMs process text sequentially through unidirectional, left-to-right next-word prediction. Human readers, however, perform backward eye movements (regressions) about 20% of the time to re-examine earlier words when sentence integration fails, a cognitive re-parsing mechanism that standard predictive AI architectures do not possess.
Editorial Notes:
- This article was edited by a Neuroscience News editor.
- Journal paper reviewed in full.
- Additional context added by our staff.
About this AI research news
Author: James Devitt
Source: NYU
Contact: James Devitt – NYU
Image: The image is credited to Neuroscience News
Original Research: Open access.
“Eye movements reveal a dissociation between prediction and structural processing difficulty in language comprehension” by William Timkey, Kuan-Jung Huang, Byung-Doh Oh, Grusha Prasad, Suhas Arehalli, Tal Linzen, Brian Dillon. PNAS
DOI:10.1073/pnas.2532230123
Abstract
Eye movements reveal a dissociation between prediction and structural processing difficulty in language comprehension
In the process of extracting a meaning from a text, our eyes linger much more on some words than others, and we often reread earlier portions of the text. These disruptions to the reading process are particularly common in syntactically ambiguous sentences.
What explains the difficulty presented by these sentences? One prominent hypothesis explains it as a special case of the impact of a word’s predictability (operationalized via surprisal) on the difficulty of processing the word.
This contrasts with theories that attribute these disruptions to errors in the structure-building process. Earlier attempts to address this debate have been inconclusive because of small numbers of participants, coarse measurements of the reading process that are ill-suited to disentangling these competing views, and a limited range of surprisal estimates.
Here, we conduct a large-scale study (n = 368) examining eye movements during the reading of syntactically challenging sentences, using 409 types of surprisal estimates from language models with multiple architectures and training settings.
We find a stark dissociation: Early effects of syntactic disambiguation are well-approximated by language model surprisal, but syntactic disambiguation incurs a significant additional cost, reflected in an increase in rereading that is not explained by language model surprisal. We conclude that surprisal can capture routine structure-building, but not the cost of detecting or correcting errors in the structure-building process.
