Recursive Language Models Face Uncertainty: The Surprising Performance of a Long-Term Context Retrieval System
Long context management is still a major challenge for language models: even with extended context windows, models often fail to extract, reason, and use information across long contexts. Recent works such as Recursive Language Models (RLMs) have tackled this challenge with an agent-based approach to breaking down long content into small recursive queries using programmatic interactions with direct understanding. Although promising, the success of RLMs is highly dependent on how these cues to contextual interactions are selected, which remains unexplored. In this paper, we study this problem and introduce Long-Term Self-Reflection System Search (SRLM), a framework that leverages system-based context interactions with self-reflection and uncertainty. SRLM uses three internal signals: stability, length of follow-up thinking, and verbal confidence. These serve as complementary indicators of the model's internal uncertainty, and the model uses them to evaluate and compare candidate context interactions. Extensive testing across various benchmark data sets, context lengths, and baseline models, shows that SRLM consistently outperforms baselines, delivering up to 22% improvement over RLMs under the same time budget. Our findings indicate that replication itself is not the primary driver of performance in RLMs, and simple self-reflective program searches can match or outperform RLM without requiring accountability or obvious replication mechanisms. We find that for context lengths within the context window of the model, iterative RLMs generally degrade performance relative to the baseline model, while SRLM produces consistent and robust gains in both short and long contexts. We also find that RLM performs poorly in semantically deep tasks, where heuristic search is insufficient and a broad understanding of the context is required, while self-reflection in SRLM provides a semantic signal that better guides thinking in these challenging situations of long context.
