Loop Engineering for Hierarchical Retrieval: Reading a Long Document with a Table of Contents

Part II of Enterprise Document Intelligence, a series that builds an enterprise RAG system with four building blocks: document analysis, querying, retrieval, and execution. The recovery brick is a trilogy:

This friend (7quater) handles the case where the brick meets production: a document too long to read, with a table of contents too long to discard.

📓 Notebooks that might work are on GitHub: doc-intel/notebooks-vol1.

We are working on this in NIST SP 800-53 Rev. 5 (Security and Privacy Controls for Information Systems and OrganizationsUS Government work, US public domain) and NIST Cybersecurity Framework 2.0. Active code methods call OpenAI services governed by the OpenAI Terms of Use.

1. Problem: the table of contents is too long to render

NIST SP 800-53 is 492 pages. It describes the security controls that the US federal system must meet, one control at a time, twenty of them. The user asks a simple question:

“What does account management need?”

The answer is five pages, control AC-2, on pages 46 to 50. A naive RAG embeds all pages and takes the top-k most similar to the query. It fails in a particular way here: words account, managers, controlagain access stay for hundreds of pages, because all the text is about controls. Top-k returns AC-2 mixed with AC-3, AC-17, a few research controls, and a list of names, and the production model has to guess which one was meant. The bill is paid twice: you have embedded 492 pages, and the answer is still vague.

One opens the table of contents instead. Here's the catch that makes this article necessary: ​​the table of contents itself 358 entries. You wouldn't give your colleagues all 358 lines over 492 pages. The technician examines the chapter write first, eleven articles, choice Controlsopen twenty familieschoose Access Controlopening its door controlsthen sits down AC-2. Level up first, then down, one small decision at a time. That's the loop that this article creates, and it's the thesis of the series in a nutshell: raise a professional. Do what the pros do. Do not dump the entire document, or the entire table of contents, into the model at once.

2. Table of contents, three levels deep

Here is the original content page of the document, next to the toc_df the parser reads in its native frame.

The printed page stops at the family level. The analyst digs deeper. Article 5B turns the traditional PDF structure into a toc_dfone line per subject, and in this document that is 358 lines across three levels, from the top eleven chapters down to all 316 individual controls (AC-1, AC-2and so on), each with its own range of pages. Twenty control families live in between. That tree is the one that goes to bring it back. If the document does not have a traditional outline to navigate, article 5septies reconstructs one from the page; here the framework is clean, so we use it directly.

3. Loop: one level at a time

Retrieval goes up-down the tree. It offers only LLM the current leveleleven chapter titles first, each as a single line: title, page width, and where the titles are not confusing short keyword. The model selects the branch that answers the query. If that branch is wider and has better children, the return opens it, its children become the next level, and the step repeats. It stands on a leaf, or in a part small enough to be read in its entirety.

There are two things that come out of this situation. A long table of contents is never completely entered: the model reads eleven lines, then twenty, then twenty-two, never 358 at once. And going down is effectively optional. In a short paper the top level has no children to open, so the loop works once and behaves exactly like a flat path. The model judges at every level whether there is a reason for deepening.

4. Step by step, by code

A step is a single task, reason_on_toccalled once per level. It reads the level as text, not a table, so the level of twenty entries is twenty short lines, not a grid that expands a column with each keyword.

# Top-down: feed ONE level at a time, never the whole 358-row TOC.
# Each entry is one compact line: title + page range (+ keyword hits).
level = toc_df[toc_df.level == toc_df.level.min()]      # 11 chapter titles

while True:
    pick    = reason_on_toc(question, level, client=client)   # one LLM call
    section = level[level.id.isin(pick.section_ids)]
    kids    = immediate_children(toc_df, section)
    if kids.empty or section.n_pages <= SMALL:               # leaf, or small enough
        break
    level = kids                                             # open it, descend

# 11 chapters -> 20 families -> 25 controls -> AC-2 ACCOUNT MANAGEMENT (pp. 46-50)

It started with the question of managing the account against the real document, and went down one level at a time. At each level the model reads the topics of that level, the reasons, and then chooses the branch to open. The reason below is the model itself, word for word from the beginning.

Fifty-six entries were read across three sub-calls, never a 358-line table of contents. The generation then reads five pages, and some 315 controls never enter the notification.

If the title alone is not enough to separate the two candidates, the retrieval depends on that keyword, leaving one line for each entry: how many query keywords fall within that branch. It is an abbreviation in case a word is used in one section and defined in another only (at least the right used throughout Access Controlbut explained once in List of words). It is never a column for each keyword.

5. If we read the whole section

The loop stops descending in three cases: the selected branch is a leaf, it is already small enough to read in its entirety, or the query is list which needs everything under it. Listing case by NIST CSF from Article 12 (listing): all GOVERN clauses requested, router selects all Appendix A. CSF Coreand because the listing is comprehensive, retrieval reads the entire appendix rather than down to a single topic.

Here's that second drop, in the same step of the route but with the listing question. It reaches the category that holds the list at one level and stops, because there is nothing better that sits below it in the table of contents.

That gives the loop three areas of control from Article 13bis (loop engineering): a bullet (broad branch with children), a termination (leaf, part, or list), and recovery that changes each iteration (it goes down the level, it can't read the one it just judged), with the tree bound so it can't spin. It differs from Part III pipeline loops in how it reacts: document structure, not production output.

6. Tokens and accuracy, together

Hierarchical routing wins on both axes at the same time, which is rare.

Accuracy. Flat top-k over 492 pages competes with AC-2 against all other control mentions account or access. The route commits to AC-2 account management by word and read it in its entirety, so the five answer pages come together instead of meeting five neighbors.

Tokens. The mindless pipe embeds 492 pages and pays for discovery in all of them for all questions. The router from above is not embedded in the body. It reads fifty-six short subject lines for every three calls, and lives five pages of response. Even the table of contents is not sent complete. In the chorus of thousands of such documents, that difference is the line between a system that works and one that doesn't.

7. From a single document to a folder of documents

This topic resides in a single document. A multi-document folder is the same map one level higher: the top level is a list of files, each with a title and a one-line summary, and the next level is a list of the contents of each file. The routing step is the same: you select the relevant files at the top level, and then go down to their categories as this article goes down to controls. The movement is the same, only the number of levels increases. We cover that collection case in detail at Part IVstarting with Article 14 (corpus problem) and corpus tables (corpus_toc_df, corpus_index) they hold.

8. Conclusion

A long document is sent with its own map, and retrieval moves it up and down instead of scoring each page. The router reads one level of the table of contents at a time, one row per entry, and then down: eleven chapters Controlstwenty families to Access Controltwenty-two controls to AC-2 account managementfive of the 492 pages, and another 315 controls were never immediately available. It stands on a leaf, in a section small enough to read completely, or in a list that wants everything under a heading. That's a loop tied inside the retrieval, it's optional if the table of contents is shallow, and it stores the tokens and raises the precision at the same time, that's why it grows unchanged: the documents folder is the same map with one more level.

9. Sources and further study

• Document analysis: Article 5A and Article 5B: parse_pdf as well as toc_df this article will pass.
• Analyzing the questions: Article 6A, Article 6B, Article 6C: where the keywords and the shape of the question appear.
• Retrieval: Article 7A, Article 7B, Article 7C: a mix of the first section and the TOC router this article moves like a tree.
• Article 12 (listing): the whole section is read for questions that require everything. Article 13bis (loop engineering): the instruction of the loop this return loop is yours. Article 14 (corpus problem): the same route step one level up, over the documents folder.