I of this series with Angela Shi. This pitfalls article lists the failure modes we both kept seeing on production RAG systems, and that pushed us toward the four-brick contract in the first place.
I’ll admit something. Even when we work on this series, we dump big documents into ChatGPT. One PDF, one question, send, read the answer. The model is good, the vendor pays the token bill, and for a one-off that is the right path.
The series exists for the other case. In enterprise work the question is almost never about one document. A claims handler runs the same question across a broker’s full back-catalogue. A compliance team scans every contract in a portfolio. At that scale, dump-it-in-ChatGPT stops working, and it gets expensive fast.
What follows is the recap of the mistakes we keep seeing, one brick at a time. The fixes live in Part II.
1. Parsing: how the document loses its shape
Parsing fails when the team treats the document as text rather than as a structured object. Three patterns keep showing up: discarding tables and layout (Pitfall 1), dumping the entire document into the prompt (Pitfall 2), and chunking the document into fixed-size windows that ignore its structure (Pitfall 3). The fix is a structural parser that produces typed tables instead of strings or arbitrary windows.
A meta-version of these three runs through the rest of the article too. Teams skip parsing correctness and spend weeks tuning chunk size, reranker thresholds, top-K, and embedding choices, and never reach the precision they expected. Every lever they measure sits on top of the parser’s output: a parser that flattened the table on page 47 produces noise no chunker can recover, a parser that lost the column headers produces ambiguity no reranker can rank past. The literature does not help. The most-cited vendor writeup on RAG techniques runs 194 pages on chunking and zero on parsing. Fix parsing first. Retrieval tuning is for pipelines whose parser already preserves structure.
1.1 Pitfall 1: The PDF had a table. The parser returned a string.
The default reflex is to extract the PDF as a single blob of text and let the LLM sort it out. Modern parsers make this easy: one function call, one string back, done.
The cost shows up the first time a table arrives with grouped row labels. A claims contract has a benefit table where the same row name (Premium, Deductible) appears under two categories (Health, Dental). Flattened to text, the categories disappear into the token stream and the LLM sees Plan A Plan B Plan C Health Premium 100 200 300 Deductible 5 10 15 Dental Premium 50 80 120 Deductible 2 4 6. Ask “what is the Premium for Plan B?” and there are two valid answers: 200 (Health) or 80 (Dental). The flat string carries no grouping marker. The model picks one. It picks wrong some of the time, and there is no signal that says it picked wrong.
The same problem hits multi-column layouts (a contract page with a footnote sidebar), headers and footers (page number polluting every retrieval), and reading order on scanned PDFs. Each one is a different failure mode, but they share the same root: the parser threw away the structure the document carried.
The fix is a relational parser that produces typed tables (line_df, page_df, toc_df, …) instead of a flat string. Each line carries its bounding box, its page, its font, its section. Tables get their own grain. Downstream bricks read structure, not blobs.
1.2 Pitfall 2: Pay for 1200 pages on every question
A second mistake on the parsing side is one Angela and I do ourselves on small projects: skip parsing entirely and stuff the whole PDF into the chat. It is fast to write, it works for one document, and the vendor pays the token bill on the free tier.
On a real corpus, the same reflex becomes expensive in three steps. First, the PDF is no longer 12 pages, it is 1200. Second, the question is no longer one, it is 200 per day. Third, the team adds five more documents to the chat to “give the model more context”, and the per-question token count grows linearly. The bill goes from cents to thousands per month, and the answers get worse because the model has more haystack and the same needle.
The fix is to match the technique to the document and the question: when the answer fits on three pages, send three pages, not 1200. The same principle here: parsing once, retrieval scoped, generation on the smallest context that holds the answer.
Here is the bill on a realistic enterprise scenario: a 1200-page reinsurance contract that a compliance team queries 200 times a day. Two approaches, same question. The first extracts every line of text from the PDF and stuffs the result into every prompt. The second is the pipeline this series builds: parsing produces structured tables, retrieval returns the three relevant pages, generation reads only those.
On a 1200-page contract the dump pays roughly four hundred times the input cost of the scoped pipeline. The dump grows with the document and the question count; the pipeline grows only with the answer. Per contract, per year of two hundred questions a day, that is the difference between burning $131,000 and burning $329.
Prompt caching tilts the math without flipping it. Anthropic’s 90%-off cache reads and OpenAI’s 50%-off cached input both apply only on cache hits, evict on a TTL the team does not control, and bill full price on miss. At the 90% rate the dump still costs $13,140 a year on the same workload, forty times the scoped pipeline, and still growing with the document size, not the answer.
A hosted RAG (OpenAI’s file_search, AWS Knowledge Bases, similar) sits between the two: cheaper than the dump because the vendor chunks the document and retrieves what looks relevant, more opaque than the pipeline because the chunking, the embedding model, and the ranking are not yours to inspect. It is the convenient middle ground for prototyping a single document. It is rarely the answer at enterprise scale, where audit and reproducibility matter as much as the bill.
For one contract, the absolute number is six figures. For ten thousand contracts in a corporate portfolio, the same ratio decides whether the annual cost line stays in the tens of thousands or jumps into the millions.
1.3 Pitfall 3: Tuning chunk_size. The PDF had structure.
The third parsing mistake is the assumption that a PDF is a string. The team imports pdfplumber, pypdf, or PyMuPDF, calls the function named extract_text or get_text, pipes the result into a RecursiveCharacterTextSplitter, and spends the next month tuning chunk_size and chunk_overlap to push retrieval precision up two points. The PDF carried structure. The convenience API erased it. The knob the team is turning is downstream of where the loss happened.
A 200-page contract embeds a clickable bookmark TOC (the outline every reader displays in the sidebar), section headings rendered in distinct font sizes (24pt for parts, 18pt for sections, 14pt for sub-sections, the same cues a reader’s eye uses to scan), and tables stored as cell-level bounding boxes a structural parser can reconstruct. The structure is right there in the PDF object model. extract_text() skips over it and hands the splitter an undifferentiated stream. The splitter then cuts at chunk_size=500 because that is the knob the team has been tuning, on top of input the PDF’s own typography could have anchored for free.
The cost is precision. A chunk that ends mid-table contains half a row. A chunk that starts mid-section carries no heading to anchor the answer’s context. The answer the LLM produces from those chunks is technically grounded in the corpus, but the grounding is on cropped fragments rather than meaningful units. Retrieval has nothing to filter on, since every chunk looks roughly the same. Generation has nothing to cite cleanly, since citations point at arbitrary windows. The audit chain shows lines like “chunk 1142 of 10,000” with no readable meaning.
Markdown-aware and section-aware splitters fix the symptom and leave the upstream problem in place. They chunk on the heading text they guess from the flat string, but they cannot rebuild the bounding boxes, the font hierarchy, or the table grid that extract_text() already threw away. The chunker is fighting on cropped input.
The fix is the structural parser the rest of the article keeps pointing at. The parser keeps the PDF’s typography (line_df carries bbox, font, page, section path), keeps the table grid (the table extractor produces typed cells, not strings), keeps the TOC (toc_df from the PDF bookmarks plus font-size detection). The downstream bricks read the structure. No 500-character window ever crosses a section boundary, because there is no window. There is structure.
2. Question parsing: how you ignore the user
Question parsing fails when the team treats the user’s natural-language question as if it were a query. Two reflexes keep coming back: passing the raw string straight to retrieval (Pitfall 4), and stopping at keyword extraction when the question carried answer shape, scope, and format constraints too (Pitfall 5). The fix is a typed ParsedQuestion that carries all of it.
2.1 Pitfall 4: “Just embed the question.”
The cheapest wiring in any RAG framework is to take what the user typed, embed it, send it to retrieval. Type the question, call the API, ship. The question carries many things: a scope, an expected answer shape, a format, sometimes a condition, sometimes a negation, sometimes a reference to an earlier turn, often an implicit constraint on the document the user has in mind. The embedding flattens all of them into one vector that mostly captures the content words. The bricks downstream consume whatever survived the flattening, which is usually not what the user meant.
Real questions take every shape. A short sample of what shows up in production, with the reason each one breaks naive embedding:
- A terse, structured ask. “Cancellation period plan B in days.” Five tokens, three constraints: a scope filter (plan B), an answer type (a duration), a format (in days). The embedding flattens all three into one vector retrieval scores against the corpus.
- A negation. “What is NOT covered by this policy?” The embedding barely encodes NOT as a function word, so retrieval returns the chunks most similar to the rest of the sentence, which describe what IS covered. Generation paraphrases the opposite of what was asked.
- Nested conditions plus a question. “For plan B, assuming a one-year contract, what is the cancellation period if I cancel after six months?” The conditions belong to retrieval scope, the question belongs to generation framing. The embedding mixes them; the wrong brick consumes the wrong field.
- A multi-part comparison. “Exclusions or deductible, which one matters more?” Retrieval returns chunks about both; generation gets no signal that the user wants a comparison and not a list.
- An elided reference. “And what about Plan C?” Five words, no context. The embedding has nothing to anchor on.
The list does not close. Each new corpus, each new audience, each new product brings its own question shapes. Some are terse, some explain at length, some carry operators, some lean on the previous turn. The question parser is the brick that absorbs the variety so the bricks downstream see a typed object they can route on. Without it, every new shape becomes a new silent failure.
The shortcut is the same in each case. The question carries structure (constraints, operators, scope, intent, references). The embedding flattens that string into one vector. Retrieval acts on what survives, generation reads what retrieval found, and the user gets back something that may or may not match what they asked.
The cost is contradiction the pipeline cannot detect. The retrieved chunks look relevant. The model writes a fluent paragraph. The user reads a confident answer about coverage when they asked about exclusions, with no flag, no warning, no signal that the question’s operator was lost on the way in.
The common counter is to wedge in a small LLM call that returns a JSON dict: intent, scope, keywords. That solves the no structure problem but not the no contract problem. The dict keys drift between prompts (one call returns scope, the next returns scope_filter), retrieval reads one key, generation reads the other, and the silent miss reaches the user. A typed ParsedQuestion Pydantic schema turns the drift into a parse-time error the audit log catches. The win is not the JSON; it is the validation.
It also blocks every downstream improvement. You cannot route a question to a specialised pipeline if you do not know what kind of question it is. You cannot ask for the user’s confirmation on an ambiguous term if you have not flagged the ambiguity. You cannot decompose a multi-part question if you have not recognised it has multiple parts.
The fix is a typed ParsedQuestion object: the question parsing brick turns the raw string into a structured object with keywords, answer shape, scope filters, an execution plan. The string is the input; everything downstream consumes the typed object.
2.2 Pitfall 5: “Just use HyDE.” Or trust the embedding.
The second question-parsing mistake is the assumption that the brick does not need to exist. The user types “what’s the cancellation period for plan B?”. The pipeline passes the string to an embedding model, retrieves the top-K chunks by cosine, hands them to generation. There is no question parser. Modern devs distrust hand-rolled keyword extraction (with good reason: brittle lists, drift, language-specific edge cases) and reach for the embedding instead, which appears to absorb the question’s meaning for free.
Embeddings absorb something. They produce a dense vector close to passages that read like the question. They do not produce the answer shape, the scope, the format constraint, or the implicit “in this document” clause. Those carry no embedding signal until the pipeline writes them down somewhere typed.
A common workaround the field reaches for at this point is HyDE (Hypothetical Document Embeddings): the LLM generates a hypothetical answer to the question, the pipeline embeds that hypothetical, retrieval scores corpus chunks against it instead of against the question. It works on benchmarks, and devs reach for it as the smart escape from the embedding-only trap. The reason it works rarely gets stated plainly: the hypothetical answer contains the keywords a real answer would contain, and those latent keywords are what the embedding picks up. HyDE is LLM-driven keyword extraction in disguise, one extra generation per query, no expert validation, no audit. When it underperforms, the reflex is to reach for a stronger model. The deterministic version of the same insight is to ask the domain expert for the concept vocabulary and store it once. In enterprise the answer worth shipping is the one a domain expert would validate, not the one a more capable model happens to imagine.
The format constraint “in days” is the sharpest case. Encoded into the question vector, the “days” signal biases the top-k toward chunks about “response time within 30 days” or “Day 1 of the policy”, both pure noise for a question about cancellation. The constraint belongs in the generation brief, not in the retrieval query. Pipelines that skip question parsing send the same encoded vector to retrieval and the same raw string to generation, and the wrong brick consumes the wrong field.
The fix is not a smarter embedding. The fix is a question parser that produces a typed object with answer shape, scope, format, and decomposition as separate fields, each routed to the brick that consumes it. The keyword case becomes one field of that object, validated against an expert dictionary so the term premium maps to prime, cotisation, price without the dev maintaining the list by hand. Article 6 develops the parser and the two typed briefs that come out the other side, one for retrieval and one for generation.
3. Retrieval: the vector DB reflex and its blind spots
Retrieval fails when “just embed it and rank by cosine” becomes the only tool in the box. Three habits cause it: treating RAG as a synonym for vector DB (Pitfall 6), treating the chunk as the only granularity when the answer is one line inside a larger passage (Pitfall 7), and stopping at references to elsewhere in the document (Pitfall 8). The fixes are hybrid retrieval, two granularities returned together, and a reference-resolution loop.
3.1 Pitfall 6: “Just use a vector DB”
This is the biggest mistake we see, and the most expensive to undo because it dictates the whole infrastructure stack. The pattern is fixed: chunk the corpus, embed every chunk, embed the question, return the top-k chunks by cosine similarity. Done.
The cost shows up on every question where a keyword would have helped more than a vector. Acronyms (“RC” in insurance, “SCR” in solvency), product codes, numeric ranges, rare names, legal references like Section 4.2(a)(iii). Embeddings flatten these into a dense vector and lose the discreteness. The retrieval brick returns a passage about something similar instead of the passage that contains the term.
More generally, embeddings work when the question is paraphrased prose against paraphrased prose. They struggle when the question is a token: a code, a number, a regex-shaped pattern, a precise reference. A small anecdote that stuck with me. A few months ago I used a chat assistant inside a copywriting tool to find a specific phrase in a long document I had pasted in. At some point the assistant tried to find the phrase with a regular expression. The regex came back empty. I went looking: the original PDF had a typographic character (a curly quote, I think) that my copy-paste had replaced with a straight quote. The model had been right to reach for a regex. The token match was the fundamental operation. The pipeline around it just could not handle a one-character difference.
Anthropic’s tooling pushes this further: when an agent needs to find a span, it reaches for grep-like primitives before it reaches for an embedding. That is the direction the field is moving in, slowly, because conversations are made of words, and words match best on tokens, not on vectors.
The deeper issue is cultural. The name Retrieval Augmented Generation says nothing about vectors. It says retrieval, which is a fifty-year-old field with many techniques. Yet when we talk to developers building RAG systems, almost every conversation goes the same way: “yes, we use a vector database for retrieval.” It is treated as the default, not as one choice among many.
Angela and I even argued about coining a different name for what we build. ROG, for Retrieval Only Generation, because in enterprise the retrieval is the work and the generation is the wrapper around it. The historical RAG definition pointed the other way: a parametric model generates, retrieval augments it. We kept “RAG” in the end because that is how the work is searched and known, and we did not want to invent another acronym just to make a point. But it is worth saying plainly: there is no pure vector search anywhere in this series. Embeddings appear, but as a fallback.
The fix is hybrid retrieval by default: keyword detectors (exact, free, deterministic) running in parallel with embedding detectors, with an LLM arbiter at the end that ranks the aggregated candidates with reasons. The popular shortcut “RAG equals vector DB” is the single biggest source of expensive failures we have seen at scale.
3.2 Pitfall 7: The chunk is right. The pipeline stopped there.
The second retrieval mistake is subtler and shows up only when you try to ground an answer in the source. The pipeline retrieves a chunk, hands it to the LLM, the LLM returns an answer. Where in the chunk was the answer? Nobody asks, because the chunk was the unit.
This breaks every downstream feature that depends on knowing where the answer is. Highlighting on the source PDF. Citations with line numbers. A compliance trail. The system can say “the cancellation period is 30 days” but cannot point to the line it read.
The instinct is to recover the location after the fact, by string-matching the LLM’s quote against the source. It fails the moment the model paraphrases (which it does whenever the quote runs more than a few tokens) and the citation points at a near-miss line. The location has to be computed on the way in, not retrofitted from the output.
The chunk is also the wrong unit on the other side of the fix: the amount of surrounding text the LLM needs depends on the question. Take “what is the date of the events?” on a 200-page incident report. The keyword date of events hits one line; that line carries the date. Two lines around it are enough to ground the answer. Returning the chunk that contains the line, let alone the whole chapter, buries the date in noise the model has to wade through. A question about the contract’s cancellation policy asks for a different size: a paragraph or two, because the policy is built from several conditions that interact. Same chunker, same document, different right answer for how much surrounding text to keep.
The fix is not a better chunker. The fix is to retrieve at two granularities at once: one precise enough to highlight on the source (the line where the keyword hit), one sized to what the question needs (two lines for a date, a paragraph for a policy). Article 7 builds the retrieval brick around this split, and gives the two scopes the names Angela and I argued about for weeks before settling on.
3.3 Pitfall 8: “See Section 4.2” and never look
The third retrieval mistake shows up the first time the document refers to itself. The retrieved chunk reads “the exclusions are listed in Section 4.2”, and the pipeline stops. Retrieval found the chunk that mentions the exclusions; it did not follow the pointer. Generation gets the chunk, sees the reference, and has two equally bad options: invent the contents of Section 4.2 from its pre-training priors, or refuse with “the document does not specify”. The document does specify. The pipeline just did not look.
The cost is a silent breach of the audit chain. The user is told the system grounds in the corpus, but when references are unresolved the answer is reasoning from priors. That is exactly what the four-brick contract was meant to prevent. Worse, this failure is uncatchable from the outside: the answer reads fluent either way, and the cited Span covers the chunk that mentioned Section 4.2, not the section itself. A reviewer who clicks the citation hits a sentence that says “see Section 4.2” and a confident answer next to it. The chain of evidence stops one hop short.
Agentic RAG handles this the agentic way: the LLM calls a fetch_section tool when it sees a reference. It works, at a cost the team often does not see. Every reference resolution becomes a non-deterministic loop, the audit trail forks per agent step, the per-question cost grows with the depth of the reference chain.
The deterministic alternative is a two-pass loop with a typed trigger. The first pass produces a structured answer that flags the pending reference instead of fabricating around it. The orchestrator follows the reference to the cited section, runs retrieval on the right pages, and the second pass comes back grounded on Section 4.2 itself. Article 11 develops the trigger field on the answer schema, the resolver that maps a reference to the right pages, and the orchestrator pass that wires them together.
4. Generation: where the audit chain dies
Generation fails when the brick is treated as the API call that returns a string. Two patterns repeat: shipping the raw LLM string with no flag, no schema, no audit (Pitfall 9), and trusting the LLM’s “not found” claim without an external proof of absence (Pitfall 10). The fix is a typed answer wired to programmatic checks the model has no access to.
4.1 Pitfall 9: No flag, no schema, no audit. Just text.
The retrieved passage goes into a prompt, the LLM returns a string, the system passes the string to the user. Production RAGs ship like this every day. The brick is the API call.
The cost is that you have no signal the answer is reliable. The model returns a fluent sentence whether the passage contained the answer or not. There is no answer_found flag, no quote of the supporting span. When the model invents a number, the system has no signal to catch it before it reaches the user.
Structured outputs (OpenAI’s response_format, Anthropic’s tool use, Pydantic AI) close part of this. A typed response with answer_found and quote fields says what the model thinks it grounded on. What they do not close is the model rating itself. “confidence”: 0.95 arrives with the same conviction whether the quote is real or invented. The same brick that read the passage is the one rating the answer.
The escalation is programmatic verification the model has no access to. A regex check that the cited quote appears verbatim in the cited span. A set-coverage check on enumeration answers (the question asks for four exclusions, the schema returns four, every entry maps to a distinct chunk in the passage). A type check on the answer value (the answer should be a Duration, the model returned “around a month”). Each check is a verdict the dispatcher routes on. The model fills the schema; the verifier decides whether to ship.
A second cost falls out of this: downstream tools cannot react to the model’s state. The dispatcher cannot trigger a refetch because nothing told it the retrieval was incomplete. The audit log cannot reconstruct the decision because the raw text carries no provenance. The pipeline becomes one-shot: either the answer is good, or you re-run the whole thing.
The fix is the typed answer schema plus the verifier that closes the loop. Article 8 develops the schema, the dispatcher that picks the right shape per answer type, and the verifier that closes the loop.
4.2 Pitfall 10: “Not in the chunks” is not “not in the corpus.”
The second generation mistake is trusting the LLM when it says “not found.” Retrieval is rarely empty: with embeddings, cosine top-k always returns something, so the LLM gets a handful of chunks and decides whether the answer is in them. When it says no, the pipeline ships answer_found=False to the user. The system has just delegated the verification to the same brick that read the chunks.
The LLM’s “not found” means “not in these chunks.” It does not mean “not in this corpus.” The model saw the top-k passages, not the document and not the rest of the archive. Two failure modes hide behind a confident refusal: the answer was there in the top-k and the model missed it (LLM mistake), or the answer was elsewhere in the corpus and retrieval missed it (retrieval miss). The user reads “the document does not specify” and assumes the corpus has been checked. It has not.
The fix is to back the “not found” with a deterministic absence proof. The expert keyword dictionary, every term and every curated synonym for the question’s concept, runs as a literal substring search across the full corpus, not against the retrieved chunks. Zero matches and the system says “not in this corpus” with a defensible audit trail. At least one match but the LLM still said no, retrieval missed and the orchestrator triggers a second pass on the pages where the keywords appeared. Keywords prove absence; embeddings cannot. The fix uses bricks the article has already pointed at: the expert dictionary from Pitfall 5 and the keyword retrieval from Pitfall 6.
5. What you should expect from Part II
Each of the ten mistakes above is a structural choice the team made early, before they had a contract that named the brick. The contracts that make these failures impossible are developed in the rest of the series: a relational parser that keeps the document’s structure, a typed question that carries every constraint downstream, hybrid retrieval at two granularities, a reference-resolution loop, and a typed answer wired to programmatic checks the model has no access to. Each one is the brick the four-brick split required.
The same vector-reflex problem shows up in a different shape in agentic systems. When an agent has to pick a tool from a catalog of hundreds, the default reflex is again to embed the tool descriptions and rank by similarity. The result is the same: imprecise on codes, blind to the difference between “reads” and “writes”, opaque to audit. The fix has the same shape: words first, embeddings as fallback, audit on every choice.
If you find yourself nodding through this article because your own pipeline does most of these, that is the most useful kind of nodding. The fixes are developed in the articles that follow.
6. Sources and further reading
Other articles in the series:
External references:
- Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels, ACL 2023. The original HyDE paper. Pitfall 5 explains why the technique works (the LLM-generated hypothetical contains the keywords a real answer would) and argues the deterministic equivalent is the expert dictionary.
- Anthropic, Introducing Contextual Retrieval, September 2024. The LLM-generated context-prepending approach to chunking, adjacent to Pitfall 3. The series solves the decontextualization problem with structured metadata rather than LLM-generated blurbs.
- Anthropic, Prompt caching with Claude. The cost lever that tilts Pitfall 2’s math at the 90%-off cache-read rate without flipping it.
- Pinecone Learn, Chunking Strategies for LLM Applications. The field’s reference chunking survey with the precision-vs-richness matrix Pitfall 3 argues sits inside a frame that
extract_text()already corrupted. - LlamaIndex, Building Performant RAG Applications for Production. Names the decoupled chunks for retrieval vs synthesis pattern, the same insight Pitfall 7 frames as anchor vs context.
- Liu, Instructor: Structured outputs for LLMs. The Pydantic-typed-output library and the schema-as-contract argument. Direct support for the Pydantic-vs-dict pushback in Pitfall 4 and the typed answer in Pitfall 9.
- Zaharia, Khattab et al., The Shift from Models to Compound AI Systems, BAIR 2024. The academic frame for the four-bricks architecture this article assumes.
