Long Context Isn’t Free — I Built a Safe Prompt-Pruning Layer That Makes LLM Systems Work

I’ve worked on, conversation state tends to grow quickly over time. It’s common to resend large portions of the history on each turn—including older tool outputs, repeated RAG retrievals, and context that’s no longer relevant. As this accumulates, prompts can become significantly larger, which may increase inference cost and latency, and in some cases affect reasoning performance.

I built a deterministic pipeline that prunes this redundant state before the prompt ever reaches the model. The version I implemented avoids LLM calls, embeddings, and external dependencies. Relying strictly on standard library components ensures every pruning decision remains fully deterministic and reproducible.

State tracking executes in three distinct passes: Expired Context Elimination, Duplicate Context Elimination, and Dependency Restoration. The third pass is what helps make the first two safer in practice. It ensures that nothing a later message depends on is accidentally removed.

While building it, I ran into two bugs that changed the design. My first benchmark corpus used a fixed number of duplicates and stale tool calls, which made reduction percentages shrink as conversations grew. That didn’t reflect what I would expect from real-world behavior.

My dependency restoration logic also went completely untested at first because my synthetic data never created a case where a required message was actually removed. Both issues are covered here, along with how fixing them changed the results.

After correcting the pipeline, I benchmarked it across three workloads: plain chat, a RAG assistant, and a tool-heavy agent. Each was tested at five conversation sizes, for a total of 15 configurations on two different machines. Across these runs, all labeled required facts were preserved. The system also reached a stable fixed point after a single pass, which means pruning an already pruned prompt produces no further changes.

Token reduction depends on the workload. It is about 2 to 4 percent for plain chat, 27 to 32 percent for a RAG assistant, and 33 to 34 percent for a tool-heavy agent. Even at 2,000 turns and 131,000 tokens, preprocessing stayed under 50 milliseconds.

Full code, all 35 tests, and the raw terminal output are included below so you can run the pipeline yourself.

Complete code:

Every prompt I ship keeps getting heavier

I kept seeing the exact same pattern pop up across every long-running agent I built. A conversation starts out perfectly clean. Fifty turns later, it’s an absolute mess.

By that fiftieth turn, the prompt payload you’re shipping out on every single request includes the system prompt, the entire chat history, four tool outputs (two of which are completely stale because the tool ran twice), six retrieved chunks (three are near-duplicates because the user circled back to an old topic), a SQL result from twenty turns ago that’s just sitting there, and a single user preference stated once and never brought up again.

None of this stuff is technically wrong. Every single piece made sense the exact moment it was injected. The real issue is that nothing ever gets cleared out. The prompt turns into an append-only log of every historical event, and you’re dumping the whole thing onto the model, every single turn, indefinitely.

That’s not just a storage problem. Reasoning performance measurably degrades as input length grows, even when the added content is irrelevant to the task [1], and models are worse at using information buried in the middle of a long context than information near the edges [2].

The immediate knee-jerk reaction when you see this bloating is just basic truncation. Slice the last N messages and drop everything else. It’s literally one line of code. The catch is it silently breaks your conversation chains in ways you won’t even realize until it blows up on a real user:

Turn 3:   User: My preferred output format is CSV.
Turn 4:   Assistant: Got it.
...
Turn 47:  User: Export the results.

If your window is capped at the last 20 turns, turn 3 disappears by turn 47. The assistant loses the context that the user asked for a CSV. That data was not stale; it was a hard dependency, and simple positional truncation cannot differentiate between old, expendable context and old context that a later turn still relies on.

That is the exact design constraint this project addresses. Any mechanism that removes redundant state must distinguish between those two categories. Recency alone is insufficient.

Borrowing an idea from operating systems

Here is the reframe I built the rest of this around: an operating system is constantly deciding which pages stay resident in RAM and which get evicted. A long-running LLM conversation has the exact same problem, except nothing plays the role of the memory manager. Context just accumulates forever because no process owns the job of deciding what stops earning its place in the prompt.

The Prompt Pruner in this article is that missing piece.

Every piece of conversation state (a user turn, an assistant turn, a tool output, a retrieved chunk) is a Message object with a role, a turn number, and a bit of bookkeeping metadata:

@dataclass
class Message:
    id: str
    role: str
    content: str
    turn: int
    tool_call_key: Optional[str] = None
    expires_after_turn: Optional[int] = None
    defines_keys: list = field(default_factory=list)

Three passes run over that list in order. Each one is a pure function: messages in, a filtered list out. No model is involved anywhere in the pruning step itself.

Why there’s no model inside the pruner itself

I could have used an embedding model to score message relevance, which would have made this fuzzier and a lot easier to write. I didn’t, and it is not for the sake of purity.

Once a pruning decision depends on a model’s judgment, you lose the ability to reason about what a conversation will look like on the next turn. The same input no longer guarantees the same output. A pruning layer meant to make a production system more predictable should not be the least predictable part of it. Everything here runs on dataclasses, regex, and dict lookups, which is exactly the toolset this problem needs.

Pass 1: Expired Context Elimination

If a tool gets called more than once under the same key (the same search query, the same SQL lookup, the same file read), only the newest result is still trustworthy. Everything earlier under that key is expired.

def _pass1_expired_context_elimination(self, messages):
    last_occurrence = {}
    for m in messages:
        if m.tool_call_key:
            last_occurrence[m.tool_call_key] = m.id

    kept, removed = [], []
    for m in messages:
        if m.tool_call_key and last_occurrence[m.tool_call_key] != m.id:
            removed.append(m)
        else:
            kept.append(m)
    return kept, removed

Pass 2: Duplicate Context Elimination

Retrieval pipelines constantly pull up identical or near-duplicate passages, especially when a user loops back to an earlier topic. This pass normalizes the whitespace and casing, keeps only the first occurrence, and drops every duplicate after it.

def _pass2_duplicate_context_elimination(self, messages):
    seen, kept, removed = {}, [], []
    for m in messages:
        if m.role == ROLE_RETRIEVED_DOC:
            norm = " ".join(m.content.lower().split())
            if norm in seen:
                removed.append(m)
                continue
            seen[norm] = m.id
        kept.append(m)
    return kept, removed

Pass 3: Dependency Restoration (and the bug that made me build it properly)

This pass is the reason the first two are safe to run at all. If Pass 1 or Pass 2 drops a message that happens to be the only place a still-referenced fact got defined, this pass catches it and puts it back.

The mechanism is intentionally simple: a message marks itself with a literal DEFINE: tag, and a later message references it using REF:. If a REF survives into the final kept set but its matching DEFINE got dropped upstream, Dependency Restoration restores that missing message to the list.

def _pass3_dependency_restoration(self, all_messages, kept_messages, removed_messages):
    kept_ids = {m.id for m in kept_messages}
    by_id = {m.id: m for m in all_messages}

    key_definer = {}
    for m in all_messages:
        for key in m.defines_keys:
            key_definer[key] = m.id

    referenced_keys = set()
    for m in kept_messages:
        referenced_keys.update(m.references())

    restored = []
    for key in referenced_keys:
        definer_id = key_definer.get(key)
        if definer_id and definer_id not in kept_ids:
            restored_msg = by_id[definer_id]
            kept_messages.append(restored_msg)
            kept_ids.add(definer_id)
            restored.append(restored_msg)

    kept_messages.sort(key=lambda m: (m.turn, m.id))
    return kept_messages, restored

Here is the bug. I ran my first full benchmark across all three workloads and five sizes, and every single row printed “Restored (deps): 0.” Every one. My first reaction was that this looked great—a perfect safety record. It wasn’t. It meant Pass 3 had never actually restored a single message on any run at any size.

I went back into my corpus generator to find out why, and the answer was embarrassing once I saw it. My synthetic conversations only ever attached DEFINE markers to plain user messages, while Pass 1 and Pass 2 only ever remove tool outputs and retrieved documents. The two categories never overlapped. Dependency Restoration was sitting in the codebase, fully written, but completely untested by my own benchmark because nothing I generated ever gave it a reason to fire.

The fix was to let some tool outputs also define a dependency, the same way a real “get user settings” tool call might surface a fact the conversation depends on later, even though that exact tool call could get superseded and marked expired by Pass 1. Once I added that, the numbers changed immediately: 2 restorations at the smallest tool-agent conversation, climbing to 127 at the largest. Required facts stayed at 100 percent preserved the entire time, but now that number actually meant something, because the pass being tested was capable of failing and didn’t.

I want to be direct about the limitation that’s still here even after the fix: dependency detection is literal identifier matching, not understanding. It catches an exact REF matching an exact DEFINE. It will not catch a user paraphrasing, asking “what format did I mention earlier” with no matching tag. A semantic dependency resolver would need an embedding model or an LLM call, and that is explicitly outside what this deterministic pipeline does. I’d rather ship a narrower guarantee I can prove than a broader one I can’t.

Design goals, and why each one exists

Deterministic. Same input always yields the same output. Keeping the model out of the loop eliminates run-to-run variance and any risk of hallucinating what should remain in the prompt.

Dependency-safe. It never silently drops a fact that a later turn still needs. Positional truncation completely lacks this property, which makes it non-negotiable here. A pruner saving 40 percent of tokens that occasionally breaks a conversation is a worse trade-off than one saving 4 percent that never breaks anything.

Idempotent. Running the pruner a second time does nothing. If that is not true, you cannot safely re-prune on every single turn of a growing conversation without risking compounding drift.

Lightweight. The pruning step should never become the exact bottleneck it was built to eliminate.

The benchmark: three workloads, and a mistake I almost shipped

My first version of the synthetic corpus generator picked a fixed number of duplicate passages and repeated tool calls (six tool calls and eight duplicates) regardless of how long the conversation was. I ran it at five sizes and the reduction percentage went down as the conversation got longer: 9.9 percent at 50 turns, dropping to 0.3 percent at 2,000 turns. That runs backward from actual production traffic, and it is not defensible. If the amount of waste in a benchmark is just a constant I hand-picked, anyone reading it is right to ask whether I built the benchmark to prove the algorithm works.

So I threw that generator out and rebuilt it around an explicit workload model instead: retrieval-per-turn, retrieval overlap rate, tool call rate, and tool repetition rate. All of these were fixed before running a single benchmark, based on what seemed like plausible production behavior, rather than adjusted afterward to hit a number I liked. Three workloads came out of that:

Normal chat. No retrieval, occasional tool calls, mostly ordinary back-and-forth.

RAG assistant. Retrieves documents on every turn, with a real chance any given passage overlaps something retrieved recently, because users revisit topics and retrieval re-surfaces the same chunks.

Tool agent. Frequent calls across five tool types (search, SQL, calculator, filesystem, web fetch), high repetition rate, modeling something that re-plans and re-queries constantly.

Every synthetic corpus also ships with ground truth: every message some later message depends on gets labeled required up front. So “did pruning keep everything it needed to” is a check against known labels, not a guess.

Here is the complete output. All 15 configurations, not a slice of it:

Workload	Turns	Tokens before	Tokens after	Reduction	Facts kept	Idempotent	Overhead
Normal chat	50	1,175	1,153	1.87%	Yes	Yes	0.17 ms
Normal chat	200	4,820	4,629	3.96%	Yes	Yes	0.79 ms
Normal chat	500	12,078	11,660	3.46%	Yes	Yes	1.82 ms
Normal chat	1000	24,379	23,381	4.09%	Yes	Yes	3.79 ms
Normal chat	2000	48,241	46,514	3.58%	Yes	Yes	8.27 ms
RAG assistant	50	3,494	2,551	26.99%	Yes	Yes	0.60 ms
RAG assistant	200	14,009	9,599	31.48%	Yes	Yes	2.18 ms
RAG assistant	500	35,347	24,133	31.73%	Yes	Yes	6.19 ms
RAG assistant	1000	70,358	47,950	31.85%	Yes	Yes	11.89 ms
RAG assistant	2000	140,766	95,087	32.45%	Yes	Yes	28.95 ms
Tool agent	50	3,279	2,176	33.64%	Yes	Yes	0.47 ms
Tool agent	200	12,955	8,585	33.73%	Yes	Yes	2.04 ms
Tool agent	500	32,412	21,677	33.12%	Yes	Yes	6.46 ms
Tool agent	1000	65,366	43,351	33.68%	Yes	Yes	15.02 ms
Tool agent	2000	131,591	87,625	33.41%	Yes	Yes	43.04 ms

Every single row says Facts kept: Yes and Idempotent: Yes. Not most rows. All fifteen.

The pattern by workload makes sense once you look at where the waste actually comes from. Normal chat barely retrieves anything and rarely repeats a tool call, so there is almost nothing for Pass 1 or Pass 2 to catch; it stays around 4 percent no matter how long the conversation runs. The RAG assistant retrieves every turn with real overlap, so Duplicate Context Elimination carries most of the weight, landing around 32 percent. The Tool agent combines both problems (frequent tool repetition and retrieval overlap) and hits the highest reduction at 33 to 34 percent.

Different workloads accumulate different kinds of waste. The pruner responds directly to whatever waste is sitting in front of it, rather than producing a flat number that would suggest the benchmark was reverse-engineered to hit a target.

If I only got to keep one result from this whole benchmark, it is the safety property: 15 out of 15 configurations preserved 100 percent of required facts. Zero missing dependencies, across three structurally different workloads and a 40x range in conversation length. Truncation by position cannot offer that. For me, this was the most important signal when evaluating whether the approach might be usable in production.

That number is also computed, not hand-counted. The benchmark script itself tallies how many of the 15 (workload, size) pairs preserved every required fact and how many reached the idempotent fixed point, printing an aggregate summary block after the 15 detailed runs:

============================================================
SUMMARY
============================================================
Configurations run:               15 (3 workloads x 5 sizes)
Required facts preserved:         15/15
Reached fixed point (idempotent): 15/15

Workload            Token reduction range    Facts preserved   Idempotent
Normal chat          1.9-4.1%                YES               YES
RAG assistant        27.0-32.5%               YES               YES
Tool agent           33.1-33.7%               YES               YES

I wrote this check because I caught myself hand-counting table rows for an early draft. That metric belongs in the script output, not my own eyes squinting at a terminal log. If a test run ever hits anything less than 15/15, it means I broke the pruner and have a regression to hunt down, not a typo to edit in the post.

I left one specific metric out of the final numbers: tokens removed per millisecond of execution overhead. The code computes it—it peaks at around 4,000 tokens per millisecond on small tool-agent runs and lands between 900 and 1,700 tokens at larger scales. It stays in the codebase as an internal field because it helps track scaling costs, but it belongs outside the main table. Readers cannot act on it the way they can with raw token count, reduction percentage, or millisecond overhead. Three direct metrics showing a clear trade-off are better than a fourth that acts as a novelty.

The idempotence result is the part I liked tracking the most. Proving prune(prune(x)) == prune(x) means the pipeline hits a stable fixed point on the first pass. Running it again on an already-pruned prompt changes nothing:

That rules out oscillation. It also rules out cumulative shrinkage across turns if you re-pruner on every single message of a growing conversation, which is exactly how this runs in production.

Reproducing it on a second machine

I ran the full benchmark on a Linux container running Python 3.12.3, then again on Windows 11 in PyCharm using Python 3.12 in a separate venv. Every token count and message count matched exactly across both machines. Only the millisecond timings moved, which is standard for different hardware, and even those stayed under 50 milliseconds for the largest, most tool-heavy conversation on both setups.

One thing I noticed while comparing the two runs and want to be straight about: prompt-build time (the time to serialize the final message list into a string) occasionally came out slower after pruning than before on the Normal Chat workload. Not by much—under a millisecond—but the direction was backward.

My read is that Normal Chat only removes 2 to 4 percent of messages, so the before and after list sizes are nearly identical. At sub-two-millisecond operations, system jitter and garbage collection pauses easily swamp the actual signal. Adding a warm-up call and using the median of 30 runs mostly stabilized the metric, but the anomaly still pops up on that workload. It is noise at a scale where the delta is smaller than the measurement error, so I left it raw rather than massaging the data.

What this benchmark deliberately doesn’t measure

This benchmark isolates token reduction and pruning overhead because those are the metrics the pipeline actually controls. End-to-end LLM latency is a completely separate variable. It depends on provider architecture, batching, regional caching, and network conditions that this project cannot see. Trying to convert a token reduction percentage directly into a latency delta means inventing an arbitrary conversion constant.

The baseline reality is simple: cutting 30 to 34 percent of input tokens means the model does less work per call. In general, inference cost and latency tend to increase with prompt size [4], making this a useful cost lever. But a true latency number requires a live validation pass against your specific provider. Publishing a generic latency figure here would mean making claims about infrastructure I do not control, rather than evaluating the pruning layer itself.

How this fits into a real agent loop

The pruner lives in exactly one spot: right after the conversation history is pulled together for a turn, and just before it gets serialized into the final prompt string.

from prompt_pruning import PromptPruner, PromptBuilder

pruner = PromptPruner()
builder = PromptBuilder()

def handle_turn(conversation_state, new_user_message):
    conversation_state.append(new_user_message)

    pruned_messages, report = pruner.prune(conversation_state)
    prompt = builder.build(pruned_messages)
    response = call_llm(prompt)

    conversation_state.append(make_assistant_message(response))
    return response

Because the pipeline is idempotent, calling prune() every turn of a growing conversation is safe. Running the pruner ten times on a history pruned nine times yields the exact same result as a clean run from scratch. This makes “run it every turn” a safe default, eliminating the need to track state or reason about previous passes.

The only integration decision left is how REF and DEFINE tags get attached to messages. Here they are literal markers inside the message content, which is the simplest mechanism for a prototype. A production system would likely attach them as structured metadata on the message object so the tags never leak into the raw text the model reads. Pass 3’s logic remains the same either way. An upstream process still has to determine what counts as a dependency worth tagging, because Pass 3 can only restore what it is explicitly told to track.

What this doesn’t cover

Dependency detection is literal, not semantic. If a reference is paraphrased and lacks a matching tag, the script will miss it.

These workloads are also completely synthetic. I chose the three parameter sets based on plausible production behavior, not real telemetry. If you have production logs, regenerating these three workload categories from actual usage is the obvious next step. The numbers will shift depending on what your actual traffic looks like.

This pipeline omits semantic compression, embeddings, and LLM-scored pruning. Those are valid, alternative approaches, but avoiding them keeps this implementation fully deterministic and dependency-free. LLMLingua is a prime example of the learned alternative; it uses a small language model to score and drop tokens, reaching much higher compression ratios than this script [3]. Choosing between them is a direct trade-off: you exchange determinism and zero-dependency execution for tighter compression.

The token counts are also approximations. The script uses a whitespace and punctuation-boundary heuristic instead of a production subword tokenizer like tiktoken. Because the heuristic runs consistently before and after pruning, the relative reduction percentages remain accurate, even if the absolute numbers do not perfectly match an official tokenizer.

Finally, there is no direct latency measurement for the infrastructure reasons detailed earlier.

Where I’d take this next

Two extensions make sense here rather than expanding the existing three passes.

The first is a hybrid approach. Keep these three deterministic passes as a fast, safe first stage, then hand the output to an embedding-aware or LLM-scored compression tool like LLMLingua. This catches semantic redundancy that literal identifier matching misses, such as two passages saying the same thing in different words.

Running the deterministic passes first preserves the safety guarantee. If the learned stage misbehaves, the conversation drops back to the deterministic baseline instead of failing unpredictably. Architecturally, the learned pass only operates on the output of Pass 3. A model bug in the compression step might shrink a prompt too aggressively, but it cannot reintroduce a dependency failure that the deterministic passes already cleared.

The second extension is closing the gap between synthetic data and production reality. The next step is regenerating these same three workload categories from actual production traces. Keeping the benchmark methodology identical—the same fixed parameters, ground-truth dependency labels, and 15-configuration sweep—ensures the results remain directly comparable to these published figures while replacing guesses with real telemetry.

Actual usage logs will likely shift the RAG and tool-agent metrics in either direction. Real-world traffic does not automatically mean higher compression. For instance, a production system with aggressive upstream deduplication might already eliminate the waste this synthetic model assumes. That would be a highly useful finding in its own right, rather than a failure.

A third, smaller point to note: the REF/DEFINE convention used here is just a placeholder. A production system should derive these tags automatically from structured data, tool call arguments, session variables, or explicit user settings, rather than relying on manual text markers.

While the deterministic logic in Pass 3 remains identical either way, the actual value of this pipeline depends entirely on how cleanly you can generate accurate dependency tags upstream. That is an architecture-specific integration problem rather than something a general-purpose pruning library can solve out of the box.

Production serving systems already treat prompt bloat as a memory management problem at the infrastructure layer, evicting and sharing KV cache pages much like an operating system handles physical memory [4]. You can think of this project as operating one layer above that, at the prompt construction phase, before the request ever reaches the infrastructure.

The two layers do not compete. A smaller, deduplicated prompt provides a better input to a well-managed cache rather than acting as a replacement for it.

The Core Takeaway

If a critical fact is preserved, the pipeline needs to demonstrate that retention rather than assume it based on the absence of obvious errors. This principle guided the design of the system.

Long-running conversations do not always require a larger, smarter model to decide what to remember. Often, the more robust solution is a predictable, three-pass system that can programmatically prove what it did not lose.

Sources

[1] Levy, M., Jacoby, A., & Goldberg, Y. (2024). Same task, more tokens: The impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 15339–15353). Association for Computational Linguistics.

[2] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.

[3] Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023). LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 13358–13376). Association for Computational Linguistics.

[4] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23) (pp. 611–626). Association for Computing Machinery.

All code, benchmark numbers, and test results in this article are my own, generated by running the included codebase directly and reproduced on two separate machines. No proprietary datasets, copyrighted text, or third-party code were used in building or benchmarking this system. All code, the corpus generator, the pruner, the benchmark harness, and all 35 tests, is available in the repository linked below.