models that represent true knowledge within? This post presents BizzaroWorld, a machine learning experiment that attempts to localize fact-remembering circuits in the Gemma model family using patch activation in 60 fast pairs and 20 information categories. The technical work here is heavily influenced by the work done by Prakash et al.¹, looking at business tracking within the LLaMa series of models.
The goal: to localize where the true information resides within the transformer, and whether that location is consistent with the model scale. The complete codebase is available here.
Experimental Setup
First of all, I wanted to know why the logit difference in pure and corrupted fast pairs was a good way to approach this problem. Finding the facts seems like finding organizations like Prakash et al. he did, so I focused on Indirect Object Identification (IoI). This is clearer when I present the three true commandments and their pure intentions.
- “When red paint meets yellow paint the result is” -> “Orange”
- “The Epic Inferno was written by ” -> “Dante”
- “The Roman god Mercury is the same as the Greek god” -> “Hermes”
It seemed to me that, to answer questions like this, LLMs could do so the need to find businesses within their representation. So, I looked for some answers about how the logit difference and IoI would work in this context, and, in this case, the ARENA² tutorial was very helpful. I have found that the log difference between clean and damaged fast pairs provides a clear scale signal that is well suited to quantifying the causal effect of a corrective intervention, so that is what I measured.
Therefore, I designed a truth battery with 60 quick clean/dirty pairs, across 20 categories of truths.
Before starting any patch tests, I wanted to identify the highest data pairs; those where the logit difference between clean and dirty runs was more pronounced and informed by the opening patch. So I created my own metric to measure this, TotalSwingwhich calculates the net effect of patching these data pairs. Here is an example of what I am talking about.
My theory was that calculating the logit difference on both sides and subtracting it would be a clean signal, since the right-hand side – applying a clean target to the corrupted information – usually leads to a negative logit difference as shown above. That said, both values end up being added together, resulting in a nice, nice scale that I can sort through all the 60 pairs I've created.
It worked well, and with this I came up with a CSV file with all the instructions, sorted by TotalSwing. I called these quick pairs of gold, and, using them, I created three test methods for each test I was going to do.
And now, I was ready to start exploring.
Splitting the components of Gemma-2B
LLMs are big buildings. In order to know when anything is happening, we need to brutally isolate ourselves. Neel Nanda's TransformerLens³ was instrumental in doing just that. I wanted to connect to all the different pieces of the puzzle i.e. the residual streams before and after all the relevant parts (attention heads and MLP sublayers) in all layers. And that's exactly what I did. I did four tests, where I slowly reduced the model to its pieces.
- Trial 1 = patching the last token
- Test 2 = patching before and after each sublayers
- Experiment 3 = patching in the business token field
- Test 4 = patch before each attention head
The numbers generated by this test highlighted a clear finding: there is a true three-stage recall cycle within the Gemma family of models.
Layer 1 – Storage (layers 0-14, business token space): Facts are encoded as directions in the residual distribution of the business token. Residual streaming reasonably dominates here, contributing 40× more than attentional results and 18× more than MLP results. 86.7% of the top-15 fastest pairs released their stored signal in layers 13–15, with the worst layer of 16.3 in all test methods (Pearson r = -0.83 between model confidence and damage score).
Phase 2 – Routing (distributed attention heads): The signal goes from the enterprise token location to the projection endpoint through the shared attention heads. No one head was solely responsible, although the 2nd head was equally ineffective; for example, it was active in 40% of the fast pairs in all experimental mode A. However, the damage of each head (ΔLD = -0.68) was negligible compared to the full damage of the rest of the stream (ΔLD = -11.47).
Phase 3 – Reading (layers 15–17, token storage): The response is returned, not counted. Delayed blocks are transient which means the signal is already encoded and is just being read. These findings were consistent across all three assessment methods and 20 knowledge categories.
Three Phase Cycle in Scale: Gemma-12B-IT
I was following a general result, so the next step was to see if this pattern held for the main model, the Gemma-12B-IT as well. Although I wanted to test it with even larger Gemma models, such as the Gemma-31B or Gemma-27B, I ran into a disk space constraint for my university's HPC, which I'll talk about later. However, I was able to replicate the entire 12B model suite, including all test methods A, B, and C.
I got some interesting results while doing so, but first, let's revisit how and where these two structures differ.
Apart from this architectural difference, everything else, such as the tokens used⁴, is the same between the two models. However, I noticed something very different in Gemma-12B-IT regarding the behavior of the tokenizer which contributed to the faster selection of pairs.
When I did the first triage pass, as I explained above to measure my 60 quick pairs, this big Gemma model removed three quick gold pairs, even if they both use the same tokenizer.
This exclusion occurs because, during the forward pass, the model converts individual tokens into identical ID tokens. For these passes to work, the shape of the array must be consistent with everything else, otherwise matrix multiplication does not work. I had noticed this when I was doing this process with Gemma-2B, where I noticed a strange behavior, like the physics unit “hertz” being drawn on two tokens⁵. It doesn't make much sense. I expected my 60 data to go through the Gemma-12B-IT without problems, but I was wrong here. This effect was obviously more pronounced when I did the first tests with the LLaMa-70B⁶, which I will explain in the upcoming work section below, but it surprised me.
The result of this is that cross-model mechanistic comparisons are constrained by the tokenized portion of the dataset, and reported differences must be interpreted with that privacy in mind.
Therefore, before designing any true batteries for such tests, we need to apply the facts to all the models being tested, so that any tokenizer drift can be reported immediately, and affected quick pairs can be replaced before they start doing any tests.
After noticing this confusion, I did exercises 1 to 4, in all test methods A, B, and C.
I found that this three-stage cycle is replicable in scale: the storage is shifted to layers 0-27, the route is distributed without the dominant head, and the reading is focused on the last layers, such as the Gemma-2B structure, scaled equally.
Here are some statistics that show this.
Each category of reality shows its behavior, which was consistent with my belief.
Another interesting thing I noticed is that the effect of the attention heads of the Gemma-12B-IT seemed to be more spread out and blurred than what I saw on the Gemma-2B. This finding is highlighted by the average heatmaps of ld_delta in the two models above, in the attention heads.
In Gemma-12B-IT, the heat map is almost empty, except for small colored cells around layers 20 and 28⁷. Once again, the magnitude of the logit difference values was greater in the residual distribution than in these sublayer components.
Disk Quota Issues, Future Considerations, and Conclusion
Throughout this experiment, I discovered many interesting techniques that I was eager to try, such as path avoidance (with DCM) and CMAP. I also wanted to test whether balancing or running the entire pipeline using a fine-tuned variant would change this three-stage true recall circuit. However, I was restricted by the 30GB disk quota limit. In fact, as noted above, I had already configured the 8-bit version of the LLaMa-70B and faced the exclusion of 22 fast pairs, due to token differences. Inference was working fine, but I just couldn't extend the process beyond that due to disk issues, so I shelved the idea later.
In my opinion, extending this line of work by amending the method is the next natural step, as it can show a lot more. Route patching, formalized by Goldowsky-Dill et al⁸., sharpens patching functionality from node-level to edge-level precision. Standard activation measures the causal effect of a node by changing its output and looking at all downstream effects; path routing, instead, separates each edge in the computation graph, revealing precisely which parts communicate with each other.
These findings establish the basis for targeted intervention: knowing when the memory is alive is a necessity to know where to intervene when it fails. To further these ideas, I would like to follow through on my original plans, and see how the attention heads work together using SAEs. Yes, the remaining stream is doing the hard work, but what does that mean? I need more information.
In summary, the logical next step for this work is the replication of various structures in LLaMA and other variants⁹. Additionally, the distributed path finding in Experiment 4 warrants a path reversal test to determine direct causal relationships between components.
References
- Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine tuning improves existing methods: A case study for business tracking, 2024
- Gemma uses the SentencePiece token
- Although something like the physical unit “Watt” was a single token, as you would logically expect
- LLaMa uses Tiktoken or SentencePiece token, depending on the model version
- If you look closely, some cells are also highlighted, but they are dull compared to Gemma-2B
- Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Behavior of a spatial model with path avoidance, 2023.
- It would be especially interesting to include a mixed-language model like LLaDA-8B here, since the attention mechanism is very different from standard autoregressive transformers, which require a custom hook infrastructure beyond what TransformerLens currently supports.
