When I was working on my problem of breaking down information on purpose, I ran into a confusing roadblock. My setup included a teacher model, which is Roberta-large (completed with a split for my purpose), and a student model, which I tried to train without losing too much accuracy compared to the teacher.
I have tried many mapping techniques, connecting all the 2nd layer to the students' layer, calling the two teacher layers to be one, and giving the tools to make the norm as giving (0.3 in L1 and 0.7 in L2). But no matter what combination I tried, the teachers' accuracy never matched the students' model.
That's when I started exploring How to Write Highly Informative Covers in my student model so that the student can increase their performance. I was looking for a way to measure whether the layer is the type of teacher model that really matters with distillation.
In that search, I stumbled upon an interesting paper– “Spectralkd: A unified framework for interpreting and filtering Transformers for analytical testing..
Curious, I decided to align with the concept and details of the text – again Boom!!!, it actually worked!For the first time, my student model began to think almost like its teacher.
Source: Author
Here too thickness graph of the layer of good Fine Roberta is great model. Based on the selection that looks good, I chose Layers 1-9 and 21-23 It's mine Student model In the era of information destruction, those who hold rich information.
I can't share my data or code for privacy reasons, but I will go as far as to say A paper-based method you are my soul Text-based conversionand how you can think about doing the same.
Behind the scenes: How FFT reveals the soul of a model
So, let's start with intensity of the showand step into the real magic here: Fast Fourier Transform (FFT).
In the spectralkd paper, the authors introduce a framework that helps us visualize the vision of the Vision Transformer (VITS), not only what they predict, but also How information flows through components. Instead of relying on observation or visualization, they use an observational analysis, method to measure the frequency richness of the model's internal representations.
Think of each layer of the transformer as a musician in an orchestra, some layers play high notes (fine detail), and others play low notes (broad features). FFT helps us to listen to each player's music separately and filter out which one has the most powerful songs, that is, the richest information signals.
Source: Author
Step 1: Map features, raw material
B is the batch size C is the number of channels and, H, w area height and width.
Step 2: Using the Fourier Transform
The authors apply a 1-dimensional FFT to the channel side to translate this real performance into the frequency domain: F(x) = fft(x)
This means: At every point in the area (B, H, W), a 1D FFT integrated across all channels. The result is A complex time frame (From fft actual output + imaginary parts). F (x) So it tells us how much of each frequency is present in that layer representation.
And if you're wondering, “Why fft?” – Hold that thought. Because later in this blog, we will reveal it certainly the FFT is a suitable tool measure the dimensions of the internal model.
Step 3: Measuring frequency power
Re(f(x)) is the real part, Im(f(x)) it is part of thinking.
Step 4: Leveling the map
Now we want to summarize this intensity for all the lateral positions:
This step tells us the size of one channel
And then you can just average the individual channels. Vailà! You now have the consistency of a single layer Vision Transformer.
Betting on the frequency domain: a pecctralkd lens
Let's look at the fast Fourier transform:
Xₖ is the input sequence (your signal, feature, or activation pattern). Xₙ is the frequency component of the Frequency Index. N is the number of points in the sequence (ie, the number of channels or features).
Each term ⁻ʲ²πᵏⁿ / ᴺ functions as Rounding Phasara small strange wave that circulates in the field, and together, they form one of the most beautiful ideas in signal processing.
Source: Author (here, the rotational phasor e⁻ʲ²πᵏⁿ / ᴺ increases with G (T) in the complex plane)Source: Author
.OMG! What happened here? Let me break up.
When you multiply your hidden function xₙ (say, across channels or feature sizes) with this pass, you're essentially asking:
“Wow, layer, how much k-th type of variation Do you contain your presentations? “
Each frequency k corresponds to a variable pattern scale beyond the characteristic features.
Deduction of lower prices k Broad, smooth, smooth structures (As a topic level context), while high k Fast, good variety (Like TOKEN-Level Nuances or syntactic signals).
Now here's the fun part: If a certain layer overlaps in a regular pattern, the iteration of Fineier Transform Qinaly, and the sum of the formula with the formula produces a Strong Answer therefore k.
Otherwise, the rotation cancels out, which means that frequency does not play a major role in that layer image.
Therefore, the four changes do not add anything new; Just figuring out how our layer captures information at different output scales.
It's like zooming in and seeing:
Other layers present Him with smooth, mental (low frequency) meanings,
Some buzz with sharp, detailed interactions between tokens (high frequencies).
FFT basically turns hidden hidden states into frequency fingerprints – A map of what kind of information the layer is focused on.
And that's exactly what spectralkd uses to figure out which layers actually doing the heavy lifting At the time of information entry.
If you still need to see a visual and a better understanding of the four transformations, you can just go through the video of 3Blue1Brown, “but what is the basic transformation? A visual introduction.”
From perception to language: Visual robustness guided my classifier objective
Source: Author
Let the Later activation tensor be:
where:
N = number of samples (batch size)
L = sequence length (number of tokens / time steps)
H = hidden dimension (number of channels / features generated by the layer)
For each sample I has a matrix xᵢ ∈ Rᴸ ˣ ᴴ (Sequence of positions X Hidden features)
Now again, you can compute the FFT of that Xᵢ and scale the Frequency length using real and imaginary components and scale across channels, then each layer.
Frequency length:
Frequency of available channels:
Frequency across layer:
Here, K is the number of stored characters.
Lasting
Their analysis shows a great deal of important insight:
Not all layers contribute equally. Of the similar constructions of the transformer, there are only a few – fast and the last one Layers display powerful visual activity, true “hotspots” of information flow.
Different versions of Transformer, same tunes. Despite the diversity of structures, Hierarchical variables and uniformitarians share some similar patterns, which lie in the way all these models learn and represent information.
Building on these findings, spectraalk introduces a Simple, paraliter-free distillation of information (KD) Strategy. By selectively working on the appearance of the early and previous layers between the teacher and the student model, the student learns mimics the teacher's physical signatureeven in a middle class that has never been clearly aligned.
The results are striking on paper: The broken reader (Deit-Tiny) is not only similar in performance to benchmarks such as ImageNet-1K, and you learn to think visually like a teacherwhich captures significant local and global information reliability.
Finally, spectralkd bridges Translation and distillationwhich provides a new way to visualize what is happening inside the transformers during the study. It opens up a new line of research, the authors note “Distillation Dynamics”A journey that revolves around how knowledge emerges, oscillates, and adapts within networks of teachers and students.
Progress
Fundamentals of core spectral & transformer
Vaswani, A. All the attention you need.Neurips2017.
Dosovitsky, A. Image calls 16 × 16 words: image recognition transformers in scale. Arxiv Preprint Arxiv: 2010.11929, 2020.
Raghu, M., Unterthiner, T., KornBlith, S., Zhang, C., & Dosovitskiy, A. Do vision transformers see as neural neural networks?Neurips2021.
Han, k. et al. Survey at Vision transformer.Yee tpami2022.
Interpretation and analysis of appearance
Chefer, H., Gur, S., & WELF, L. Dynamic translation is more than noticeable.Cvpr2021.
Yes, c. et al. Attention: A global perspective on Transformer attention.Yee tvcg2023.
Zeng, J. et al. Pulling back the layers: interpreting vit stories.ACM Multimedia2024.
Information decomposition and model compression
Hinton, G. Reinforcement of information in a neural network. Arxiv Preprint Arxiv: 1503.02531, 2015.
Phuong, M., and Laghter, C. In understanding the understanding of analytical knowledge.ICML2019.
Park, W. et al. Shared Information Preparation.Cvpr2019.
Chandrasegaran, K. et al. Going back to label consistency and distillation consistency: What was missing?ICML2022.
Huang, T. et al. A breakdown of knowledge from a powerful teacher.Neurips2022.
Pham, C. et al. Often attention to information distillation.WACV2024.
Fan, J. et al. SCALEKD: Changemakers with the power to transform can be great teachers. Arxiv Preprint Arxiv: 2411.06786, 2024.
Son, S. et al. A role of rubbing for proper guided distillation information for Vision Transformers.ECCV2025.
