some of the problems we face in using AI algorithms, we tend to focus on their processing power.
But now, NVIDIA GPUs are powering the latest models of major languages, and companies are competing to build the fastest AI accelerators. These new chips promise more processing power, more cores, and more performance per second.
So, one would think that the future of AI depends on building powerful processors! But there is a problem. Most of today's AI systems are not limited to how fast they can perform calculations. They are limited in how fast they can access data. Important data for them to do their calculations.
In other words, the future of AI may depend less on computation and more on memory.
I know that may not seem intuitive, but let's step back and think about something: Imagine hiring the most efficient and fastest chef in the world.
This chef can prepare food at an amazing speed. However, all ingredients are stored in a warehouse a few miles away. Before the chef can cook, someone needs to grab the ingredients and take them to the kitchen.
No matter how skilled a cook is, there will be times when they just stand there, waiting for the ingredients they need to start cooking to arrive.
Modern AI systems face a similar challenge. The processors they use can do calculations, but they can't work on raw data. If the processor can calculate faster than the memory can deliver the information, performance is limited by data movement rather than aggregation.
This is known in computer science as a memory bottleneck. It is one of the most important and least discussed challenges in modern AI.
Scale of Modern Models
To better understand why memory has become such an important problem, we need to consider the size of today's AI models. Early machine learning models may have thousands or millions of parameters, while modern machine learning models contain billions or trillions.
Each of these arrays represents a numerical value that must be stored in memory and accessed repeatedly during training and interpretation.
Okay, that sounds reasonable, but let's back it up with some numbers! Let's say we have a model that contains 70 billion parameters. Before even a single calculation can be performed, the system needs a place to store all those parameters.
Now, I want you to imagine thousands of users interacting with the model simultaneously. Computer hardware must continuously transfer large amounts of information between memory and processors.
As you may have concluded, the challenge is no longer just to do the math; rather, it transfers data to the hardware quickly enough.
Moving data can be more expensive than running a computer on it, which is one of the main objections to computing. Over the past decades, processor performance has improved dramatically, as engineers have gotten better at designing chips that can perform calculations faster and faster. On the other hand, however, the development of the memory system is moving at a slower pace.
This created a growing imbalance that became more apparent as AI systems developed. Today's processors can perform billions of operations per second, but they often spend a lot of time waiting for data to arrive.
This restriction of data manifests itself in different ways within AI systems. We can see it while moving data between memory and processors, between GPUs, across servers, or between data centers.
These models will continue to grow, and as they grow, the problem of moving data will play a major role in the overall performance of the system!
Understanding AI Memory
So far, I think I've written the word “memory” more than 20 times (I didn't count!). You may be wondering, what kind of memory is he talking about? Most people are familiar with RAM, the memory included in laptops and desktop computers. AI systems use different types of memory for different purposes.
1- RAM: Random Access Memory stores data used by the CPU. It's big but not very fast compared to the special AI memory.
2- VRAM: Graphics Processing Units contain dedicated memory known as Video Random Access Memory (VRAM). Used during training and definition to store model parameters, training sets, activations, and average statistics. The amount of VRAM available usually determines whether the model can fit on the GPU.
3- High Bandwidth Memory (HBM): Modern AI accelerators increasingly rely on High-Bandwidth Memory (HBM). HBM is specifically designed to move large amounts of data very quickly. Rather than simply increasing memory capacity, HBM focuses on increasing memory bandwidth, which is the rate at which information can be transferred.
Although the size of the memory (its capacity) is important, its bandwidth plays a major role. Another way to think about those two concepts is the highway (stay with me here for a second). Capacity is the number of vehicles that can be carried on a highway, and bandwidth is the number of lanes available.
You can have a large parking lot, but if all cars have to travel in one lane, traffic becomes a limiting factor.
Now, as we mentioned earlier, AI systems use different types of memory in different ways. The memory challenge manifests itself differently during training and explanation.
Training: Training needs to store model parameters, gradients, activations, and optimizer conditions. So, as a result, the memory requirements become larger. This will require distributing the memory across multiple GPUs.
Hypothesis: Understanding generally requires less memory than training, but presents a different challenge. The model must serve requests continuously while receiving parameters and generating output quickly. For interactive systems like chatbots, latency is now a big problem!
A fast memory can bring information, a model can respond quickly. This is one of the reasons why memory technology remains important even after the training is over.
Some Final Thoughts
Most discussions about improving AI performance focus on larger models and faster processors. However, hardware engineers are increasingly realizing a different reality.
Building intelligent AI systems is not just a matter of adding more computing power. It also needs to solve the data transfer problem. Fortunately, many researchers are focused on improving the data movement problem. In doing so, they explore different approaches:
- Improved memory architecture.
- Quick links.
- Memory efficient algorithms.
- Model compression techniques.
- Near-memory computing.
- Optical and photonic communication technologies.
Each of these methods tries to answer the same question: How do we transmit large amounts of information in an efficient way?
The answer may determine the future trajectory of AI.
Modern AI systems rely on both computation and communication. While processors perform calculations, memory systems determine how quickly data can access them. As models continue to grow in size and complexity, memory capacity and bandwidth become increasingly important factors in overall performance.
The next big breakthrough in AI hardware may not come from a processor with more cores or higher clock speeds. It may come from a better way to transfer data.
So, which of these methods (if any) is the answer? Well, at the moment, we don't know, but we are slowly getting there.
Some references
- Brown, TB, et al. (2020). Language Models Young Learners Shoot.
- Chowdhery, A., et al. (2022). PALM: Measuring Language Modeling with Methods.
- Dao, T., Fu, DY, Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Faster Intuitive Attention and Effective Memory with IO-Awareness.
- Kwon, W., et al. (2023). Efficient Memory Management for a Large Language Model Serving with PagedAttention.
- Zhao, WX, et al. (2023). A Survey of Major Language Models.
