# Introduction
Two years ago, the concept of agent AI was simple: the bigger the model, the better the agent. Larger context windows, more parameters, sharper thinking. What don't you like? NVIDIA's research team spent 2025 quietly building a case against that assumption, and the resulting debate has reshaped how many production agents are built in 2026. Much of what an agent does on a day-to-day basis is not extensive, creative, or novel. A small number of specialized tasks are performed repeatedly with little variation, and a model trained to be a generalist is too much work. That is the opening of small language models (SLMs), and they go from a footnote to a real architectural decision in the design of an agent.
This article looks at five concrete ways SLMs are showing up in next-generation agents right now, from the research that supports them to the tools and numbers you should know if you're deciding whether your next agent needs a boundary model at all.
# 1. Managing Repetitive Work Models Were Not Built
The basic case for SLMs in agents comes from a single, widely discussed paper from NVIDIA Research: Minimal Language Models are the future of Agentic AI. The authors argue that large-scale language models are more important in general conversation, but agent systems often call on language models to perform a small number of specialized tasks repeatedly, such as passing a command, selecting a tool, or returning a result in a fixed JSON format. That's a completely different job than holding an open conversation, and you don't need a trained model to do everything.
The central claim of the paper is straightforward: SLMs are sufficiently powerful, inherently more efficient, and more cost-effective in many appeals to agent systems, and are therefore the future of agent AI. What makes this more than an idea piece is the thinking behind it. Agents value reliability over innovation, and a small model fine-tuned to always follow a consistent output format and field configuration is often more reliable for that one task than a large general-purpose model asked to do the same thing on the fly. Large models still earn their place with original or open-minded novel ideas. They just stopped being the default for everything in between.
# 2. Works Directly on the Device, No Cloud Tour Required
One of the practical changes enabled by SLM is to move the model itself off the remote server and onto whatever hardware the agent is already running on, such as a phone, a laptop, a piece of industrial equipment. Sending a request to a data center takes hundreds of milliseconds, while routing at the edge happens in tens, and for an agent meant to feel responsive in that time, that gap is the difference between something that feels fast and something that feels like it's thinking too much.
The hardware caught on faster than most people expected. Apple A19 Pro's neural accelerators give the iPhone 17 Pro enough integrated AI power to run 8 billion parameter models at more than 20 tokens per second – fast enough for a real-time conversation – and Apple's M5 Max can handle models up to 30B parameters with acceptable latency. Quantization is a big part of why this works on consumer hardware at all. The Phi-4-Mini model compressed to 4-bit precision takes up about 1.2 GB of memory instead of 7.6 GB for full precision, while running. 95% of its benchmark performance — small enough to fit neatly into a phone with 8 GB of RAM.
The tools are the same Ollama for local and Microsoft operations Where is the family model? it has become common practice for developers to build this type of agent behavior on the device, especially in use cases where the agent needs to continue working even when the network connection is not guaranteed.
# 3. Fine Tuning of Tool Call Specialists
The standard small model, which came out of the box, is really bad at hitting tools. It inflates function names, invalidates parameters, and breaks the expected output format more often than you'd like. The adjustment is not a large model – it is more focused. Fine-tuning a small model to a particular tool's schema produces more than 90% accuracy at a cost effective for each query, because the model stops trying to be a generalist and starts being good at just one small task.
The research that supports this is surprising. The fine-tuned SLM achieved a pass rate of 77.55% in ToolBench tests, ahead of basic methods using very large models motivated by chain reasoning. You don't need to run a massive training to get there. In practice, 1,000 to 5,000 high-quality samples per instrument are sufficient to reach 95% accuracy—once in a well-defined schema, which is a realistic amount of data for a small group to generate in-house.
If you want to take a closer look at which specific models are leading the way at the moment, KDnuggets has recently put together five small, lightweight models designed specifically for toolkits, which take a few billion parameters each and are designed to run without a data center behind them.
# 4. Enabling Different Systems Where Large and Small Models Divide the Work
The most interesting architectural application of SLMs does not change the main models directly; it pairs them. The pattern that has become standard in 2026 places a highly rational boundary model in the role of organizer, strategy manager and ambiguity solver, while smaller domain-specific models act as workers, each configured for a single atomic task such as classification, classification, or summarization. Some call this the executive-worker architecture; others call it heterogeneous model routing. In any case, the idea is to use expensive thinking where necessary and let cheaper models handle the volume.
The cost difference this produces is hard to ignore. Boundary model with surrounding value $15 million tokens handle 30% of transactionspaired with a smaller model around $0.15 million tokens hold the remaining 70%it costs about 10 times less than moving everything through the boundary model alone. The pattern held up in controlled studies as well. One study comparing the homogeneous setup of all 7B-parameter agents against a disparate setting, where less 3B models managed a low-level job while a 7b model remained as a validator, found that the heterogeneous system maintains performance almost identical to the all-7B baseline while cutting latency 31.6% and total API cost in 41.8%. NVIDIA has included tools for building this type of system Nemoaimed at teams that want to integrate fine-tuned SLMs for routine work with occasional calls to a larger hard-case model.
# 5. Keeping Sensitive Data On The Device Rather Than Sending It Anywhere
The final change is less about speed or cost and more about where the data is allowed to go in the first place. An agent that runs entirely on on-premise hardware doesn't need to send user chat, documents, or behavior to a third-party API to get a response, which is especially important when you're working with healthcare records, financial information, or anything covered by strict compliance rules.
In healthcare use cases or industrial security in particular, data often cannot leave the local network at all, eliminating cloud-hosted edge models as an option, no matter how good. Smaller models abandon that restriction entirely by working where the data resides. Sending the Edge to something like Apple Silicon or a Qualcomm chip costs only the hardware of the device itself, with the host of a smaller private model running. 10,000 Daily quizzes are usually effective $500 to $2,000 per month, in comparison $5,000 to $50,000 per month in equal volume with the large API model.
This is also the only realistic option for fully air-gapped environments – areas that have absolutely no Internet connectivity by design – where a cloud-dependent agent can't operate without a budget. For agents built for regulated industries or early offline products, this is not a good thing to have. That's the whole reason an agent would be there at all.
# Wrapping up
None of this means that frontier models are on the way out. Truly novel thinking, long open context, and unprecedented functionality are still the dominant models, and that's not changing anytime soon. What has changed is the assumption that every single call an agent makes requires that level of horsepower. Most of the actual work of an agent – partitioning, routing, formatting, tool calls – turns out to be small enough that a small, well-engineered model can handle it in the same way, often faster, and at a fraction of the cost.
The best growing agents in 2026 are not those built on the single largest model available. They're the ones built with the right size model for each piece of work – borderline intelligence where it's found and smaller, specialized models everywhere.
Long Shithu is a software engineer and technical writer who likes to use cutting-edge technology to make interesting stories, with a keen eye for detail and the ability to simplify complex concepts. You can also find Shittu Twitter.
