Meet LLMRouter: An Intelligent Routing System Designed to Improve LLM Inference by Dynamically Selecting the Best Fit Model for Each Question.

LLMRouter is an open source routing library from the U Lab at the University of Illinois Urbana Champaign that treats model selection as a first-class system problem. It sits between applications and dozens of LLMs and selects a model for each query based on task complexity, target quality, and cost, all exposed through an integrated Python API and CLI. The project ships with over 16 routing models, a data generation pipeline over 11 benchmarks, and a plugin system for custom routers.

Router families and models supported

LLMRouter organizes routing algorithms into four families, Single-Round Routers, Multi-Round Routers, Personalized Routersagain Agentic Routers. Single round routers include knnrouter, svmrouter, mlprouter, mfrouter, elorouter, routerdc, automix, hybrid_llm, graphrouter, causallm_routerand foundations smallest_llm again largest_llm. These models use techniques such as k nearest neighbors, support vector machines, multilayer perceptrons, matrix factorization, Elo estimation, inverse binary learning, automatic model clustering, and graph-based routing.

Many circular routes are displayed router_r1pre-trained instance of Router R1 connected to LLMRouter. Router R1 constructs multiple LLM routes and aggregation as a sequential decision process where the route itself is an LLM that alternates between internal logic steps and external model calls. It trains reinforced learning using a reward-based rule that balances format, effect, and cost. In LLMRouter, router_r1 is available as an additional installation target with pinned dependencies tested on it vllm==0.6.3 again torch==2.4.0.

The personal route is managed by gmtrouterdefined as a personal graph-based router with user preference learning. GMRouter represents LLM's multi-user interaction as a heterogeneous graph for users, questions, answers, and models. It uses a message-passing architecture over this graph to infer user-specific route preferences from several snapshot data, and experiments show accuracy and AUC advantages over non-personalized bases.

Agent routers in LLMRouter extend routing to multi-step logic workflows. knnmultiroundrouter it uses k nearest neighbor reasoning over a sequence of multiple turns and is intended for complex tasks. llmmultiroundrouter presents an LLM-based agent router that performs multistep routing without its training loop. These router agents share the same configuration and data formats as other router families and can be changed with a single CLI flag.

A data generation queue for submitting a dataset

LLMRouter ships with a full data generation pipeline that transforms standard benchmarks and LLM results into routing data sets. The pipeline supports 11 benchmarks, Natural QA, Trivia QA, MMLU, GPQA, MBPP, HumanEval, GSM8K, CommonsenseQA, MATH, OpenBookQA, and ARC Challenge. It goes through three distinct stages. First, data_generation.py it extracts queries and labels from the ground truth and creates JSONL train segmentation and evaluation. Second, generate_llm_embeddings.py build candidate embedding LLMs from metadata. Thirdly, api_calling_evaluation.py calls LLM APIs, evaluates responses, and aggregates scores by embedding route records. (GitHub)

Pipeline output query files, LLM JSON embeddings, query embedding tensors, and routing data are JSONL files. The route entry includes fields such as task_name, query, ground_truth, metric, model_name, response, performance, embedding_idagain token_num. Configuration is handled entirely through YAML, so developers point scripts to new datasets and candidate model lists without changing code.

Chat interface and plugin system

cooperation, llmrouter chat presents a premium Gradio-based chat platform over any router and configuration. The server can bind to a custom host and port and can expose a public share link. Query modes control how the router sees context. current_only uses only the user's latest message, full_context includes a history of dialogue, too retrieval add question with high k same history questions. The UI visualizes model options in real-time and is driven by the same route optimization used for batch rendering.

LLMRouter also provides a plugin system for custom routers. Newer routers stay under the hood custom_routersa small section MetaRouterand get started route_single again route_batch. Configuration files under that directory define data methods, hyperparameters, and default API endpoints. Plugin detection scans the project custom_routers folder, a ~/.llmrouter/plugins documentation, and any additional methods in LLMROUTER_PLUGINS environment variable. Examples of custom routing include randomrouterwhich chooses the model randomly, too thresholdrouterwhich is a trainable route that measures the difficulty of the question.

Key Takeaways

• Route as a summary of the first phase: LLMRouter is an open routing layer from UIUC that sits between various LLM applications and pools and centralizes model selection as a function of cost and quality assumptions rather than ad hoc scripts.
• Four routing families including 16 plus algorithms: The library measures more than 16 routers into four families, single round, multi round, personal, and agent, including knnrouter, graphrouter, routerdc, router_r1again gmtrouterall exposed through integrated configuration and CLI.
• Multi round RL routing through Router R1: router_r1 includes the framework of Router R1, where the LLM router intervenes in the internal steps of “thinking” and the external “route” calls and is trained with a reward based on the rule that includes the format, the result, and the cost to improve the performance cost trade-off.
• Graph-based personalization with GMRouter: gmtrouter model users, questions, answers and LLMs as nodes in a multivariate graph and use message passing to learn the user's specific choice of routes in several histories, achieving about 21% of accuracy gains and greater AUC improvements than robust foundations.
• End to end pipe and stretch: LLMRouter provides a benchmark-driven data pipeline, training and reporting CLI, Gradio chat UI, centralized API key management, and a supported plugin system MetaRouter which allows teams to register custom routers while reusing the same routing data sets and infrastructure.

Check it out GitHub Repo and technical details. Also, feel free to follow us Twitter and don't forget to join our 100k+ ML SubReddit and Subscribe to Our newspaper. Wait! are you on telegram? now you can join us on telegram too.