Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders.<\/em> Subscribe Now<\/em><\/p>\n\n\n\n Researchers at Katanemo Labs have introduced Arch-Router, a new routing model and framework designed to intelligently map user queries to the most suitable large language model (LLM).\u00a0<\/p>\n\n\n\n For enterprises building products that rely on multiple LLMs, Arch-Router aims to solve a key challenge: how to direct queries to the best model for the job without relying on rigid logic or costly retraining every time something changes.<\/p>\n\n\n\n As the number of LLMs grows, developers are moving from single-model setups to multi-model systems that use the unique strengths of each model for specific tasks (e.g., code generation, text summarization, or image editing).\u00a0<\/p>\n\n\n\n LLM routing has emerged as a key technique for building and deploying these systems, acting as a traffic controller that directs each user query to the most appropriate model.<\/p>\n\n\n\n Existing routing methods generally fall into two categories: \u201ctask-based routing,\u201d where queries are routed based on predefined tasks, and \u201cperformance-based routing,\u201d which seeks an optimal balance between cost and performance.<\/p>\n\n\n\n However, task-based routing struggles with unclear or shifting user intentions, particularly in multi-turn conversations. Performance-based routing, on the other hand, rigidly prioritizes benchmark scores, often neglects real-world user preferences and adapts poorly to new models unless it undergoes costly fine-tuning.<\/p>\n\n\n\n More fundamentally, as the Katanemo Labs researchers note in their paper, \u201cexisting routing approaches have limitations in real-world use. They typically optimize for benchmark performance while neglecting human preferences driven by subjective evaluation criteria.\u201d\u00a0<\/p>\n\n\n\n The researchers highlight the need for routing systems that \u201calign with subjective human preferences, offer more transparency, and remain easily adaptable as models and use cases evolve.\u201d<\/p>\n\n\n\n To address these limitations, the researchers propose a \u201cpreference-aligned routing\u201d framework that matches queries to routing policies based on user-defined preferences.<\/p>\n\n\n\n In this framework, users define their routing policies in natural language using a \u201cDomain-Action Taxonomy.\u201d This is a two-level hierarchy that reflects how people naturally describe tasks, starting with a general topic (the Domain, such as \u201clegal\u201d or \u201cfinance\u201d) and narrowing to a specific task (the Action, such as \u201csummarization\u201d or \u201ccode generation\u201d).\u00a0<\/p>\n\n\n\n Each of these policies is then linked to a preferred model, allowing developers to make routing decisions based on real-world needs rather than just benchmark scores. As the paper states, \u201cThis taxonomy serves as a mental model to help users define clear and structured routing policies.\u201d<\/p>\n\n\n\n The routing process happens in two stages. First, a preference-aligned router model takes the user query and the full set of policies and selects the most appropriate policy. Second, a mapping function connects that selected policy to its designated LLM.\u00a0<\/p>\n\n\n\n Because the model selection logic is separated from the policy, models can be added, removed, or swapped simply by editing the routing policies, without any need to retrain or modify the router itself. This decoupling provides the flexibility required for practical deployments, where models and use cases are constantly evolving.<\/p>\n\n\n\n The policy selection is powered by Arch-Router, a compact 1.5B parameter language model fine-tuned for preference-aligned routing. Arch-Router receives the user query and the complete set of policy descriptions within its prompt. It then generates the identifier of the best-matching policy.\u00a0<\/p>\n\n\n\n Since the policies are part of the input, the system can adapt to new or modified routes at inference time through in-context learning and without retraining. This generative approach allows Arch-Router to use its pre-trained knowledge to understand the semantics of both the query and the policies, and to process the entire conversation history at once.<\/p>\n\n\n\n A common concern with including extensive policies in a prompt is the potential for increased latency. However, the researchers designed Arch-Router to be highly efficient. \u201cWhile the length of routing policies can get long, we can easily increase the context window of Arch-Router with minimal impact on latency,\u201d explains Salman Paracha, co-author of the paper and Founder\/CEO of Katanemo Labs. He notes that latency is primarily driven by the length of the output, and for Arch-Router, the output is simply the short name of a routing policy, like \u201cimage_editing\u201d or \u201cdocument_creation.\u201d<\/p>\n\n\n\n To build Arch-Router, the researchers fine-tuned a 1.5B parameter version of the Qwen 2.5 model on a curated dataset of 43,000 examples. They then tested its performance against state-of-the-art proprietary models from OpenAI, Anthropic and Google on four public datasets designed to evaluate conversational AI systems.<\/p>\n\n\n\n The results show that Arch-Router achieves the highest overall routing score of 93.17%, surpassing all other models, including top proprietary ones, by an average of 7.71%. The model\u2019s advantage grew with longer conversations, demonstrating its strong ability to track context over multiple turns.\u00a0<\/p>\n\n\n\n In practice, this approach is already being applied in several scenarios, according to Paracha. For example, in open-source coding tools, developers use Arch-Router to direct different stages of their workflow, such as \u201ccode design,\u201d \u201ccode understanding,\u201d and \u201ccode generation,\u201d to the LLMs best suited for each task. Similarly, enterprises can route document creation requests to a model like Claude 3.7 Sonnet while sending image editing tasks to Gemini 2.5 Pro.\u00a0<\/p>\n\n\n\n The system is also ideal \u201cfor personal assistants in various domains, where users have a diversity of tasks from text summarization to factoid queries,\u201d Paracha said, adding that \u201cin those cases, Arch-Router can help developers unify and improve the overall user experience.\u201d<\/p>\n\n\n\n This framework is integrated with Arch, Katanemo Labs\u2019 AI-native proxy server for agents, which allows developers to implement sophisticated traffic-shaping rules. For instance, when integrating a new LLM, a team can send a small portion of traffic for a specific routing policy to the new model, verify its performance with internal metrics, and then fully transition traffic with confidence. The company is also working to integrate its tools with evaluation platforms to streamline this process for enterprise developers further.<\/p>\n\n\n\n Ultimately, the goal is to move beyond siloed AI implementations. \u201cArch-Router\u2014and Arch more broadly\u2014helps developers and enterprises move from fragmented LLM implementations to a unified, policy-driven system,\u201d says Paracha. \u201cIn scenarios where user tasks are diverse, our framework helps turn that task and LLM fragmentation into a unified experience, making the final product feel seamless to the end user.\u201d<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>The challenges of LLM routing<\/h2>\n\n\n\n
A new framework for preference-aligned routing<\/h2>\n\n\n\n
![\"Preference-aligned](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/07\/image.png?w=800\")
Arch-Router in action<\/h2>\n\n\n\n
![\"Arch-Router](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/07\/image_88212c.png?w=800\")
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n