Overview

Switchyard is a powerful Rust proxy and library designed to streamline and enhance your Large Language Model (LLM) traffic. It intelligently routes requests across various LLM providers, seamlessly translates between popular API formats (OpenAI and Anthropic), meticulously records operational metrics, and offers flexible, typed, and composable routing algorithms. Switchyard empowers you to connect coding agents like Claude Code or Codex to open-source models, allowing them to communicate using their native APIs while Switchyard handles the underlying model interaction and translation.

Key Features

• Universal API Translation: Effortlessly convert requests and responses between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats.
• Advanced Multi-Backend Routing: Choose from a variety of routing strategies, including random distribution, LLM-as-classifier, signal-driven stage routing, or implement your own custom algorithms.
• Comprehensive Operational Metrics: Gain deep insights into your LLM traffic with Prometheus metrics covering requests, errors, latency, token usage, and routing overhead.
• Flexible Deployment: Utilize Switchyard as a standalone proxy, embed its routing logic directly into your Rust applications, or leverage its command-line interface for launching agents.

Typical Use Cases

• Connecting Agents to Diverse Models: Enable AI coding agents to interact with any OpenAI-compatible endpoint or leverage open-source models served by vLLM, NVIDIA NIM, or Ollama without code modifications.
• A/B Testing and Benchmarking: Distribute traffic across multiple LLM models to conduct rigorous A/B tests, compare performance, and manage costs effectively.
• Intelligent Traffic Steering: Implement sophisticated routing logic, such as signal-driven stage routers, to dynamically direct traffic based on conversation context, tool results, or error signals.
• Building Custom LLM Gateways: Integrate Switchyard's routing and translation capabilities into your existing infrastructure to create a robust and adaptable LLM gateway.