Switchyard

Switchyard routes each LLM call to the cheapest model that can still do the job. Without changing a line of your agent.

Get started →

_\*Total cost based on average ISP token cost_

What is Switchyard

Switchyard picks which model serves each LLM call.

Use Switchyard

Switchyard runs inside gateways you may already have.

• NeMo Relay — a native plugin. Load a routes.toml into a Relay deployment

you already run. Setup →

• LiteLLM — a routing plugin for LiteLLM's Router and proxy.

Setup →

• More integrations coming soon.

flowchart LR
    subgraph R["LiteLLM · NeMo Relay"]
        P["Switchyard"]
    end
    P--> M["Efficient model"]
    P--> N["Capable model"]
    P--> O[etc.]
    G[You] -->|"request"| P
    style P fill:#76B900,stroke:#5A8F00,color:#000

Integrate Switchyard into your gateway or harness

Embed the routing algorithms in your own. Switchyard picks the model; your harness makes the call, so your transport, retries, and credentials stay untouched.

• Install: pip install nemo-switchyard
• Then follow Path 2 — Embed the Library:

construct an algorithm, drive its step stream, make the answer call.

• Also available for Rust as switchyard-libsy; Path 2 has the Cargo.toml

block.

flowchart LR
    subgraph R["Your LLM gateway / harness"]
        P["Switchyard"]
    end
    P--> M["Efficient model"]
    P--> N["Capable model"]
    P--> O[etc.]
    G["Your users"] -->|"request"| P
    style P fill:#76B900,stroke:#5A8F00,color:#000

Run Switchyard as a standalone proxy

A server in front of an agent, when you have no gateway to put Switchyard in. Point Claude Code, Codex CLI, or any OpenAI/Anthropic SDK client at it; Switchyard decides per turn which model serves it.

• Install: cargo install --locked switchyard-server
• Then follow Path 3 — Run the Standalone Proxy:

write routes.toml, start the server, point your agent at it.

flowchart LR
    P["Switchyard
standalone proxy"]
    P--> M["Efficient model"]
    P--> N["Capable model"]
    P--> O[etc.]
    G[You] -->|"unchanged native API"| P
    style P fill:#76B900,stroke:#5A8F00,color:#000

Components

Pre-1.0 software. APIs, configuration, and routing behavior can change between releases — pin the version you integrate.

| Component | Stability | Use it for | Guidance | |---|---|---|---| | switchyard-libsy | Beta | Routing embedded in your own gateway or harness. You own model calls, credentials, and retries. | Trial integrations. API will change before v1.0. | | switchyard-llm-client | Alpha | HTTP model calls and protocol translation alongside libsy. | Experiments and pilots. | | switchyard-runner | Alpha | Running configured routes inside another runtime, such as NeMo Relay. | Integration work and supervised pilots. | | switchyard-server | Demo | A standalone OpenAI- and Anthropic-compatible proxy. | Demos and evaluation only. Not for production. |

Get Started

Three paths, in the same order as above. Using Claude Code or Codex? Point it at this README and ask it to set up the path you want.

Path 1 — Load the NeMo Relay Plugin

You finish with an existing NeMo Relay deployment routing through Switchyard. Requires NeMo Relay >=0.8.0, <1.0.0 and a Rust toolchain.

Follow the plugin README's Install and Configure Relay sections. For the deployment file, use the routes.toml from Path 3, step 2.

Path 2 — Embed the Library

You finish with your own harness picking a model per request and still making every model call itself. Shown in Python; the Rust API has the same shape.

1. Install.

pip install git+https://github.com/NVIDIA-NeMo/Switchyard.git

The API below is newer than nemo-switchyard 0.2.0 on PyPI, so install from source until the next release. Rust: depend on switchyard-libsy and switchyard-protocol from this repository instead. Pin both to the commit you tested — @ for pip, rev = "" for Cargo — before depending on them.

2. Construct an algorithm. It selects a category — efficient or capable — and you map categories to model IDs when each request runs.

from switchyard.libsy import LlmResponse, Step
from switchyard.libsy.algorithms import stage_router
algorithm = stage_router(picker="efficient_first", confidence_threshold=0.5)

3. Drive it. run_stream yields steps. Serve each CallModel with your own client — call.models is ordered by preference, and call.fail(error) reports a failed call; Done carries the pick.

models = {"efficient": ["fast"], "capable": ["quality"], "any": ["quality", "fast"]}
async for step in algorithm.run_stream(request, models):
    match step:
        case Step.CallModel(call):
            call.respond(LlmResponse.Agg(await my_client(call.request, call.models[0])))
        case Step.Done(outcome):
            model, request = outcome.selected_model_ids[0], outcome.request

4. Make the answer call with model and request, using your own HTTP client, retries, and credentials.

The complete runnable version — streaming and a working client — is examples/libsy.py. Types: switchyard-libsy, switchyard-protocol.

Path 3 — Run the Standalone Proxy

You finish with a server on localhost:4000 that any OpenAI or Anthropic client can call. Needs Rust with Cargo.

1. Install the server.

cargo install --locked switchyard-server

2. Write routes.toml. A stage router over the same model pair as the benchmark above: how to reach a provider, which models to use, how to choose between them. --config takes any path; this writes it to the current directory.

cat > routes.toml <<'TOML'
schema_version = 1
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"
[targets.capable]
id = "anthropic/claude-opus-4.8"
llm_client = "openrouter"
[targets.efficient]
id = "z-ai/glm-5.2"
llm_client = "openrouter"
[routes.switchyard]
id = "switchyard"
type = "stage_router"
capable_target = "capable"
efficient_target = "efficient"
picker = "efficient_first"
confidence_threshold = 0.5
TOML

Every key is documented in the TOML schema reference.

3. Start it. --dry-run loads the config, prints server OK: and the model IDs it exposes, then exits without starting the server.

export OPENROUTER_API_KEY="your-openrouter-key"  # pragma: allowlist secret
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

4. Send a request. The route's id is the model name clients ask for.

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}'

The same route also answers on /v1/messages (Anthropic Messages) and /v1/responses (OpenAI Responses). /v1/stats reports which target served what, and /metrics exposes Prometheus counters for requests, errors, latency, tokens, and routing overhead.

5. Point a coding agent at it.

export ANTHROPIC_BASE_URL="http://localhost:4000"
export ANTHROPIC_MODEL="switchyard"
claude

Codex CLI and other OpenAI clients use the OpenAI variables instead:

export OPENAI_BASE_URL="http://localhost:4000/v1"

Routing Algorithms

Most use an LLM as a judge. All of them pick between an efficient model and a capable one; what differs is when the decision is made and how.

| Algorithm | How it decides | Route type | Benchmark | |---|---|---|---| | Capability | The first request is judged by an LLM. | llm_classifier | 71.2% at $79.32 | | Stage | Tool responses are judged by pattern matching or an LLM. | stage_router | 72.7% at $68.19 | | Capability + Stage | Combines the two above. | composite | not yet benchmarked | | Escalation | Starts efficient. Responses are judged by an LLM for issues, then escalated. | llm_classifier + mode = "escalation" | 75.7% at $85.00 | | Advisor Gate | One model serves every turn; a stronger advisor approves its plans and "done" claims, or sends it back. | advisor | lifts a weak executor 43.8% → 54.7% | | Sub-Agent-Aware | Delegated sub-agent traffic routes separately from the parent agent. | subagents on passthrough or stage_router | not yet benchmarked | | Custom | The first request is judged by an LLM against criteria you define, routing among 2+ of your own models. | llm_classifier + target_selector policy | not yet benchmarked | | Random | Each request is routed at random, uniform or weighted. | random | baseline mechanism |

Benchmarks are Terminal-Bench 2.1 against a $98.06 Opus 4.8 baseline at 76.0%. A passthrough route registers one target under one model ID with no routing decision. See the Routing Overview for the common route shape and self-hosted targets.

Documentation

• Core Concepts: LLM clients, targets, routes, model IDs, and routing algorithms
• Routing Overview: choose and configure a routing algorithm
• TOML Schema: every configuration key
• Architecture: how the proxy and library components fit together
• switchyard-server: server configuration, routing algorithms, and metrics
• switchyard-libsy: embed routing algorithms in a Rust application
• switchyard-protocol: provider-neutral request, response, and streaming types
• switchyard-translation: request, response, and stream translation
• switchyard-nemo-relay-plugin: install Switchyard as a native NeMo Relay plugin

Benchmark Provenance

| Configuration | Accuracy | Total cost | vs. Opus 4.8 baseline | |---|---:|---:|---| | Opus 4.8 baseline | 76.0% | $98.06 | — | | Escalation | 75.7% | $85.00 | 99.6% of accuracy, 13.3% cheaper | | Stage | 72.7% | $68.19 | 95.7% of accuracy, 30.5% cheaper | | Capability | 71.2% | $79.32 | 93.7% of accuracy, 19.1% cheaper | | Kimi K2.6 alone | 55.8% | $76.28 | | | GLM 5.2 alone | 52.4% | $16.47 | | | DeepSeek V4 Pro alone | 48.7% | $96.92 | | | Ultra 3 alone | 39.0% | $29.66 | |

These are the v0.2.0 Terminal-Bench 2.1 results from Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard. Those runs used NVIDIA-internal inference endpoints, so absolute solve rates may shift on another serving stack; the routing parameters are the ones that ran.

The escalation deployment is checked in at benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml, with OpenRouter targets substituted so it is publicly runnable. To run the harness, see benchmark/README.md; for latency and routing overhead rather than task success, see Soak Testing.

Community

• Issues: GitHub Issues
• Code of Conduct: Code of Conduct

License

Apache 2.0 License. Copyright NVIDIA Corporation.