Three-tier model routing on Amazon Bedrock with LiteLLM

Originally published on AWS Builder Center. Any opinions in this article are my own and may not reflect the opinions of AWS.

Table of contents

The challenge: the bill follows traffic, not difficulty

Most applications on Amazon Bedrock pick one model during the prototype and keep it for everything, so a greeting and a design review cost the same per token. A frontier model overpays for the routine majority. A small model gives the hard minority weaker answers. In the enterprise workloads I see, most traffic is routine: classification, short answers, extraction, and everyday coding help. What’s missing is a reliable way to tell routine from hard on every request, with a safety net when that call is wrong.

What I built

One OpenAI-compatible endpoint, bedrock-auto, that looks like a single model to any client. Behind it, a LiteLLM gateway classifies each request with Auto Router v2 and sends it to one of three tiers on Amazon Bedrock:

The router always assigns one of four classes, so COMPLEX and REASONING share a model. A fourth model, Amazon Nova 2 Lite, does the classifying and never answers a request.

How the routing decision works

1/ Keyword rules first. Domain terms set a minimum tier, so a request that mentions “threat model” never lands on Tier 1.

2/ A small classifier model decides the rest. Nova 2 Lite reads the request and the last three user turns and returns a class. If it fails, the router falls back to a local heuristic scorer instead of failing the request.

3/ Conversation state overrides both. Session affinity keeps a conversation on its first-turn model, so a “thanks!” doesn’t move it and lose the prompt cache. classification_mode: user_turn classifies new user requests only, so an agent’s tool results stay on the model that started the task.

- model_name: bedrock-auto
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_default_model: tier2-balanced
    complexity_router_config:
      tiers:
        SIMPLE: tier1-fast          # OpenAI GPT-6 Luna, no reasoning
        MEDIUM: tier2-balanced      # Claude Sonnet 5.5, low effort
        COMPLEX: tier3-frontier     # Claude Opus 5.5
        REASONING: tier3-frontier
      classifier_type: llm
      classifier_llm_config: {model: classifier-fast, timeout_ms: 1500}   # Amazon Nova 2 Lite
      classifier_fallback: heuristic
      classifier_context_window_size: 3
      classification_mode: user_turn
      keyword_tier_rules:
        - keywords: ["hi", "hello", "thanks"]
          tier: SIMPLE
        - keywords: ["5g core", "network slicing", "sip trunk"]
          tier: COMPLEX
        - keywords: ["root cause", "architecture review", "threat model"]
          tier: REASONING
      enable_context_window_escalation: true
      session_affinity: true
      max_tokens_from_tier_model: false   # keep the client's max_tokens, see below

I use a classifier model because the heuristic scorer measures vocabulary, not difficulty. A public walkthrough shows it sending “prove that the halting problem is undecidable” to the cheapest tier because the prompt is short.

Why this architecture

Clients call one endpoint, bedrock-auto, over HTTPS. AWS WAF and an internal Application Load Balancer pass requests to the LiteLLM gateway on Amazon ECS with AWS Fargate. Amazon Nova 2 Lite classifies each request, and the router sends it through a VPC endpoint to GPT-6 Luna, Claude Sonnet 5.5, or Claude Opus 5.5. Aurora Serverless v2 holds keys, budgets, and spend logs, and ElastiCache holds session pins.

• The classifier runs on its own model, inside AWS. It has a 1.5-second timeout, so it needs a model that answers quickly every time, and it shares the gateway’s credentials and fallbacks. In one early run its calls kept running past the timeout, the router fell back to the heuristic scorer for most decisions, and accuracy dropped to 57 of 120. Giving the classifier its own model fixed that.
• Amazon ECS on AWS Fargate, with a database and a cache. One LiteLLM worker per 1 vCPU, 4 GiB task, scaling on CPU, as the LiteLLM production guide recommends. Amazon Aurora Serverless v2 with PostgreSQL holds virtual keys, budgets, and spend logs. Amazon ElastiCache shares router state across tasks.
• Private by default. An internal load balancer, an interface VPC endpoint to Amazon Bedrock, and a task role that can invoke only the four inference profiles.

Three lessons from building it:

• AWS WAF’s core rule set blocks large prompts. It blocks request bodies over 8 KB, a fixed limit on an Application Load Balancer (AWS WAF baseline rule groups). I use a per-IP rate-based rule and the known-bad-inputs rule group instead.
• Pin and verify the gateway image. On March 24, 2026, LiteLLM 1.82.7 and 1.82.8 on PyPI contained credential-stealing code (LiteLLM security update). I pin the image by digest and check its cosign signature with LiteLLM’s public key before each deploy, because Amazon ECS doesn’t check image signatures.
• Require TLS on the database yourself. Aurora PostgreSQL 16 and older doesn’t require TLS by default (Aurora PostgreSQL security). I set rds.force_ssl to 1 in a cluster parameter group, and the gateway connects with sslmode=verify-full against the RDS CA bundle.

Results

I ran a test suite through the gateway from inside the VPC, with two gateway tasks. It routed 120 prompts that I wrote and labeled with the cheapest tier that should answer correctly, 40 per tier. The set is synthetic and evenly split, so it shows how the router behaves, not your traffic mix.

Routing accuracy

The run landed 114 of 120 prompts on their labeled tier. 1.7 percent went to a cheaper tier and 3.3 percent to a more expensive one. The Nova 2 Lite classifier made 145 of 150 routing decisions. One classifier call timed out, the circuit opened, and the heuristic scorer made the other five. Two earlier runs, with Nova 2 Lite on Tier 1 and the 5.0 Claude models, landed 112 of 120.

• Short code requests go up a tier. All three over-routed Tier 2 prompts ask for one artifact: a Go function, an IAM policy, and a retry function. The everyday-code test prompt went to Tier 3 three times out of five.
• Questions over supplied context go down. Two went to Tier 1. Whether that’s a miss depends on GPT-6 Luna’s answers, which this test doesn’t grade.
• One short rewrite went up: “Shorten this to under ten words” went to Tier 2.

Latency, pinning, and reasoning tokens

• The router adds 0.65 seconds at the median over a direct Tier 1 call, 1.48 against 0.83 seconds, classifier included, across 20 samples inside the VPC.
• Session pinning needs a shared cache past one task. In an earlier run with two tasks and no shared store, only two of six follow-ups stayed on their first-turn tier. With Amazon ElastiCache, all six stayed, including a “thanks!” that matches a Tier 1 keyword rule. The same follow-up sent without a session went to Tier 1 all three times.
• Agent turns stay put. After a tool result, the request stayed on the tier that started the task.
• Tier 1 skips reasoning. GPT-6 Luna reasons at medium effort by default. With reasoning_effort: none it returned zero reasoning tokens, called directly and through the router.
• Tier 2 runs at low effort. In my preflight check, Amazon Bedrock rejected thinking: disabled for Sonnet 5.5, so the tier uses reasoning_effort: low, the lowest value LiteLLM v1.103.2 accepts. It returned zero reasoning tokens on a short prompt. So did Opus 5.5, so a short prompt doesn’t show how much either model reasons on harder work.

Two decisions from the log

“Classify this ticket as Billing, Technical, or Account: ‘I was charged twice this month.’” stayed on Tier 1:

ComplexityRouter: routing decision cause=llm_classifier, tier=SIMPLE, score=n/a, signals=('llm-classifier:SIMPLE',), routed_model=tier1-fast

“Prove that the halting problem is undecidable”, the prompt the heuristic scorer sent to the cheapest tier, went to Tier 3:

ComplexityRouter: routing decision cause=llm_classifier, tier=REASONING, score=n/a, signals=('llm-classifier:REASONING',), routed_model=tier3-frontier

What it costs

On-demand Standard tier prices in US East (N. Virginia) for the US cross-Region inference profiles. Check Amazon Bedrock pricing for your Region.

The estimate assumes 55 percent of requests need Tier 1, 35 percent Tier 2, and 10 percent Tier 3, then applies the measured misroutes. Each request has 2,000 input and 400 output tokens, plus a classifier call of 600 input and 20 output tokens.

Routing costs 69 percent less than sending everything to Opus 5.5, and 37 percent less than sending everything to Sonnet 5.5. The classifier adds $0.25 per 1,000 requests and the misroutes add $0.20. Tier 1 costs $0.44 per 1,000 requests, so nearly the whole bill is Tiers 2 and 3, and every request kept off them counts. Routing stays cheaper than all Sonnet 5.5 until about 29 percent of requests need Tier 3, with the rest split evenly between Tiers 1 and 2. Past that, it buys quality, not savings. The estimate leaves out reasoning tokens, which bill as output on Tiers 2 and 3, prompt caching, and the gateway’s fixed monthly cost, which can outweigh the savings at low volume.

Build it yourself

You need model access to the four models in a US Region. To install and run LiteLLM, follow its production deployment guide and production best practices. This is what I added on top.

1/ Define the tiers and the classifier. Each is a LiteLLM model name that points at a US cross-Region inference profile, using LiteLLM’s Amazon Bedrock provider. Each max_tokens applies when the client sends none. Add the bedrock-auto entry from above to the same model_list. GPT-6 Luna also needs bedrock:InvokeModel on your account’s default Amazon Bedrock project, as its model card shows.

model_list:
  - model_name: tier1-fast
    litellm_params:
      model: bedrock/us.openai.gpt-6-luna
      aws_region_name: us-east-1
      reasoning_effort: "none"
      max_tokens: 2048
  - model_name: tier2-balanced
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-5-5
      aws_region_name: us-east-1
      reasoning_effort: "low"
      max_tokens: 4096
  - model_name: tier3-frontier
    litellm_params:
      model: bedrock/us.anthropic.claude-opus-5-5
      aws_region_name: us-east-1
      max_tokens: 8192
  - model_name: classifier-fast
    litellm_params:
      model: bedrock/us.amazon.nova-2-lite-v1:0
      aws_region_name: us-east-1

2/ Share state across tasks. Point the router at Redis, as the production guide shows:

router_settings:
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

3/ Choose how routed requests get their output limit. With max_tokens_from_tier_model: false, the client’s max_tokens goes through. To cap every routed answer no matter what the client asks for, write the tier inside tiers as {model_name: tier3-frontier, litellm_params: {max_tokens: 8192}}.

4/ Pin the model list. LiteLLM reads prices, output limits, and which Amazon Bedrock API to call from its model list. The list bundled in v1.103.2 doesn’t include GPT-6 Luna, Sonnet 5.5, or Opus 5.5, and for a model it doesn’t know, LiteLLM picks a different Bedrock API without warning. Each task downloads the list from a fixed LiteLLM commit at startup, checks its SHA-256 hash, and writes it over the bundled copy. LITELLM_LOCAL_MODEL_COST_MAP=True stops LiteLLM from replacing it with the latest list on a restart.

Clients then call the gateway with any OpenAI-compatible SDK, using model: bedrock-auto. I’ll link the CloudFormation template and the test suite behind these numbers here once they’re published.

What to watch for

• Auto Routing is beta in LiteLLM. Pin the image and rerun your tests after every upgrade.
• Check what happens to max_tokens on routed requests. In LiteLLM v1.103.2, the router replaces the client’s max_tokens with the routed model’s largest output size from its model list, 128,000 tokens for Sonnet 5.5 and Opus 5.5, unless max_tokens_from_tier_model is false. In an earlier run with Claude Opus 5 on Tier 3, a request for 16 tokens returned 12,826 tokens after 195 seconds. With the setting false, the same routed request returned 16. The setting is described in the router’s source code, not on the Auto Routing docs page.
• Plan quota for GPT-6 Luna. On the bedrock-runtime endpoint, each output token uses 10 tokens of quota (GPT-6 Luna model card), so a Tier 1 that takes most of the traffic can reach its limit before the bill suggests it should.
• Routing accuracy isn’t answer quality. Before production, compare routed answers with Tier 3 answers on a sample of your own prompts.
• Tune the classifier’s rubric, after a licensing check. classifier_llm_config.system_prompt replaces the built-in rubric (LiteLLM announcement). Describing one function, query, or policy as MEDIUM targets the code over-routing. The rubric must define all four tiers, and if the classifier’s output stops parsing, the router falls back to the heuristic scorer. LiteLLM’s Auto Routing docs say the built-in LLM classifier needs no Enterprise license, but don’t say whether system_prompt does, so confirm with LiteLLM first. My tests used the built-in rubric.
• Alarm on classifier fallbacks. In the latest run, one classifier timeout opened the circuit and five decisions went to the heuristic scorer. Every request succeeded, so the only symptom was worse routing. Create a CloudWatch Logs metric filter on classifier-circuit-open and cause=heuristic_scorer.
• Count the model that answered. Fallbacks can move a request to another tier after the routing decision is logged, so base cost reports on responses or Amazon Bedrock invocation metrics.

Try it

Start with one internal application. Route it through bedrock-auto, label a few hundred of its real prompts, and measure the tier mix before you widen the rollout. If you try it, share your tier map and results in the comments.