Hands-on Kong AI Gateway · Part 2. One Kong AI Gateway, from a first secured LLM endpoint to fallback, budgets and MCP, one runnable use case per part. The code is in kong-ai-gateway-zero-to-hero.
Acme’s shop app answers customer questions through acme-chat, the endpoint from Part 1. One model sits behind it, and that model is a single point of failure. When it’s down, overloaded or rate-limited, the shop’s chat breaks, and so does every other app built on the same model.
You could teach every app to retry with a second provider, in every language, each with its own provider key. Or you set up LLM fallback in the gateway once, and apps keep the same endpoint and key, ask for a model name that has a fallback, and the gateway decides which model answers. In this part the shop app gets a fallback from a model running locally to a hosted one. When you stop the local model, a request waits 2 seconds for it, later ones spend a few milliseconds in the gateway, and each worker process pays the wait on its own.
- Use it when an app has to keep answering through a model or provider outage.
- Skip it when every answer must come from one specific model, for example to keep data in one region.
At a glance
- You get: one model name,
acme-chat-fallback, that answers from Ollama and switches to Claude when Ollama fails, with the same endpoint and key. - You need: Part 1 running, and a Claude API key from the Claude Console on an organization that has API credits.
- Time: on my run,
make uptook 17 seconds andmake testabout 35 seconds. - Three things I didn’t expect: each worker process keeps its own failure count (Step 3); a 429 or 5xx from the first model only fails over with
non_idempotent(How it works); and Claude’s answers name your Anthropic account in their headers (Before production).
The first choice is Ollama, running locally in Docker next to the gateway. The fallback is Claude, through Anthropic’s API. Both are examples: any provider with a chat route can sit in either position, though setup differs per provider (AI providers).
How it works
An AI Model is what apps call by its model name, here acme-chat-fallback (fallback-chat is only the entity’s label in Konnect). Part 1’s acme-chat stays as it is, so apps opt in by asking for the new name. The model lists targets (a provider plus a model) and how to choose between them. Everything for this part is in one file (02-fallback/konnect/ai-gateway.yaml, routing and access left out):
ai_gateway_models:
- name: fallback-chat
config:
balancer:
algorithm: priority
failover_criteria: [error, timeout, http_429, http_5xx, non_idempotent]
connect_timeout: 2000 # ms
max_fails: 1
fail_timeout: 10000 # ms
targets:
- name: qwen2.5:0.5b
provider: setup-ollama # Part 1's provider
weight: 100 # first choice
- name: !env CLAUDE_MODEL # read from .env when you apply
provider: !ref fallback-anthropic#name
weight: 10 # fallback
config:
type: anthropic
max_tokens: 256
Order. With the priority algorithm, the balancer “always selects from the highest-priority group first” and “if all targets in that group are unavailable, it falls back to the next group” (load balancing). Here the weight is not a traffic share. The targets are grouped by weight (priority example), and the highest weight is tried first.
When to fail over. By default the gateway retries on error and timeout: a failure while connecting, sending the request or reading the response header, or one of those taking too long (retry and fallback). With Ollama’s container stopped, each request waited out connect_timeout before it failed over. For a model that answers but fails, add the status codes http_429 and http_5xx (http_5xx is new in 2.3.0, changelog).
The status codes alone weren’t enough. Chat requests are POSTs, and the AI Model docs say to add “non_idempotent for POST requests” (AI Model). With a stub as the first target that returns a fixed status, the difference is clear. Without non_idempotent, the app got the stub’s 429 and 503, and Claude was never called. With it, Claude answered both with 200. Connection failures fail over either way, because the request never reached the model. The cost is that a request that reached the first model can be sent again to the second, so the same prompt may be processed twice.
How fast. connect_timeout decides how long a dead model can hold a request. I set 2 seconds. After a failure, max_fails: 1 takes Ollama out of rotation for fail_timeout (10 seconds); after that the gateway tries it again, and if it answers it’s back in (circuit breaker).
One format. The app keeps sending OpenAI-format requests. An AI Model translates between the format chosen for it and each provider’s native format (AI Model), so Claude’s answer comes back in the shape the app already reads.
One request through the fallback, in order, with the times from my run:
Before you start
- Part 1 running: this part adds to its gateway and uses its containers.
- A Claude API key from the Claude Console, on an organization with API credits (see When the fallback fails). Claude is only called while Ollama is down, so a full test run uses a handful of short requests.
Versions used: Kong AI Gateway 2.3.0 · kongctl 1.25.0 · Ollama 0.40.14 with
qwen2.5:0.5b· Claudeclaude-haiku-5-5· Konnect EU region
Step 1. Add your Claude key
In the repository root, open .env (Part 1 created it, with your Konnect token) and set the key and the model:
ANTHROPIC_API_KEY=<your key>
CLAUDE_MODEL=claude-haiku-5-5
The key goes into the AI Model Provider in Konnect, never into the app.
Step 2. Add the fallback
cd 02-fallback
make up
kongctl shows the changes and asks you to type yes: a Claude provider, a policy that removes Anthropic’s account headers, and a model called acme-chat-fallback. They live in their own kongctl namespace, so make clean here removes only what this part added. The last lines:
[1/3] [namespace: acme-fallback] Creating ai_gateway_model_provider: fallback-anthropic... ✓
[2/3] [namespace: acme-fallback] Creating ai_gateway_policy: fallback-hide-provider-headers... ✓
[3/3] [namespace: acme-fallback] Creating ai_gateway_model: fallback-chat... ✓
make up then waits until the data plane answers for the new model. The data plane gets its configuration from Konnect, so a call right after the apply can still fail; on my first run, a test straight after the apply got a 500.
Step 3. Stop Ollama and watch the switch
make test
The test sends the same question several times. Between the calls it stops Part 1’s Ollama container, then starts it again:
PASS Ollama answers first
stopping Ollama ...
PASS Ollama down: Claude answers
first request after the outage: 2943 ms, of which 2004 ms waiting for Ollama (connect_timeout)
PASS Anthropic account headers removed
PASS request 2 during the outage: Claude answers
2724 ms, of which 2017 ms in the gateway
PASS request 3 during the outage: Claude answers
926 ms, of which 2 ms in the gateway
PASS request 4 during the outage: Claude answers
736 ms, of which 5 ms in the gateway
starting Ollama again ...
PASS Ollama back: it answers again
“In the gateway” is the X-Kong-Proxy-Latency header, the time between Kong receiving the request and sending it upstream (response headers). In my runs it also included the failed attempt on Ollama, the 2-second connect_timeout, so during an outage it isn’t gateway overhead. Claude itself answered in 700 to 950 ms. Those are single requests on an Apple M5 Max, not a benchmark.
Why several requests wait: one circuit breaker per worker process
I expected only the first request to wait, because max_fails is 1. The first two waited. Part 1’s data plane runs two worker processes (KONG_NGINX_WORKER_PROCESSES: "2" in its compose file), and each of them took Ollama out on its own. Which requests pay depends on which worker gets them.
To see how far that goes, I ran the same data plane on Kubernetes (kind): 2 replicas with 4 worker processes each, which follows Kong’s performance page: “We recommend setting this explicitly in Kubernetes”, to “a number matching the CPU limit. We recommend 4 or 8” (performance). With Ollama unreachable from the start, 24 requests one after another through a Kubernetes Service gave 18 log lines like these, from 8 different worker processes (2 pods, 4 each):
kong-dp-...-g5lcs [warn] 3006#0: ... [ai-proxy-advanced] target ollama:qwen2.5:0.5b has failed 1 times, will temporarily be considered unavailable
kong-dp-...-r4c6z [warn] 3007#0: ... [ai-proxy-advanced] target ollama:qwen2.5:0.5b has failed 1 times, will temporarily be considered unavailable
18 of the 24 requests paid the 2-second connect_timeout. Each worker skips Ollama for 10 seconds after its own failure, and with 8 workers sharing about one request every 2 seconds, most workers saw no request inside their 10 seconds, tried Ollama again and paid again. With more traffic the circuit breaker saves more, but every worker process in every pod can still pay up to once per fail_timeout (with max_fails: 1). The load balancing page describes the circuit breaker without mentioning worker processes, so plan for what the runs show. In an outage, connect_timeout and the number of worker processes set the latency, more than max_fails does.
The switch back is per worker too. A worker keeps sending requests to Claude until 10 seconds after its own last failure, even if Ollama is already running again.
To watch it yourself, send the Part 1 request with "model": "acme-chat-fallback" and look at X-Kong-LLM-Model and X-Kong-Proxy-Latency while you stop and start Ollama with docker compose -p acme-aigw stop ollama and start ollama.
See it in Grafana
With the optional Grafana stack from Part 1 running (make up in observability/), the switch shows on Kong’s AI Gateway dashboard. For 6 minutes I sent acme-chat-fallback one request at a time, with a 2-second pause after each answer, and stopped Ollama for the middle 2. All 128 requests got a 200: 98 from Ollama, 30 from Claude.
The circuit breaker shows up in the container logs, which the stack ships to Loki. In Explore > Loki, {container="kong"} |= "considered unavailable" gives a warning each time a worker process tried Ollama again: the first request it handled after its fail_timeout ran out. That was every 12 to 21 seconds per worker at this traffic, 15 warnings in the 2-minute outage, from process IDs 2985 and 2986.
An alert on this log line, or on requests per provider, tells you that traffic moved to the fallback before the bill does.
When the fallback fails: a 400 from Claude
My first run failed the hard way: Kong switched to Claude as configured, but my Anthropic organization had no API credits yet, and Claude refused every call. The app got Anthropic’s error as it was sent:
HTTP/1.1 400 Bad Request
{"type":"error","error":{"type":"invalid_request_error",
"message":"Your credit balance is too low to access the Anthropic API. ..."}}
http_400 wasn’t in my failover_criteria, so the gateway passed the error through. Test the fallback path on its own, not only the happy path, and alert on errors from the fallback model, because it only gets traffic when something else is already broken.
Before production
- Hide the provider’s account headers. Claude’s answers carry
anthropic-organization-idandanthropic-workspace-id, and without a policy Kong passes them to every app that calls the gateway. This part attaches a Response Transformer policy that removes both, to the fallback model only: not global, listed in the model’spolicies(AI Policies). Theanthropic-ratelimit-*headers andrequest-idstill pass; add them to the list if apps shouldn’t see your quota. - Size
connect_timeoutfor your worker processes. Every worker process in every data plane node can pay it once perfail_timeout(withmax_fails: 1). The default fornginx_worker_processesisauto(configuration reference), which tries to detect the CPU cores (nginx), so set it explicitly, as Kong’s performance page recommends for Kubernetes. - Hosted fallback means prompts leave your network while the local model is down. If some prompts must never leave, don’t give their model a hosted fallback.
- The fallback has its own limits and bill. A long outage of the first choice moves all traffic to it.
This part doesn’t cover streaming responses (every request here is non-streaming), or keeping the Claude key in your own secret store with an AI Vault instead of in Konnect.
Clean up
# remove what this part added (Part 1 stays); type yes again
make clean
# stop the containers (in Part 1's folder)
cd ../01-setup && make down
Checklist
- Give every AI Model that apps depend on a second target from another provider; the highest weight is tried first.
- For chat, add
non_idempotentnext tohttp_429andhttp_5xx, or those failures go back to the app. - Set
connect_timeoutfor an outage: each worker process in each node can pay it once perfail_timeout. - Remove the provider’s account headers before answers reach apps.
- Test the fallback on its own, and alert on errors from it.
What’s next
Part 3 puts a price on each model, Claude included, and gives each app token and spend limits that the gateway enforces. Part 4 puts your first MCP server behind the gateway, so an agent’s tool calls go through the same authentication and logging as its model calls.

