Part 1: Hands-on Kong AI Gateway, your first governed LLM endpoint with Konnect

Hands-on Kong AI Gateway · Part 1. A series of curated, hands-on blog posts. Each post solves one real use case and shows how Kong AI Gateway secures, governs and scales AI traffic. The code is in kong-ai-gateway-zero-to-hero.

In this part you put an LLM behind Kong AI Gateway, lock it with an API key, and call it from your own code. Each app gets its own gateway key, and the provider’s key stays in the gateway, not in the app. The LLM is Ollama, because it runs in Docker without a paid key. Kong supports many other providers, and the last section shows where their credentials go.

The examples follow Acme, a fictional online retailer where every team used to hold its own provider key. Once the images are downloaded, a fresh clone gets to a passing test in under a minute (27 seconds in my run). The one-off download is about 3 GB, mostly the Ollama image.

Versions used: Kong AI Gateway 2.3.0 · kongctl 1.25.0 · Ollama 0.40.14 with qwen2.5:0.5b · Konnect EU region

Before you start

• Docker. I ran it with OrbStack on macOS; the images are published for amd64 and arm64. The two containers used about 1.2 GB of memory.
• A Konnect account and a personal access token. Konnect’s pricing page lists a 30-day free trial with no credit card required; check that your trial includes AI Gateway.
• mise 2026.9.9 or later. It installs the exact versions of kongctl (Kong’s CLI for Konnect) and the other tools the repository uses.
• git, make, curl and openssl.

Step 1. Clone and configure

git clone https://github.com/bhatikuldeep/kong-ai-gateway-zero-to-hero.git
cd kong-ai-gateway-zero-to-hero
cp .env.example .env
mise install

In .env, set KONNECT_TOKEN to your token and KONNECT_REGION to your organisation’s region, for example eu. If port 8000 is taken, change GATEWAY_PORT too.

Step 2. Start it

The repository gives each part its own folder with the same four commands: make up, make test, make down and make clean.

cd 01-setup
make up

make up first creates a certificate for the data plane and a key for the shop app. Then kongctl shows what it will create in Konnect and asks you to type yes: the gateway, its five resources and the shop app’s key. Last, it starts Kong and Ollama in Docker, so the output ends with:

Executed 7 changes.
...
 Container acme-aigw-kong-1 Healthy
Ready: run 'make test'

With the images already pulled, make up took about 25 seconds in my run, model download included. Run it again and it reports Konnect is up to date.

Step 3. Call it with and without a key

make test

PASS  no API key is rejected
PASS  API key in the URL is rejected
PASS  shop-app key gets an answer
PASS  answered by the Ollama model
PASS  token usage reported (48 tokens)
      round trip 1215 ms on this machine (model 1205 ms, gateway 5 ms)

The first call includes loading the model. After that, answers took 130 to 590 ms, of which Kong took 2 to 18 ms. Those are single requests on an Apple M5 Max, not a benchmark.

To call it with curl, from 01-setup (the first line reads the port and the shop app’s key from .env):

export $(grep -E '^(GATEWAY_PORT|SHOP_APP_KEY)=' ../.env)
curl -s -i "http://127.0.0.1:${GATEWAY_PORT:-8000}/v1/chat/completions" \
  -H 'content-type: application/json' \
  -H "apikey: ${SHOP_APP_KEY}" \
  -d '{"model": "acme-chat",
       "messages": [{"role": "user", "content": "Name one HTTP method."}]}'

HTTP/1.1 200 OK
X-Kong-LLM-Model: ollama/qwen2.5:0.5b
X-Kong-Upstream-Latency: 47
X-Kong-Proxy-Latency: 4

... "content":"One HTTP method is \"GET\"." ...
"usage":{"prompt_tokens":34,"completion_tokens":8,"total_tokens":42}

X-Kong-LLM-Model shows the model Kong selected. X-Kong-Proxy-Latency is the time Kong took before sending the request upstream, and X-Kong-Upstream-Latency the time until the model’s first byte (response headers). Send the same request without the apikey header and you get 401. Kong rejects it, and the model never sees the request.

From your own code

The gateway speaks OpenAI’s chat format, so the OpenAI SDK works. I expected the SDK’s usual Authorization: Bearer header to carry the key too; on its own it gets a 401. Send the key in the apikey header instead:

import os
from openai import OpenAI

client = OpenAI(
    base_url=f"http://127.0.0.1:{os.environ.get('GATEWAY_PORT', '8000')}/v1",
    api_key="not-used",  # the gateway reads the apikey header instead
    default_headers={"apikey": os.environ["SHOP_APP_KEY"]},
)
reply = client.chat.completions.create(
    model="acme-chat",
    messages=[{"role": "user", "content": "Name one HTTP method."}],
)
print(reply.choices[0].message.content)

Run it in the same shell as the export above, with the openai package installed; I used uv run --with openai python app.py. Streaming works too: add "stream": true to the request. In the SDK, stream=True makes the call return a stream of chunks instead of one reply, so loop over it rather than reading reply.choices[0].

How it works

Konnect is Kong’s hosted control plane, where the configuration comes from. The data plane is the Kong container you just started: it handles every request, and the Konnect control plane is never in the path of a request. If Konnect is unreachable, the data plane keeps serving its last configuration (architecture).

The gateway is made of five resources, which AI Gateway calls entities:

They all come from one file, konnect/ai-gateway.yaml. The model and the consumer carry the idea (abridged):

    models:
      - ref: setup-chat
        name: setup-chat
        config:
          route:
            paths: [/v1]
            model:
              body_param: model
              values: [acme-chat]
        access:
          auth_strategies: [setup-key-auth]
        targets:
          - name: qwen2.5:0.5b
            provider: setup-ollama

    consumers:
      - ref: shop-app
        name: shop-app
        type: api-key
        credentials:
          - ref: shop-app-key
            type: api-key
            api_key: !secret {source: !env SHOP_APP_KEY}

Apps ask for acme-chat. They don’t need to know which model or provider answers it.

values: [acme-chat] is the name apps ask for, and targets is what answers (AI Model). access.auth_strategies makes every call present a key. The shop app’s key comes from .env through !secret, so it never appears in the YAML or in git.

Use your own LLM

Ollama needs no credentials. A hosted provider does, and they go in the provider, not in your apps. This is adapted from Kong’s OpenAI example; as that page says, set OPENAI_API_KEY to Bearer <your-key>:

    model_providers:
      - ref: openai
        name: openai
        type: openai
        config:
          auth:
            type: basic
            headers:
              - name: Authorization
                value: !secret {source: !env OPENAI_API_KEY}

The model’s targets then name an OpenAI model and this provider, and the shop app keeps calling acme-chat with its own gateway key. I haven’t run the OpenAI path in this part; I plan to in a later one, with what it costs.

Details for operators

Before production

• The certificate is self-signed and valid for 365 days. Rotate it by registering a second one before deleting the first (AI Data Plane Certificate).
• Run several data plane nodes behind a load balancer. They’re stateless, so you scale by adding nodes (architecture).
• By default Konnect receives usage, cost and latency metadata, not request and response bodies. Two opt-in settings change that: log_payloads on a model puts full bodies into what attached logging policies receive, including Konnect-bound ones, and log_blocked_content on a guardrail policy forwards only the content that triggered a block (telemetry).
• Need it fully self-hosted? This post manages the entities in Konnect. Self-hosted Kong Gateway runs the same capabilities as AI plugins on Services and Routes, and decK converts AI Gateway 2.x entity files into that configuration (deck file ai2kong, deck ai sync) (AI Gateway on-prem).

Clean up

# stop the containers, keep the model and the gateway
make down
# remove everything, including the gateway in Konnect (asks first)
make clean

Checklist

• One AI Consumer and key per app, so you can revoke one without touching the others.
• Set key_in_query: false on every key-auth strategy.
• Provider keys live in the AI Model Provider, never in apps.
• Pin the data plane image by digest, and upgrade data planes before you raise min_runtime_version.

What’s next

Part 2 will put more than one model behind acme-chat. The plan after that: budgets and cost per team, guardrails, tools for AI agents over MCP, and finally CI, upgrades and Kubernetes, each building on this gateway.

References

• AI Gateway architecture
• AI Model
• AI Auth Strategies
• OpenAI provider
• AI Gateway changelog
• Declarative configuration with kongctl
• Companion repository