From plugin to entities: local Ollama behind Kong AI Gateway 2.x

In the previous post I ran a model locally with Ollama and put Kong in front of it with one ai-proxy plugin. That works, but the plugin config mixes three different questions into one block: where the model runs, what callers ask for, and who is allowed to call.

AI Gateway 2.x on Konnect splits those apart. You declare a provider (where the model runs), a model (the name callers use) and an auth strategy with consumers (who may call). This post moves the same local Ollama onto that model, end to end, with kongctl and a data plane in Docker.

Versions used: Kong AI Gateway 2.2.0 (kong/kong-ai-gateway:2.2.0) · kongctl 1.19.0 · Ollama 0.40.11 with llama3.2:1b · Konnect EU region

What changes from the plugin setup

AI Gateway 2.x is a dedicated control plane for AI traffic in Konnect, with first-class entities instead of plugins you assemble yourself (concepts). The data plane still runs wherever you run it; Konnect only sends it configuration and receives telemetry, and is never in the path of requests (architecture).

That last row is the point. Your application code asks for local-chat in OpenAI format. Whether that’s a 1B model on a laptop or a hosted model with fallback is the gateway’s business (AI Model).

What you need

• A Konnect account and a personal access token (PAT) in KONNECT_TOKEN.
• kongctl 1.19.0. I pin it per project with mise, as in this post: mise use github:Kong/[email protected].
• Docker with Compose v2, and openssl and jq.
• About 2 GB of RAM free for the model.

The examples use the EU region (https://eu.api.konghq.com). Use us, au or your own region if your org lives there.

1. A certificate for the data plane

The data plane proves who it is to Konnect with a client certificate. You register the public half on the gateway; the private key never leaves your machine (data plane certificates).

mkdir -p local-llm/certs && cd local-llm
openssl req -new -x509 -nodes -newkey rsa:2048 -days 365 \
  -subj "/CN=kong-dp" -keyout certs/tls.key -out certs/tls.crt

Use RSA as above. My first attempt used an EC key that openssl wrote with explicit curve parameters, and Konnect rejected it as “not valid ‘pem-encoded-cert’”.

2. Declare the gateway and its entities

Save this as ai-gateway.yaml. Everything lives under one AI Gateway, in a kongctl namespace so it doesn’t touch anything else in the org:

# ai-gateway.yaml
_defaults:
  kongctl:
    namespace: local-llm

ai_gateways:
  - ref: gw
    name: local-llm
    display_name: Local LLM
    deployment_type: hybrid

    # Lets a data plane holding the matching private key connect.
    data_plane_certificates:
      - ref: dp-cert
        title: local-dp
        cert: !file ./certs/tls.crt

    # Where the model runs. Ollama needs no credentials.
    model_providers:
      - ref: ollama
        name: ollama
        display_name: Ollama
        type: ollama
        config:
          auth:
            type: basic

    # How callers authenticate.
    auth_strategies:
      - ref: key-auth
        name: key-auth
        display_name: API key
        type: key-auth
        config:
          key_names: [apikey]
          hide_credentials: true

    # What callers see: one OpenAI-style model name.
    models:
      - ref: local-chat
        name: local-chat
        display_name: Local chat
        type: model
        enabled: true
        capabilities: [generate]
        formats:
          - type: openai
        config:
          balancer:
            algorithm: round-robin
            read_timeout: 120000   # ms; CPU inference is slow
          route:
            paths: [/v1]
            model:
              body_param: model
              values: [local-chat]
        access:
          auth_strategies: [key-auth]
        targets:
          - name: llama3.2:1b
            provider: ollama
            config:
              type: ollama
              upstream_url: http://ollama:11434

    consumers:
      - ref: app-a
        name: app-a
        display_name: App A
        type: api-key
        credentials:
          - ref: app-a-key
            name: app-a-key
            display_name: App A key
            type: api-key
            api_key: !secret {source: !env APP_A_KEY}
            ttl: 0

A few things worth knowing:

• type: basic with no headers is how a provider says “no credentials”. For a hosted provider you’d put the API key here, once, instead of in every plugin.
• upstream_url: http://ollama:11434 is the address as the data plane sees it. Here that’s the Ollama container on the same Docker network.
• route.model.values: [local-chat] is the name clients put in the request body. The target llama3.2:1b is what Ollama actually runs.
• read_timeout: 120000: the default upstream timeout is 60 seconds. On CPU, the first request after the model loads can take longer than that, and the gateway answers 504 Gateway Time-out. Two minutes is plenty for a 1B model.
• !secret {source: !env APP_A_KEY} reads the API key from your environment, so it never sits in the file.

Plan first, then apply the plan you reviewed:

export APP_A_KEY=$(openssl rand -hex 16)

kongctl plan --mode apply -f ai-gateway.yaml \
  --pat "$KONNECT_TOKEN" --base-url https://eu.api.konghq.com \
  --output-file plan.json

kongctl apply --plan plan.json \
  --pat "$KONNECT_TOKEN" --base-url https://eu.api.konghq.com

The plan for a fresh org lists seven creates: the gateway, the certificate, the provider, the auth strategy, the consumer, its credential and the model. I use apply rather than sync while experimenting: sync also deletes whatever it finds in the declared collections that isn’t in your file.

3. Run Ollama and the data plane

The data plane needs the gateway’s two endpoints, one for configuration and one for telemetry. Look them up by name:

curl -s -H "Authorization: Bearer $KONNECT_TOKEN" \
  "https://eu.api.konghq.com/v1/ai-gateways?page%5Bsize%5D=100" \
  | jq -r '.data[] | select(.name=="local-llm") | .endpoints'

Filter with select on the name as shown. Taking the first gateway in the list instead pointed my data plane at a different gateway in the org, and every connection failed with 401.

Write the two hosts, without https://, to a .env file next to the Compose file:

# .env
KONG_CLUSTER_CONTROL_PLANE=<id>.eu.cp.konghq.com:443
KONG_CLUSTER_SERVER_NAME=<id>.eu.cp.konghq.com
KONG_CLUSTER_TELEMETRY_ENDPOINT=<id>.eu.tp.konghq.com:443
KONG_CLUSTER_TELEMETRY_SERVER_NAME=<id>.eu.tp.konghq.com

Then compose.yaml:

services:
  ollama:
    image: ollama/ollama:0.40.11
    restart: unless-stopped
    volumes:
      - ollama:/root/.ollama

  kong:
    image: kong/kong-ai-gateway:2.2.0
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      KONG_ROLE: data_plane
      KONG_DATABASE: "off"
      KONG_KONNECT_MODE: "on"
      KONG_CLUSTER_MTLS: pki
      KONG_CLUSTER_CONTROL_PLANE: ${KONG_CLUSTER_CONTROL_PLANE}
      KONG_CLUSTER_SERVER_NAME: ${KONG_CLUSTER_SERVER_NAME}
      KONG_CLUSTER_TELEMETRY_ENDPOINT: ${KONG_CLUSTER_TELEMETRY_ENDPOINT}
      KONG_CLUSTER_TELEMETRY_SERVER_NAME: ${KONG_CLUSTER_TELEMETRY_SERVER_NAME}
      KONG_CLUSTER_CERT: /certs/tls.crt
      KONG_CLUSTER_CERT_KEY: /certs/tls.key
      KONG_LUA_SSL_TRUSTED_CERTIFICATE: system
    volumes:
      - ./certs:/certs:ro

volumes:
  ollama: {}

Ollama has no published port this time. Only the gateway can reach it, so the gateway’s API key is the only way in.

docker compose up -d
docker compose exec ollama ollama pull llama3.2:1b

Within a couple of seconds of starting, the data plane has its configuration and the /v1 route exists. If docker compose logs kong shows 401 on the cluster connection, the certificate or the endpoints don’t belong to this gateway.

4. Call the model

Without a key, and with a wrong one:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"local-chat","messages":[{"role":"user","content":"hi"}]}'

{"message":"No API key found in request"}

A wrong key returns 401 with {"message":"Unauthorized"}. With the consumer’s key:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -H "apikey: $APP_A_KEY" \
  -d '{"model":"local-chat","messages":[{"role":"user","content":"Name three HTTP methods."}]}'

{
  "object": "chat.completion",
  "model": "llama3.2:1b",
  "choices": [{"index": 0, "finish_reason": "stop",
    "message": {"role": "assistant", "content": "Here are three HTTP methods:\n\n1. GET\n2. POST\n3. PUT"}}],
  "usage": {"prompt_tokens": 30, "completion_tokens": 18, "total_tokens": 48}
}

The response header X-Kong-LLM-Model: ollama/llama3.2:1b shows which provider and model actually answered. The client asked for local-chat and never needed to know.

What this buys you, and what it doesn’t

• One place per concern. Provider credentials live on the provider, access rules on the model, keys on consumers. Rotating a provider key doesn’t touch any route.
• A stable name for clients. local-chat can later point at several targets, with load balancing, retries and fallback across providers (load balancing), without a client change.
• Per-consumer traffic. Every call is tied to a consumer, which is what rate limits, token budgets and cost reporting build on.
• The entities live in Konnect. Self-managed Kong runs the plugin form from the previous post; decK can convert the entity config into it (on-prem). Going the other way, the migration guide converts existing plugin config to entities.
• It’s more setup for one model. For a single local model on a laptop, the one-plugin version is less to read. The entity model pays off once you have more than one provider, consumer or team.

In production

• Data plane certificates: one per data plane fleet, so you can revoke one fleet without touching the others. Rotate by registering the new certificate first, rolling the data planes, then deleting the old one; deleting a certificate drops every data plane still using it (docs).
• Keys: keep them out of files (!secret from the environment or a secret manager) and give each application its own consumer.
• Timeouts: set read_timeout from your slowest realistic response, not from the default. A long answer from a hosted model can exceed 60 seconds too.
• Ollama in production wants a GPU host. The CPU numbers from the previous post don’t scale to real traffic.

Clean up

docker compose down -v

GW_ID=$(curl -s -H "Authorization: Bearer $KONNECT_TOKEN" \
  "https://eu.api.konghq.com/v1/ai-gateways?page%5Bsize%5D=100" \
  | jq -r '.data[] | select(.name=="local-llm") | .id')

curl -s -X DELETE -H "Authorization: Bearer $KONNECT_TOKEN" \
  "https://eu.api.konghq.com/v1/ai-gateways/$GW_ID"

Everything in this post was declared under the one AI Gateway, so deleting the gateway is the whole clean-up on the Konnect side.

Takeaways

• AI Gateway 2.x replaces one plugin block with a provider, a model, an auth strategy and consumers, each owning one concern.
• Clients call a stable model name in OpenAI format; the gateway decides which provider and model answer.
• Register the data plane’s certificate on the gateway, and point the data plane at that gateway’s endpoints.
• Raise read_timeout on the model when the backend is slow; the 60-second default returns 504 on a cold CPU model.
• For one local model, the plugin is simpler. The entities pay off with more providers, consumers or teams.

References

• AI Gateway 2.x concepts, architecture, get started
• Entities: AI Model, AI Model Provider, auth strategy, data plane certificate
• Ollama provider, load balancing and fallback
• Migrate to AI Gateway 2.x
• kongctl