mTLS with Kong Gateway · Part 2. A short series on mutual TLS with Kong Gateway: clients to Kong, then Kong to the backend. The code is in kong-mtls.

The use case: a backend API only accepts callers that present a certificate from its own CA. Kong sits in front of it, so Kong must present one, and Kong should check the backend’s certificate too. As a bonus, Kong passes the client’s identity to the backend, but only on the routes that need it.

Part 1 covered client apps to Kong; this post uses the same lab.

What this post builds

CallResult
Any caller → Kong → backend, /inventory200; the backend sees Kong’s certificate, CN=kong-gateway
Anyone → backend directly, no certificate400 No required SSL certificate was sent
Bonus: client-a → Kong → backend, /inventory/whoami200; the backend also learns the consumer, client-a

How it works

A backend CA signs two certificates: the one the backend serves, and the one Kong presents to the backend.

Set up once

Kong holds its client certificate and key as a Certificate entity, and the backend CA certificate as a CA Certificate entity; the service that points at the backend references both. In this lab, decK uploads them to Konnect, which pushes them to the data plane. The backend trusts only certificates from the backend CA.

On every call

  • 1: the client app calls Kong. /inventory doesn’t authenticate it: mTLS to the backend proves who Kong is, not who the caller is.
  • 2 to 4, the TLS handshake between Kong and the backend. The backend presents its certificate and requests Kong’s. Kong checks that the backend CA signed it and that its name matches the service’s host, inventory (502 if not). Kong then sends kong-upstream.crt and proves it holds the key, and the backend checks it against the same CA (400 if not).
  • 5 to 7, the request. The backend answers, and Kong returns the reply.

Who issues the certificates

The backend’s team runs the backend CA, because the backend decides whom it trusts. Kong’s team sends a certificate signing request (CSR) for Kong and gets back kong-upstream.crt. The backend team also shares its CA certificate, so Kong can check the backend.

Prerequisites

The same as Part 1: Docker, mise, openssl, curl, and a Konnect account with a personal access token. make up creates a control plane named mtls-identity, and make clean deletes it.

Versions used: Kong Gateway 3.16.0.2 · decK 1.70.1 · kongctl 1.28.1 · nginx 1.30.6 (the backend) · Konnect (hybrid mode)

Step 1: Start the lab

git clone https://github.com/bhatikuldeep/kong-mtls
cd kong-mtls
mise install
eval "$(mise activate bash)"   # or zsh: so make uses the pinned tools
cp .env.example .env           # set KONNECT_TOKEN, and KONNECT_REGION if not eu
make up                        # certificates, Konnect control plane, data plane

With the Part 1 lab already running, git pull and make up are enough. make up ends with Ready: run 'make test'.

Step 2: Check the certificates

make up created backend-ca, inventory (the backend’s certificate) and kong-upstream (Kong’s) in certs/. Check both against the backend CA:

openssl verify -CAfile certs/backend-ca.crt \
  certs/kong-upstream.crt certs/inventory.crt
certs/kong-upstream.crt: OK
certs/inventory.crt: OK

Step 3: Read the Kong configuration

make up writes the two entities to kong/certs.yaml, with fixed IDs so the service can reference them:

ca_certificates:
  - id: 5b1e0c62-6f0a-4c39-9a11-000000000003   # the backend CA (certificate only)
    cert: |
      -----BEGIN CERTIFICATE-----
      ...
certificates:
  - id: 5b1e0c62-6f0a-4c39-9a11-000000000004   # Kong's client certificate and key
    cert: |
      ...
    key: |
      ...

The service in kong/kong.yaml points at them:

services:
  - name: inventory-api
    url: https://inventory:8443
    client_certificate: 5b1e0c62-6f0a-4c39-9a11-000000000004   # what Kong presents
    tls_verify: true                                     # check the backend...
    ca_certificates: [5b1e0c62-6f0a-4c39-9a11-000000000003]   # ...against its CA
    routes:
      - name: inventory
        protocols: [https]
        snis: [api.example.test]
        paths: [/inventory]

The backend is nginx (backend/nginx.conf). It requires a certificate from the backend CA and answers with the subject of the certificate it received and the X-Consumer-Username header, if any:

ssl_client_certificate /certs/backend-ca.crt;
ssl_verify_client      on;

Two fields are easy to get wrong:

  • ca_certificates. From Kong Gateway 3.14, Kong checks the backend’s certificate by default, so tls_verify: true only makes it explicit. tls_verify: false isn’t a way out: a data plane with the default settings rejects it. Without the backend CA, every call gets 502, and the data plane logs upstream SSL certificate verify error: (21:unable to verify the first certificate). The backend’s certificate must also carry the service’s host as a name: Kong checks it, and the lab passes because inventory.crt names inventory.
  • client_certificate. Without it, Kong presents no certificate, and the backend’s own 400 No required SSL certificate was sent comes back to the client.

Step 4: Test it

The same api helper as Part 1:

api() { curl -s --resolve api.example.test:8443:127.0.0.1 \
  --cacert certs/gateway-ca.crt "$@"; }
u=https://api.example.test:8443

api $u/inventory
{"client_certificate":"CN=kong-gateway","consumer":""}

The backend saw Kong’s certificate. Now the backend directly, without a certificate and then with Kong’s (BACKEND_PORT in .env, 9443 by default; -k skips checking the backend’s certificate for this test):

curl -sk https://127.0.0.1:9443/ | grep -o '<title>[^<]*'
curl -sk --cert certs/kong-upstream.crt --key certs/kong-upstream.key \
  https://127.0.0.1:9443/
<title>400 No required SSL certificate was sent
{"client_certificate":"CN=kong-gateway","consumer":""}

Whoever holds Kong’s key can call the backend, so protect it like a password. Then run every check (if an earlier run used up this minute’s rate limits, it waits for the next minute first):

make test
...
26 passed, 0 failed

Step 5 (bonus): Pass client identity only where needed

/inventory sends no identity: the backend saw "consumer":"". When the backend needs to know which client called, add Part 1’s mTLS Auth to the routes that need it. Kong then sends X-Consumer-Id, X-Consumer-Custom-Id and X-Consumer-Username on those routes, over its own certificate:

      - name: inventory-whoami
        protocols: [https]
        snis: [api.example.test]
        paths: [/inventory/whoami]
        plugins:
          - name: mtls-auth
            config:
              ca_certificates: [5b1e0c62-6f0a-4c39-9a11-000000000001]   # Part 1's client CA
              ssl_verify: true
              consumer_by: []
              revocation_check_mode: SKIP
api --cert certs/client-a.crt --key certs/client-a.key $u/inventory/whoami
{"client_certificate":"CN=kong-gateway","consumer":"client-a"}

The backend can trust these headers only on routes that authenticate. On /inventory, Kong passes a header the client sets itself, so the backend should read them only on /inventory/whoami:

api -H 'X-Consumer-Username: admin' $u/inventory
{"client_certificate":"CN=kong-gateway","consumer":"admin"}

Step 6: Clean up

make down    # remove the containers, keep the Konnect control plane
make clean   # also delete the Konnect control plane and the certificates

Use it for your own API

  1. Get Kong a certificate from the backend CA. In production, keep the key in a vault and reference it from the Certificate entity ({vault://...}), so it isn’t in the config file. Renew it before it expires; the lab’s certificates last 30 days.
  2. Set client_certificate on each service that needs it; different backends can get different certificates. For one certificate on every service, set client_ssl: on, client_ssl_cert and client_ssl_cert_key in each data plane’s kong.conf. They are read at startup, and a service’s client_certificate overrides them.
  3. Set ca_certificates to the backend CA, and issue the backend’s certificate with the service’s host as a name.
  4. Make the backend require a client certificate from that CA. If an intermediate CA signs Kong’s certificate, raise the backend’s verify depth (ssl_verify_depth in nginx).
  5. Authenticate callers where it matters, and pass identity only on those routes.

References