mTLS with Kong Gateway · Part 2. A short series on mutual TLS with Kong Gateway: clients to Kong, then Kong to the backend. The code is in kong-mtls.
The use case: a backend API only accepts callers that present a certificate from its own CA. Kong sits in front of it, so Kong must present one, and Kong should check the backend’s certificate too. As a bonus, Kong passes the client’s identity to the backend, but only on the routes that need it.
Part 1 covered client apps to Kong; this post uses the same lab.
What this post builds
| Call | Result |
|---|---|
Any caller → Kong → backend, /inventory | 200; the backend sees Kong’s certificate, CN=kong-gateway |
| Anyone → backend directly, no certificate | 400 No required SSL certificate was sent |
Bonus: client-a → Kong → backend, /inventory/whoami | 200; the backend also learns the consumer, client-a |
How it works
A backend CA signs two certificates: the one the backend serves, and the one Kong presents to the backend.
Set up once
Kong holds its client certificate and key as a Certificate entity, and the backend CA certificate as a CA Certificate entity; the service that points at the backend references both. In this lab, decK uploads them to Konnect, which pushes them to the data plane. The backend trusts only certificates from the backend CA.
On every call
- 1: the client app calls Kong.
/inventorydoesn’t authenticate it: mTLS to the backend proves who Kong is, not who the caller is. - 2 to 4, the TLS handshake between Kong and the backend. The backend presents its certificate and requests Kong’s. Kong checks that the backend CA signed it and that its name matches the service’s host,
inventory(502if not). Kong then sendskong-upstream.crtand proves it holds the key, and the backend checks it against the same CA (400if not). - 5 to 7, the request. The backend answers, and Kong returns the reply.
Who issues the certificates
The backend’s team runs the backend CA, because the backend decides whom it trusts. Kong’s team sends a certificate signing request (CSR) for Kong and gets back kong-upstream.crt. The backend team also shares its CA certificate, so Kong can check the backend.
Prerequisites
The same as Part 1: Docker, mise, openssl, curl, and a Konnect account with a personal access token. make up creates a control plane named mtls-identity, and make clean deletes it.
Versions used: Kong Gateway 3.16.0.2 · decK 1.70.1 · kongctl 1.28.1 · nginx 1.30.6 (the backend) · Konnect (hybrid mode)
Step 1: Start the lab
git clone https://github.com/bhatikuldeep/kong-mtls
cd kong-mtls
mise install
eval "$(mise activate bash)" # or zsh: so make uses the pinned tools
cp .env.example .env # set KONNECT_TOKEN, and KONNECT_REGION if not eu
make up # certificates, Konnect control plane, data plane
With the Part 1 lab already running, git pull and make up are enough. make up ends with Ready: run 'make test'.
Step 2: Check the certificates
make up created backend-ca, inventory (the backend’s certificate) and kong-upstream (Kong’s) in certs/. Check both against the backend CA:
openssl verify -CAfile certs/backend-ca.crt \
certs/kong-upstream.crt certs/inventory.crt
certs/kong-upstream.crt: OK
certs/inventory.crt: OK
Step 3: Read the Kong configuration
make up writes the two entities to kong/certs.yaml, with fixed IDs so the service can reference them:
ca_certificates:
- id: 5b1e0c62-6f0a-4c39-9a11-000000000003 # the backend CA (certificate only)
cert: |
-----BEGIN CERTIFICATE-----
...
certificates:
- id: 5b1e0c62-6f0a-4c39-9a11-000000000004 # Kong's client certificate and key
cert: |
...
key: |
...
The service in kong/kong.yaml points at them:
services:
- name: inventory-api
url: https://inventory:8443
client_certificate: 5b1e0c62-6f0a-4c39-9a11-000000000004 # what Kong presents
tls_verify: true # check the backend...
ca_certificates: [5b1e0c62-6f0a-4c39-9a11-000000000003] # ...against its CA
routes:
- name: inventory
protocols: [https]
snis: [api.example.test]
paths: [/inventory]
The backend is nginx (backend/nginx.conf). It requires a certificate from the backend CA and answers with the subject of the certificate it received and the X-Consumer-Username header, if any:
ssl_client_certificate /certs/backend-ca.crt;
ssl_verify_client on;
Two fields are easy to get wrong:
ca_certificates. From Kong Gateway 3.14, Kong checks the backend’s certificate by default, sotls_verify: trueonly makes it explicit.tls_verify: falseisn’t a way out: a data plane with the default settings rejects it. Without the backend CA, every call gets502, and the data plane logsupstream SSL certificate verify error: (21:unable to verify the first certificate). The backend’s certificate must also carry the service’s host as a name: Kong checks it, and the lab passes becauseinventory.crtnamesinventory.client_certificate. Without it, Kong presents no certificate, and the backend’s own400 No required SSL certificate was sentcomes back to the client.
Step 4: Test it
The same api helper as Part 1:
api() { curl -s --resolve api.example.test:8443:127.0.0.1 \
--cacert certs/gateway-ca.crt "$@"; }
u=https://api.example.test:8443
api $u/inventory
{"client_certificate":"CN=kong-gateway","consumer":""}
The backend saw Kong’s certificate. Now the backend directly, without a certificate and then with Kong’s (BACKEND_PORT in .env, 9443 by default; -k skips checking the backend’s certificate for this test):
curl -sk https://127.0.0.1:9443/ | grep -o '<title>[^<]*'
curl -sk --cert certs/kong-upstream.crt --key certs/kong-upstream.key \
https://127.0.0.1:9443/
<title>400 No required SSL certificate was sent
{"client_certificate":"CN=kong-gateway","consumer":""}
Whoever holds Kong’s key can call the backend, so protect it like a password. Then run every check (if an earlier run used up this minute’s rate limits, it waits for the next minute first):
make test
...
26 passed, 0 failed
Step 5 (bonus): Pass client identity only where needed
/inventory sends no identity: the backend saw "consumer":"". When the backend needs to know which client called, add Part 1’s mTLS Auth to the routes that need it. Kong then sends X-Consumer-Id, X-Consumer-Custom-Id and X-Consumer-Username on those routes, over its own certificate:
- name: inventory-whoami
protocols: [https]
snis: [api.example.test]
paths: [/inventory/whoami]
plugins:
- name: mtls-auth
config:
ca_certificates: [5b1e0c62-6f0a-4c39-9a11-000000000001] # Part 1's client CA
ssl_verify: true
consumer_by: []
revocation_check_mode: SKIP
api --cert certs/client-a.crt --key certs/client-a.key $u/inventory/whoami
{"client_certificate":"CN=kong-gateway","consumer":"client-a"}
The backend can trust these headers only on routes that authenticate. On /inventory, Kong passes a header the client sets itself, so the backend should read them only on /inventory/whoami:
api -H 'X-Consumer-Username: admin' $u/inventory
{"client_certificate":"CN=kong-gateway","consumer":"admin"}
Step 6: Clean up
make down # remove the containers, keep the Konnect control plane
make clean # also delete the Konnect control plane and the certificates
Use it for your own API
- Get Kong a certificate from the backend CA. In production, keep the key in a vault and reference it from the Certificate entity (
{vault://...}), so it isn’t in the config file. Renew it before it expires; the lab’s certificates last 30 days. - Set
client_certificateon each service that needs it; different backends can get different certificates. For one certificate on every service, setclient_ssl: on,client_ssl_certandclient_ssl_cert_keyin each data plane’skong.conf. They are read at startup, and a service’sclient_certificateoverrides them. - Set
ca_certificatesto the backend CA, and issue the backend’s certificate with the service’s host as a name. - Make the backend require a client certificate from that CA. If an intermediate CA signs Kong’s certificate, raise the backend’s verify depth (
ssl_verify_depthin nginx). - Authenticate callers where it matters, and pass identity only on those routes.