mTLS with Kong Gateway · Part 1. A short series on mutual TLS with Kong Gateway: clients to Kong, then Kong to the backend. The code is in kong-mtls.
The use case: client apps authenticate to an API with a client certificate. Kong checks the certificate and either accepts the call, as a known consumer, or rejects it. As a bonus, that consumer then works with the ACL plugin and with plugins that can be scoped to a consumer, such as per-client rate limits.
This post builds it with the mTLS Auth plugin, an Enterprise plugin that Konnect includes. A companion repository runs it. This is mTLS between client apps and Kong; the data plane’s own mTLS link to the Konnect control plane is separate. Part 2 covers Kong to the backend.
What this post builds
| Caller | Certificate | Result |
|---|---|---|
| client-a, client-b, client-c | from the client CA, mapped to a consumer | 200, and the backend sees which consumer called |
| client-d | from the client CA, no consumer | 401 |
| anyone | self-signed, or none | 401 |
The bonus adds gold, silver and bronze rate limits, and a route only gold can call.
How it works
Three parties take part: the client app that calls the API, Kong Gateway, and the backend API behind Kong. A client CA signs the client certificates. Three terms recur: a consumer is Kong’s record of one client app; the SAN (subject alternative name) and CN (common name) are the name fields of a certificate; the SNI is the hostname the client sends when it opens the TLS connection.
Set up once
mTLS Auth only looks consumers up and never creates them, so each client app needs a consumer before its first call. A consumer is mapped to a name on the certificate in one of two ways:
- An mTLS credential on the consumer:
subject_nameis the certificate name, andca_certificatepins it to one CA. The repository uses this. - A
usernameorcustom_idequal to the certificate name, matched by the plugin’sconsumer_bysetting, which is on by default. Less setup, but no CA pin.
With credentials only, set consumer_by: []. Otherwise a certificate from any trusted CA whose name equals a consumer’s username or custom_id gets in as that consumer, whatever its credential is pinned to.
On every call
- 1 to 3 in the diagram, the TLS handshake. Kong sends its own certificate and asks for the client’s. The client app sends
client-a.crtand proves with its private key that it owns it. - 4 and 5, the checks, in plugin priority order: mTLS Auth, then ACL, then Rate Limiting Advanced. The first no ends the call:
- Trusted? The mTLS Auth plugin checks that a CA in its
ca_certificateslist signed the certificate and that it is in date. - Known consumer? Kong tries every SAN on the certificate, then the CN, and looks for a matching consumer.
- Tier allowed? and Under the limit? come from the bonus: an ACL and a rate limit.
- Trusted? The mTLS Auth plugin checks that a CA in its
- 6 to 8, the answer. The backend learns who called from the
X-Consumer-*headers, and the client app gets the reply.
Who issues the certificates
Two common setups:
- The API team runs the client CA (recommended). Each client app sends a certificate signing request (CSR), and the API team signs it. The API team controls every name.
- The client’s organisation runs its own CA. It shares its CA certificate, and the API team adds it to
ca_certificates. Agree on the exact certificate name, the CA’s expiry date, and whether the CA publishes revocation (CRL or OCSP).
With more than one trusted CA, pin each consumer to its own CA with ca_certificate, and keep consumer_by: []. In a test, a second trusted CA issued a certificate with client-a’s name. With the pin it got 401; without it, it got in as client-a.
The CA certificate can be self-signed (every root CA is, including this lab’s), but it must have CA:TRUE: Konnect rejects one without it. A self-signed client certificate, like rogue in Step 2, is rejected.
Prerequisites
Docker, mise (it installs the pinned kongctl, deck and jq), openssl, curl, and a Konnect account with a personal access token. make up creates a control plane named mtls-identity in that account, and make clean deletes it.
Versions used: Kong Gateway 3.16.0.2 · decK 1.70.1 · kongctl 1.28.1 · Konnect (hybrid mode)
Step 1: Start the lab
git clone https://github.com/bhatikuldeep/kong-mtls
cd kong-mtls
mise install
eval "$(mise activate bash)" # or zsh: so make uses the pinned tools
cp .env.example .env # set KONNECT_TOKEN, and KONNECT_REGION if not eu
make up # certificates, Konnect control plane, data plane
KONNECT_REGION is the region of your Konnect organization: eu, us, au, me, in or sg. make up waits until the data plane has loaded the configuration from Konnect, then ends with:
Ready: run 'make test'
The backend is an echo service that returns the headers it received, so the output shows exactly what a real backend would get.
Step 2: Check the certificates
make up created the certificates in certs/ with scripts/certs.sh. The plain openssl equivalent is below for reference; there’s no need to run it. The CA, once:
openssl req -x509 -new -nodes -newkey rsa:2048 -days 30 \
-subj "/CN=Example Client CA" \
-addext "basicConstraints=critical,CA:TRUE" \
-addext "keyUsage=critical,keyCertSign,cRLSign" \
-keyout client-ca.key -out client-ca.crt
The client app creates its key and a CSR. The key never leaves the app:
openssl req -new -nodes -newkey rsa:2048 -subj "/CN=client-a.example.test" \
-keyout client-a.key -out client-a.csr
The CA signs the CSR as a client certificate, with the name in the SAN:
openssl x509 -req -in client-a.csr -CA client-ca.crt -CAkey client-ca.key \
-CAcreateserial -days 30 -out client-a.crt -extfile <(printf '%s\n' \
'basicConstraints=CA:FALSE' \
'keyUsage=critical,digitalSignature,keyEncipherment' \
'extendedKeyUsage=clientAuth' 'subjectAltName=DNS:client-a.example.test')
This one is worth running. It makes the same chain check Kong makes:
openssl verify -CAfile certs/client-ca.crt certs/client-a.crt certs/rogue.crt
certs/client-a.crt: OK
CN = client-a.example.test
error 18 at 0 depth lookup:self signed certificate
certs/rogue.crt: verification failed: 18 (self signed certificate)
rogue.crt claims to be client-a, but it is self-signed. (Output from the macOS openssl, which is LibreSSL; OpenSSL 3 words it differently.)
Step 3: Read the Kong configuration
The configuration is two decK files: kong/certs.yaml, generated with the CA certificate (①), and kong/kong.yaml. Trimmed to one client app, the same CA ID appears three times:
ca_certificates: # ① the CA certificate (client-ca.crt, no key)
- id: 5b1e0c62-6f0a-4c39-9a11-000000000001
cert: |
-----BEGIN CERTIFICATE-----
...
services:
- name: orders-api
url: http://orders:8080/anything
plugins:
- name: mtls-auth
config:
ca_certificates: # ② trust certificates signed by ①
- 5b1e0c62-6f0a-4c39-9a11-000000000001
ssl_verify: true
consumer_by: [] # match by credential only
revocation_check_mode: SKIP # the lab CA has no CRL or OCSP
routes:
- name: orders
protocols: [https]
snis: [api.example.test]
paths: [/orders]
consumers:
- username: client-a
custom_id: app-1001
mtls_auth_credentials:
- id: 5b1e0c62-6f0a-4c39-9a11-0000000000a1 # decK needs a fixed ID
subject_name: client-a.example.test # the name on client-a.crt
ca_certificate: { id: 5b1e0c62-6f0a-4c39-9a11-000000000001 } # ③ pin to ①
Two settings are easy to get wrong:
snison every route. Kong asks for the client certificate during the TLS handshake, before it knows the route, so it decides by SNI. If any route that mTLS Auth applies to has no SNI, Kong asks every client for a certificate, on every API on that gateway.ssl_verify: true. It turns on TLS verification of the CRL and OCSP endpoints; it isn’t the client certificate check. From Kong Gateway 3.14,tls_certificate_verifyisonby default, and the data plane then refuses mTLS Auth withssl_verifyoff. Without this line, Konnect stored the plugin withssl_verify: false, and the data plane rejected the whole configuration: it kept running but had no routes, and loggedglobal tls_certificate_verify option is enabled, ssl_verify cannot be disabled.
kong/certs.yaml also holds the gateway’s own server certificate for api.example.test. In production, keep its key in a vault.
Step 4: Test it
--cert and --key send the client certificate. api.example.test isn’t a real hostname: --resolve points it at 127.0.0.1, and --cacert trusts the server certificate that make up created. The name matters, because the route only matches that SNI. With a different GATEWAY_PORT in .env, replace 8443 in both places:
api() { curl -s --resolve api.example.test:8443:127.0.0.1 \
--cacert certs/gateway-ca.crt "$@"; }
u=https://api.example.test:8443
api -i --cert certs/client-a.crt --key certs/client-a.key $u/orders
HTTP/1.1 200 OK
...
Three calls that fail: no certificate, the self-signed rogue, and client-d (trusted CA, but no consumer):
s=' %{http_code}\n'
api -w "$s" $u/orders
api -w "$s" --cert certs/rogue.crt --key certs/rogue.key $u/orders
api -w "$s" --cert certs/client-d.crt --key certs/client-d.key $u/orders
{"message":"No required TLS certificate was sent"} 401
{"message":"TLS certificate failed verification"} 401
{"message":"TLS certificate failed verification"} 401
Client-d gets the same answer as the forged certificate. Only the data plane log tells them apart:
docker compose logs kong | grep 'consumer matching failed'
... [mtls-auth] certificate is valid but consumer matching failed, using cn = client-d.example.test ...
The backend learns who called from X-Consumer-* headers that Kong sets. A client app that sends its own X-Consumer-Username has it replaced:
api --cert certs/client-a.crt --key certs/client-a.key \
-H 'X-Consumer-Username: admin' $u/orders \
| jq '.headers | with_entries(select(.key | startswith("X-Consumer")))'
{
"X-Consumer-Custom-Id": [
"app-1001"
],
"X-Consumer-Id": [
"664dbe40-532b-4e49-99f7-588f69ee3697"
],
"X-Consumer-Username": [
"client-a"
]
}
(The consumer ID differs in each lab.) Then run every check, the bonus included (if an earlier run used up this minute’s rate limits, it waits for the next minute first):
make test
...
26 passed, 0 failed
Step 5 (bonus): Use the consumer in other plugins
Every call now carries a consumer. That makes three things available:
- Plugins scoped to a consumer group or a consumer. Rate Limiting Advanced, Request Transformer, Response Transformer, IP Restriction and Proxy Cache accept both. HTTP Log and Request Size Limiting accept a consumer only. The supported scopes table lists every plugin.
- The ACL plugin, which checks the consumer’s groups.
- The backend, through the
X-Consumer-*headers.
Two rules from the docs. Authentication plugins, mTLS Auth included, can’t be scoped to a consumer. And when a consumer is in two groups with the same plugin, only one of them runs, so keep each client app in one tier.
The repository shows two examples.
Gold, silver and bronze rate limits
A tier is a consumer group with its own Rate Limiting Advanced plugin. identifier: consumer gives each client app its own counter. A consumer joins a tier with groups: [{name: gold}]:
consumer_groups:
- name: gold
plugins:
- name: rate-limiting-advanced
config:
identifier: consumer
limit: [10]
window_size: [60]
window_type: fixed # counters reset each minute
namespace: tier-gold # pin it, or decK reports drift
strategy: local
- name: silver # the same, with limit: [5]
- name: bronze # the same, with limit: [3]
The default window type, sliding, also weighs the previous minute, so a client that hit its limit stays blocked for longer than a minute.
Client-c is bronze. make test used up its three calls, so wait a minute, then make four calls in a row:
for i in 1 2 3 4; do
api -o /dev/null -w '%{http_code}\n' \
--cert certs/client-c.crt --key certs/client-c.key $u/orders
done
200
200
200
429
The fourth call’s body is {"message":"API rate limit exceeded"}.
A route only gold can call
An ACL plugin on the /reports route with allow: [gold] and include_consumer_groups: true gives silver and bronze 403. include_consumer_groups is off by default; without it, the ACL ignores consumer groups.
api -w "$s" --cert certs/client-b.crt --key certs/client-b.key $u/reports
{
"message":"You cannot consume this service",
"request_id":"..."
} 403
Gold gets through, and the ACL tells the backend the consumer’s groups:
api --cert certs/client-a.crt --key certs/client-a.key $u/reports \
| jq '.headers["X-Consumer-Groups"]'
[
"gold"
]
Step 6: Clean up
make down # remove the containers, keep Konnect and the certificates
make clean # also delete the Konnect control plane and the certificates
Use it for your own API
- Choose the client CA (see Who issues the certificates). Kong needs only its certificate, with
CA:TRUE. Trust only CAs that control both the SAN and the CN they issue. If an intermediate CA signs the client certificates, upload the root and have client apps send the intermediate with their certificate, or upload the intermediate and setallow_partial_chain: true. Either way,ca_certificatepins point to the CA certificate you uploaded. - Add mTLS Auth to the service or route, as in Step 3:
ca_certificates,ssl_verify: true,consumer_by: [], and every route withprotocols: [https]and itssnis. For revocation, the defaultrevocation_check_mode,IGNORE_CA_ERROR, checks the CRL or OCSP responder a certificate names and lets the call through if that check can’t complete.STRICTrejects the call instead, so every client gets401while the responder is down. - Create one consumer per client app, with an mTLS credential:
subject_nameis the name on the certificate, andca_certificatepins it to its CA. - Issue the certificates. Each client app sends a CSR, and the CA signs it with the name in the SAN. Use short validity with automated renewal, and keep the CA key off the gateway. A renewed certificate with the same name keeps working, because the mapping is by name. To cut one client off at once, delete its mTLS credential.
- Test three cases, as in Step 4: a mapped certificate gets
200, and no certificate or an unmapped one gets401. - Add consumer-based plugins as needed. Rate limit counters with
strategy: localare per node, so three data plane nodes allow three times the limit.strategy: redisgives one shared counter, exact withsync_rate: 0;clusterisn’t available in hybrid mode or Konnect.
One thing to watch for: TLS must end at Kong. An L7 load balancer in front ends TLS, so Kong never sees the certificate. Use TCP passthrough, or have the edge forward the certificate to the Header Cert Auth plugin, and make sure only the edge can reach Kong, so no one else can send that header.
What’s next
Part 2 turns it around: Kong presents its own certificate to a backend that only accepts mTLS.