mTLS with Kong Gateway · Part 1. A short series on mutual TLS with Kong Gateway: clients to Kong, then Kong to the backend. The code is in kong-mtls.

The use case: client apps authenticate to an API with a client certificate. Kong checks the certificate and either accepts the call, as a known consumer, or rejects it. As a bonus, that consumer then works with the ACL plugin and with plugins that can be scoped to a consumer, such as per-client rate limits.

This post builds it with the mTLS Auth plugin, an Enterprise plugin that Konnect includes. A companion repository runs it. This is mTLS between client apps and Kong; the data plane’s own mTLS link to the Konnect control plane is separate. Part 2 covers Kong to the backend.

What this post builds

CallerCertificateResult
client-a, client-b, client-cfrom the client CA, mapped to a consumer200, and the backend sees which consumer called
client-dfrom the client CA, no consumer401
anyoneself-signed, or none401

The bonus adds gold, silver and bronze rate limits, and a route only gold can call.

How it works

Three parties take part: the client app that calls the API, Kong Gateway, and the backend API behind Kong. A client CA signs the client certificates. Three terms recur: a consumer is Kong’s record of one client app; the SAN (subject alternative name) and CN (common name) are the name fields of a certificate; the SNI is the hostname the client sends when it opens the TLS connection.

Set up once

mTLS Auth only looks consumers up and never creates them, so each client app needs a consumer before its first call. A consumer is mapped to a name on the certificate in one of two ways:

  • An mTLS credential on the consumer: subject_name is the certificate name, and ca_certificate pins it to one CA. The repository uses this.
  • A username or custom_id equal to the certificate name, matched by the plugin’s consumer_by setting, which is on by default. Less setup, but no CA pin.

With credentials only, set consumer_by: []. Otherwise a certificate from any trusted CA whose name equals a consumer’s username or custom_id gets in as that consumer, whatever its credential is pinned to.

On every call

  1. 1 to 3 in the diagram, the TLS handshake. Kong sends its own certificate and asks for the client’s. The client app sends client-a.crt and proves with its private key that it owns it.
  2. 4 and 5, the checks, in plugin priority order: mTLS Auth, then ACL, then Rate Limiting Advanced. The first no ends the call:
    • Trusted? The mTLS Auth plugin checks that a CA in its ca_certificates list signed the certificate and that it is in date.
    • Known consumer? Kong tries every SAN on the certificate, then the CN, and looks for a matching consumer.
    • Tier allowed? and Under the limit? come from the bonus: an ACL and a rate limit.
  3. 6 to 8, the answer. The backend learns who called from the X-Consumer-* headers, and the client app gets the reply.

Who issues the certificates

Two common setups:

  • The API team runs the client CA (recommended). Each client app sends a certificate signing request (CSR), and the API team signs it. The API team controls every name.
  • The client’s organisation runs its own CA. It shares its CA certificate, and the API team adds it to ca_certificates. Agree on the exact certificate name, the CA’s expiry date, and whether the CA publishes revocation (CRL or OCSP).

With more than one trusted CA, pin each consumer to its own CA with ca_certificate, and keep consumer_by: []. In a test, a second trusted CA issued a certificate with client-a’s name. With the pin it got 401; without it, it got in as client-a.

The CA certificate can be self-signed (every root CA is, including this lab’s), but it must have CA:TRUE: Konnect rejects one without it. A self-signed client certificate, like rogue in Step 2, is rejected.

Prerequisites

Docker, mise (it installs the pinned kongctl, deck and jq), openssl, curl, and a Konnect account with a personal access token. make up creates a control plane named mtls-identity in that account, and make clean deletes it.

Versions used: Kong Gateway 3.16.0.2 · decK 1.70.1 · kongctl 1.28.1 · Konnect (hybrid mode)

Step 1: Start the lab

git clone https://github.com/bhatikuldeep/kong-mtls
cd kong-mtls
mise install
eval "$(mise activate bash)"   # or zsh: so make uses the pinned tools
cp .env.example .env           # set KONNECT_TOKEN, and KONNECT_REGION if not eu
make up                        # certificates, Konnect control plane, data plane

KONNECT_REGION is the region of your Konnect organization: eu, us, au, me, in or sg. make up waits until the data plane has loaded the configuration from Konnect, then ends with:

Ready: run 'make test'

The backend is an echo service that returns the headers it received, so the output shows exactly what a real backend would get.

Step 2: Check the certificates

make up created the certificates in certs/ with scripts/certs.sh. The plain openssl equivalent is below for reference; there’s no need to run it. The CA, once:

openssl req -x509 -new -nodes -newkey rsa:2048 -days 30 \
  -subj "/CN=Example Client CA" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -keyout client-ca.key -out client-ca.crt

The client app creates its key and a CSR. The key never leaves the app:

openssl req -new -nodes -newkey rsa:2048 -subj "/CN=client-a.example.test" \
  -keyout client-a.key -out client-a.csr

The CA signs the CSR as a client certificate, with the name in the SAN:

openssl x509 -req -in client-a.csr -CA client-ca.crt -CAkey client-ca.key \
  -CAcreateserial -days 30 -out client-a.crt -extfile <(printf '%s\n' \
  'basicConstraints=CA:FALSE' \
  'keyUsage=critical,digitalSignature,keyEncipherment' \
  'extendedKeyUsage=clientAuth' 'subjectAltName=DNS:client-a.example.test')

This one is worth running. It makes the same chain check Kong makes:

openssl verify -CAfile certs/client-ca.crt certs/client-a.crt certs/rogue.crt
certs/client-a.crt: OK
CN = client-a.example.test
error 18 at 0 depth lookup:self signed certificate
certs/rogue.crt: verification failed: 18 (self signed certificate)

rogue.crt claims to be client-a, but it is self-signed. (Output from the macOS openssl, which is LibreSSL; OpenSSL 3 words it differently.)

Step 3: Read the Kong configuration

The configuration is two decK files: kong/certs.yaml, generated with the CA certificate (①), and kong/kong.yaml. Trimmed to one client app, the same CA ID appears three times:

ca_certificates:                       # ① the CA certificate (client-ca.crt, no key)
  - id: 5b1e0c62-6f0a-4c39-9a11-000000000001
    cert: |
      -----BEGIN CERTIFICATE-----
      ...
services:
  - name: orders-api
    url: http://orders:8080/anything
    plugins:
      - name: mtls-auth
        config:
          ca_certificates:             # ② trust certificates signed by ①
            - 5b1e0c62-6f0a-4c39-9a11-000000000001
          ssl_verify: true
          consumer_by: []              # match by credential only
          revocation_check_mode: SKIP  # the lab CA has no CRL or OCSP
    routes:
      - name: orders
        protocols: [https]
        snis: [api.example.test]
        paths: [/orders]
consumers:
  - username: client-a
    custom_id: app-1001
    mtls_auth_credentials:
      - id: 5b1e0c62-6f0a-4c39-9a11-0000000000a1   # decK needs a fixed ID
        subject_name: client-a.example.test         # the name on client-a.crt
        ca_certificate: { id: 5b1e0c62-6f0a-4c39-9a11-000000000001 }   # ③ pin to ①

Two settings are easy to get wrong:

  • snis on every route. Kong asks for the client certificate during the TLS handshake, before it knows the route, so it decides by SNI. If any route that mTLS Auth applies to has no SNI, Kong asks every client for a certificate, on every API on that gateway.
  • ssl_verify: true. It turns on TLS verification of the CRL and OCSP endpoints; it isn’t the client certificate check. From Kong Gateway 3.14, tls_certificate_verify is on by default, and the data plane then refuses mTLS Auth with ssl_verify off. Without this line, Konnect stored the plugin with ssl_verify: false, and the data plane rejected the whole configuration: it kept running but had no routes, and logged global tls_certificate_verify option is enabled, ssl_verify cannot be disabled.

kong/certs.yaml also holds the gateway’s own server certificate for api.example.test. In production, keep its key in a vault.

Step 4: Test it

--cert and --key send the client certificate. api.example.test isn’t a real hostname: --resolve points it at 127.0.0.1, and --cacert trusts the server certificate that make up created. The name matters, because the route only matches that SNI. With a different GATEWAY_PORT in .env, replace 8443 in both places:

api() { curl -s --resolve api.example.test:8443:127.0.0.1 \
  --cacert certs/gateway-ca.crt "$@"; }
u=https://api.example.test:8443

api -i --cert certs/client-a.crt --key certs/client-a.key $u/orders
HTTP/1.1 200 OK
...

Three calls that fail: no certificate, the self-signed rogue, and client-d (trusted CA, but no consumer):

s=' %{http_code}\n'
api -w "$s" $u/orders
api -w "$s" --cert certs/rogue.crt --key certs/rogue.key $u/orders
api -w "$s" --cert certs/client-d.crt --key certs/client-d.key $u/orders
{"message":"No required TLS certificate was sent"} 401
{"message":"TLS certificate failed verification"} 401
{"message":"TLS certificate failed verification"} 401

Client-d gets the same answer as the forged certificate. Only the data plane log tells them apart:

docker compose logs kong | grep 'consumer matching failed'
... [mtls-auth] certificate is valid but consumer matching failed, using cn = client-d.example.test ...

The backend learns who called from X-Consumer-* headers that Kong sets. A client app that sends its own X-Consumer-Username has it replaced:

api --cert certs/client-a.crt --key certs/client-a.key \
  -H 'X-Consumer-Username: admin' $u/orders \
  | jq '.headers | with_entries(select(.key | startswith("X-Consumer")))'
{
  "X-Consumer-Custom-Id": [
    "app-1001"
  ],
  "X-Consumer-Id": [
    "664dbe40-532b-4e49-99f7-588f69ee3697"
  ],
  "X-Consumer-Username": [
    "client-a"
  ]
}

(The consumer ID differs in each lab.) Then run every check, the bonus included (if an earlier run used up this minute’s rate limits, it waits for the next minute first):

make test
...
26 passed, 0 failed

Step 5 (bonus): Use the consumer in other plugins

Every call now carries a consumer. That makes three things available:

  • Plugins scoped to a consumer group or a consumer. Rate Limiting Advanced, Request Transformer, Response Transformer, IP Restriction and Proxy Cache accept both. HTTP Log and Request Size Limiting accept a consumer only. The supported scopes table lists every plugin.
  • The ACL plugin, which checks the consumer’s groups.
  • The backend, through the X-Consumer-* headers.

Two rules from the docs. Authentication plugins, mTLS Auth included, can’t be scoped to a consumer. And when a consumer is in two groups with the same plugin, only one of them runs, so keep each client app in one tier.

The repository shows two examples.

Gold, silver and bronze rate limits

A tier is a consumer group with its own Rate Limiting Advanced plugin. identifier: consumer gives each client app its own counter. A consumer joins a tier with groups: [{name: gold}]:

consumer_groups:
  - name: gold
    plugins:
      - name: rate-limiting-advanced
        config:
          identifier: consumer
          limit: [10]
          window_size: [60]
          window_type: fixed       # counters reset each minute
          namespace: tier-gold     # pin it, or decK reports drift
          strategy: local
  - name: silver                   # the same, with limit: [5]
  - name: bronze                   # the same, with limit: [3]

The default window type, sliding, also weighs the previous minute, so a client that hit its limit stays blocked for longer than a minute.

Client-c is bronze. make test used up its three calls, so wait a minute, then make four calls in a row:

for i in 1 2 3 4; do
  api -o /dev/null -w '%{http_code}\n' \
    --cert certs/client-c.crt --key certs/client-c.key $u/orders
done
200
200
200
429

The fourth call’s body is {"message":"API rate limit exceeded"}.

A route only gold can call

An ACL plugin on the /reports route with allow: [gold] and include_consumer_groups: true gives silver and bronze 403. include_consumer_groups is off by default; without it, the ACL ignores consumer groups.

api -w "$s" --cert certs/client-b.crt --key certs/client-b.key $u/reports
{
  "message":"You cannot consume this service",
  "request_id":"..."
} 403

Gold gets through, and the ACL tells the backend the consumer’s groups:

api --cert certs/client-a.crt --key certs/client-a.key $u/reports \
  | jq '.headers["X-Consumer-Groups"]'
[
  "gold"
]

Step 6: Clean up

make down    # remove the containers, keep Konnect and the certificates
make clean   # also delete the Konnect control plane and the certificates

Use it for your own API

  1. Choose the client CA (see Who issues the certificates). Kong needs only its certificate, with CA:TRUE. Trust only CAs that control both the SAN and the CN they issue. If an intermediate CA signs the client certificates, upload the root and have client apps send the intermediate with their certificate, or upload the intermediate and set allow_partial_chain: true. Either way, ca_certificate pins point to the CA certificate you uploaded.
  2. Add mTLS Auth to the service or route, as in Step 3: ca_certificates, ssl_verify: true, consumer_by: [], and every route with protocols: [https] and its snis. For revocation, the default revocation_check_mode, IGNORE_CA_ERROR, checks the CRL or OCSP responder a certificate names and lets the call through if that check can’t complete. STRICT rejects the call instead, so every client gets 401 while the responder is down.
  3. Create one consumer per client app, with an mTLS credential: subject_name is the name on the certificate, and ca_certificate pins it to its CA.
  4. Issue the certificates. Each client app sends a CSR, and the CA signs it with the name in the SAN. Use short validity with automated renewal, and keep the CA key off the gateway. A renewed certificate with the same name keeps working, because the mapping is by name. To cut one client off at once, delete its mTLS credential.
  5. Test three cases, as in Step 4: a mapped certificate gets 200, and no certificate or an unmapped one gets 401.
  6. Add consumer-based plugins as needed. Rate limit counters with strategy: local are per node, so three data plane nodes allow three times the limit. strategy: redis gives one shared counter, exact with sync_rate: 0; cluster isn’t available in hybrid mode or Konnect.

One thing to watch for: TLS must end at Kong. An L7 load balancer in front ends TLS, so Kong never sees the certificate. Use TCP passthrough, or have the edge forward the certificate to the Header Cert Auth plugin, and make sure only the edge can reach Kong, so no one else can send that header.

What’s next

Part 2 turns it around: Kong presents its own certificate to a backend that only accepts mTLS.

References