Kong hybrid mode on Kubernetes with Helm, and what keeps working when the control plane doesn't

Hybrid mode splits Kong Gateway into two roles: a control plane that holds the configuration and serves the Admin API, and data planes that only proxy traffic. On Kubernetes, that’s two Helm releases and a shared certificate. It’s a small amount of configuration, and the details that bite are the ones around it: the certificate, the database, the licence, and what happens when the control plane isn’t there.

I first wrote this post in 2024. For this update I rebuilt the whole setup on a fresh cluster, configured it, sent traffic through it, then switched the control plane off to see what keeps working. Some things from the original no longer work, and I’ve noted those as I go.

Versions used: kind 0.33.0 · Kubernetes 1.34.0 · Helm 4.2.3 · kong/kong chart 3.4.1 · Kong Gateway 3.9.4 (open source) and Kong Gateway Enterprise 3.16.0.1 · decK 1.62.1

Why hybrid mode

In hybrid mode, only the control plane talks to the database. Data planes connect to the control plane over mutual TLS and receive their configuration from it (hybrid mode). That gives you three things:

A smaller attack surface on the edge. The nodes that face traffic have no database credentials and no Admin API.
Independent scaling. Add data planes for traffic without adding database connections.
Data planes anywhere. Different clusters, regions or networks can all take configuration from one control plane.

  decK / Admin API            Postgres
         |                       |
         v                       v
  +---------------+      +---------------+
  | control plane |------|   database    |
  +---------------+      +---------------+
         | config over mTLS (port 8005)
         v
  +---------------+   +---------------+
  |  data plane   |   |  data plane   |   <-- client traffic
  +---------------+   +---------------+

Choose your starting point first

Before any YAML, decide where your control plane lives, because it changes most of what follows.

Konnect. Kong runs the control plane and database for you; you run only data planes. Kong’s own docs call this the quickest way to start with hybrid mode (run Kong in hybrid mode). If you’re going to production, this is the option I’d evaluate first.
Self-managed Kong Gateway Enterprise, with a licence. Use the kong/kong-gateway image and load the licence as a Kubernetes secret (install on-prem with Helm).
Open-source Kong, to learn. The kong image supports hybrid mode without a licence. This is what the walkthrough below uses, so anyone can reproduce it.

What doesn’t work any more: the Enterprise image without a licence. “Free mode” has been deprecated since 3.10, and Kong says that once it’s removed, running without a licence behaves like an expired licence (breaking changes, 3.10). I tried Enterprise 3.16.0.1 without a licence: the control plane and data plane both started and connected, but every configuration write failed with HTTP 403 "Enterprise license missing or expired". You get a cluster you can’t configure.

Prerequisites

A Kubernetes cluster. I used kind on a laptop.
kubectl, Helm and openssl
decK, to configure the gateway
The Kong chart repository: helm repo add kong https://charts.konghq.com && helm repo update

Step 1: Namespace and cluster certificate

The control plane and data planes authenticate each other with a shared certificate. Generate it yourself, with the command from Kong’s install guide, and never commit the key to a repository:

kubectl create namespace kong

openssl req -new -x509 -nodes -newkey ec:<(openssl ecparam -name secp384r1) \
  -keyout ./tls.key -out ./tls.crt -days 1095 -subj "/CN=kong_clustering"

kubectl create secret tls kong-cluster-cert -n kong \
  --cert=./tls.crt --key=./tls.key

The certificate is valid for 1,095 days. Put a reminder in your calendar now.

An expired cluster certificate is a quiet way to lose every data plane at once.

Step 2: A database for the control plane

The control plane needs Postgres. The original version of this post used the chart’s built-in Postgres option. That no longer works out of the box: chart 3.4.1’s own values file notes that the default Bitnami Postgres image “is not available anymore” and asks you to supply your own.

For a lab, a plain Postgres deployment is simpler and easier to read. Create the password as a secret first, so it never appears in a values file:

kubectl create secret generic kong-pg -n kong \
  --from-literal=password="$(openssl rand -hex 16)"

# postgres.yaml - lab only: no persistence, no backups, no high availability.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: kong
spec:
  replicas: 1
  selector:
    matchLabels: {app: postgres}
  template:
    metadata:
      labels: {app: postgres}
    spec:
      containers:
        - name: postgres
          image: postgres:16
          env:
            - {name: POSTGRES_USER, value: kong}
            - {name: POSTGRES_DB, value: kong}
            - name: POSTGRES_PASSWORD
              valueFrom: {secretKeyRef: {name: kong-pg, key: password}}
          ports: [{containerPort: 5432}]
          readinessProbe:
            exec: {command: ["pg_isready", "-U", "kong"]}
            periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: kong
spec:
  selector: {app: postgres}
  ports: [{port: 5432, targetPort: 5432}]

kubectl apply -f postgres.yaml
kubectl rollout status deploy/postgres -n kong

Step 3: Install the control plane

# values-cp.yaml - holds config in Postgres, serves the Admin API,
# pushes config to data planes over mTLS. Serves no proxy traffic.
image:
  repository: kong
  tag: "3.9"

ingressController:
  enabled: false

secretVolumes:
  - kong-cluster-cert

env:
  role: control_plane
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  database: postgres
  pg_host: postgres.kong.svc.cluster.local
  pg_user: kong
  pg_database: kong
  pg_password:
    valueFrom:
      secretKeyRef: {name: kong-pg, key: password}

admin:
  enabled: true
  type: ClusterIP
  http:
    enabled: true

cluster:
  enabled: true
  tls:
    enabled: true

proxy:
  enabled: false

manager:
  enabled: false

helm install kong-cp kong/kong --version 3.4.1 -n kong -f values-cp.yaml --wait
kubectl get pods -n kong

The chart runs the database migrations as a job before the control plane starts, so you should see:

NAME                                 READY   STATUS      RESTARTS   AGE
kong-cp-kong-78659448d-bhbbw         1/1     Running     0          29s
kong-cp-kong-init-migrations-gzwt6   0/1     Completed   0          29s
postgres-5954f48f9c-tgrtb            1/1     Running     0          34s

A few deliberate choices: the Admin API is ClusterIP, so it’s not exposed outside the cluster, and there’s no Kong Manager and no proxy on this release. The control plane’s only jobs are configuration and the cluster endpoint on port 8005.

Step 4: Install the data plane

# values-dp.yaml - no database, gets config from the control plane,
# serves proxy traffic only.
image:
  repository: kong
  tag: "3.9"

ingressController:
  enabled: false

secretVolumes:
  - kong-cluster-cert

env:
  role: data_plane
  database: "off"
  cluster_control_plane: kong-cp-kong-cluster.kong.svc.cluster.local:8005
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt

proxy:
  enabled: true
  type: ClusterIP

admin:
  enabled: false

manager:
  enabled: false

helm install kong-dp kong/kong --version 3.4.1 -n kong -f values-dp.yaml --wait

Check that the data plane joined. Port-forward the Admin API and ask the control plane:

kubectl port-forward --address 127.0.0.1 -n kong \
  svc/kong-cp-kong-admin 28001:8001 &
curl -s http://127.0.0.1:28001/clustering/data-planes

You should see one entry with your data plane’s version and "sync_status": "normal".

Step 5: Configure it and send traffic

Configuration goes to the control plane, never to the data plane. A minimal decK file:

# kong.yaml
_format_version: "3.0"
services:
  - name: httpbin
    url: https://httpbin.org/anything
    routes:
      - name: httpbin
        paths: ["/anything"]
        methods: ["GET"]

deck gateway sync kong.yaml --kong-addr http://127.0.0.1:28001

Summary:
  Created: 2
  Updated: 0
  Deleted: 0

Then send a request through the data plane:

kubectl port-forward --address 127.0.0.1 -n kong \
  svc/kong-dp-kong-proxy 28000:80 &
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:28000/anything

200. When I added a second route with decK, it was answering on the data plane about two seconds later.

A tip if you run several local clusters: I once used port 18000, and my requests were answered by a different Kong, from another local cluster that already owned that port on 127.0.0.1, while my port-forward had bound only to IPv6. If you run several local clusters, bind port-forwards to 127.0.0.1 explicitly and check the Server header (curl -sI ... | grep -i server) before trusting a result.

What happens when the control plane goes away

This is the part I most wanted to test, because it’s the question every platform team asks: if the control plane is down, does traffic stop?

I scaled the control plane to zero and kept sending requests:

Situation	Result
Control plane down, existing data plane keeps running	Both routes kept returning 200. The data plane serves the last configuration it received
Control plane down, data plane pod replaced (as in a node drain or scale-up)	The new pod never became ready. Its readiness check returned 503 with `not ready for proxying: no configuration available (empty configuration present)`, and the proxy service had no ready endpoints
Control plane brought back	The new data plane became ready about 28 seconds later and served both routes

So the honest answer is: a control plane outage doesn’t stop running data planes, but it stops you from adding new ones. A rolling upgrade, a node drain or an autoscaling event during an outage removes capacity you can’t replace until the control plane is back.

Kong documents a feature for exactly this case: one node exports the configuration to storage, and data planes import it with cluster_fallback_config_import when the control plane is unreachable (control plane outage management). Kong’s reference notes that importing and exporting nodes must run exactly the same version, and recommends full image tags like 3.11.0.3 instead of 3.11 for that reason (configuration reference).

In production

The walkthrough above is a lab. For production, the same shape needs:

A control plane that isn’t a single point of failure. At least two control plane replicas, and a managed, backed-up Postgres instead of a single pod. Or let Konnect run it.
Data plane capacity that survives a drain. At least two replicas, a PodDisruptionBudget, and spare capacity, so that losing a node doesn’t depend on the control plane being reachable.
Pinned image tags. Use full versions (3.9.4, or 3.16.0.1 for Enterprise) rather than 3.9, so a pod restart never changes the version underneath you.
Upgrades in the right order. Kong is explicit: control plane nodes must be upgraded before data plane nodes (upgrade reference), and a data plane on a newer minor version than its control plane isn’t supported (version compatibility).
Certificates you can rotate. A shared certificate is the simplest option. Know when it expires, and practise replacing it before you have to. For larger estates, look at the PKI mode, where each data plane has its own certificate.
Network rules. Only data planes need to reach the control plane’s cluster port (8005), and nothing outside the cluster should reach the Admin API.
Monitoring. Alert on data planes whose sync_status isn’t normal, and on data plane readiness, not just pod liveness.

Trade-offs and when not to

Self-managed hybrid means operating a database and a control plane. That’s the real cost. If you don’t want to run Postgres with backups and upgrades, Konnect removes most of that work.
For a single small gateway, hybrid mode can be more than you need. DB-less mode with decK keeps the configuration in files and has no control plane to lose.
Kong Ingress Controller is a different pattern. Kong’s docs recommend hybrid mode without the ingress controller, or DB-less mode with it, unless Kong has advised otherwise (hybrid mode).

Troubleshooting

403 "Enterprise license missing or expired" on every change. You’re running the Enterprise image without a licence. Add the licence secret, or use the open-source image for learning.
The Postgres pod can’t pull its image. You enabled the chart’s built-in Postgres. Supply your own image or run Postgres separately, as above.
A data plane stays 0/1 ready, logging no configuration available. It can’t get configuration from the control plane. Check the control plane is up, the cluster_control_plane address, and that both sides use the same certificate.
A result that makes no sense locally. Check which gateway answered (Server header) and which process owns the port.

To clean up the lab: kind delete cluster --name <your-cluster>.

Takeaways

Hybrid mode on Kubernetes is two Helm releases and a shared certificate. Generate the certificate yourself and keep the key out of git.
Decide where the control plane lives first. Konnect removes the database and control plane work; self-managed Enterprise needs a licence; open source is for learning.
Running data planes keep serving when the control plane is down. New ones can’t start. Plan drains, upgrades and autoscaling around that, or set up fallback configuration.
Upgrade the control plane first, pin full image tags, and monitor sync status.

References

Hybrid mode, Kong docs
Run Kong Gateway in hybrid mode, Kong docs
Install Kong Gateway on-prem with Helm, Kong docs
Control plane outage management, Kong docs
Configuration reference: cluster fallback configuration, Kong docs
Upgrading Kong Gateway, Kong docs
Breaking changes, including free mode deprecation, Kong docs
kong/kong Helm chart, GitHub
kind

Why hybrid mode#

Choose your starting point first#

Prerequisites#

Step 1: Namespace and cluster certificate#

Step 2: A database for the control plane#

Step 3: Install the control plane#

Step 4: Install the data plane#

Step 5: Configure it and send traffic#

What happens when the control plane goes away#

In production#

Trade-offs and when not to#

Troubleshooting#

Takeaways#

References#