hugolesta.nl

$ cat posts/the-kafka-incident-that-was-just-a-certificate.md

The Kafka Incident That Was Just a Certificate (And the Operator That Saved MirrorMaker)

kafkakuberneteseksstrimziincident-responsetls

A Saturday afternoon alert: under-replicated partitions on the production Kafka cluster, well over the threshold. Status CRITICAL, replicas out of sync, risk of unavailability if a broker fails. The kind of page that makes you put down whatever you were holding.

The cluster is self-managed Kafka running on EKS, which means there is no AWS support engineer on the other side of this — whatever is happening, it's yours to diagnose.

By the time anyone looked at it, the number was zero. Nothing was broken. Nothing needed fixing. The cluster had rotated its own TLS certificates, restarted every broker one at a time exactly as designed, and gone back to green — and the monitor had caught the middle of that perfectly healthy operation and called it an emergency.

This is what that incident actually looked like from the inside, and why a 200-line Kubernetes operator nobody talks about was the reason the blast radius stopped where it did.

The architecture

All of this runs on Kubernetes — specifically Amazon EKS. Kafka is not a managed service here: the brokers are pods, the storage is EBS-backed PVCs, and the whole lifecycle is driven by the Strimzi operator. That matters for this story, because self-managing Kafka on Kubernetes means you inherit the operator's certificate lifecycle whether you think about it or not.

Two Kafka clusters, both Strimzi-managed, both in KRaft mode. A primary that serves production traffic, and a contingency cluster kept warm by MirrorMaker2. Each cluster lives in its own namespace, and MirrorMaker2 lives in a third.

Rendering diagram…

That last hop is the interesting one. MirrorMaker2 needs the primary cluster's CA certificate to trust its brokers — but Kubernetes secrets do not cross namespaces. There is no native mechanism that lets a pod in namespace C read a secret in namespace A. That gap is where Reflector lives.

What the alert actually was

Strimzi generates its own certificate authority for each Kafka cluster and rotates it automatically. With defaults, that means 365 days of validity and renewal starting 30 days before expiry. When renewal fires, every pod holding the old CA has to restart to pick up the new one.

The operator log is unambiguous about what happened:

16:02:35 INFO Ca:1035 - Kafka(kafka/primary): Renewing CA with subject=
  Subject(organizationName='io.strimzi', commonName='cluster-ca v0')
16:02:35 INFO Ca:1035 - Kafka(kafka/primary): Renewing CA with subject=
  Subject(organizationName='io.strimzi', commonName='clients-ca v0')

Four seconds later, the roll started. Every pod carried the same restart reason:

16:02:39 INFO KafkaRoller:473 - Kafka(kafka/primary): Rolling Pod primary-kafka-0/0 due to
  [Pod has old clients-ca certificate generation,
   Pod has old cluster-ca certificate generation,
   cluster-ca certificate renewal,
   clients-ca certificate renewal,
   Pod has old revision,
   Kafka broker TLS certificates updated]

The roll order was controllers first — 4, then 5, then 3 — followed by brokers 0, 1, 2. One pod at a time, which is the whole point of the KafkaRoller: it will not take down a second broker until the first is back and in-sync.

Full timeline, start to finish:

| Time (UTC) | Event | |---|---| | 16:02:35 | Operator renews cluster-ca and clients-ca | | 16:02:39 | First controller starts rolling | | 16:04:06 | First broker starts rolling | | 16:05:20 | Last broker starts rolling | | 16:16:29 | Monitor fires CRITICAL |

Read that last row again. The alert fired eleven minutes after the last pod restarted. The cluster was already healthy. The monitor was averaging under-replicated partitions over a 10-minute window, so it was still digesting the spike from a roll that had finished long before.

Verifying current state took two commands:

kubectl exec primary-kafka-0 -c kafka -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions | wc -l
# 0

kubectl exec primary-kafka-0 -c kafka -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --unavailable-partitions | wc -l
# 0

Zero and zero. Three brokers live, 241 topics, Kafka CR reporting Ready=True. The incident was over before it was reported.

The part that could have actually broken

Here is the failure mode that a CA rotation genuinely threatens: MirrorMaker2 authenticates to the primary cluster over TLS, using that cluster's CA as its truststore. Rotate the CA without giving MirrorMaker2 the new one, and replication dies at the handshake — every connection refused, replication lag climbing, and a contingency cluster quietly drifting out of date.

MirrorMaker2 runs in a different namespace from the cluster whose CA it needs. So how does it get the certificate?

Reflector. The source secret carries annotations that declare where it may be copied:

metadata:
  annotations:
    reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
    reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "kafka-mirrormaker"
    reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
    reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "kafka-mirrormaker"

And the mirrored copy in the MirrorMaker2 namespace records exactly what it is and when it last synced:

metadata:
  annotations:
    reflector.v1.k8s.emberstack.com/reflects: "kafka/primary-cluster-ca-cert"
    reflector.v1.k8s.emberstack.com/reflected-at: "2026-08-01T16:02:36.66+00:00"
    reflector.v1.k8s.emberstack.com/reflected-version: "615319895"
    reflector.v1.k8s.emberstack.com/auto-reflects: "True"

reflected-at: 16:02:36. The operator wrote the renewed secret at 16:02:35. Reflector propagated it one second later, and reflected-version matches the source secret's resourceVersion exactly — the copy is the new certificate, not a stale one.

The ordering is what makes this work:

  • 16:02:35 — operator renews the CA, writes the source secret
  • 16:02:36 — Reflector copies it into the MirrorMaker2 namespace
  • 16:04:28 — Strimzi rolls the MirrorMaker2 pod: [Pod has old revision]

The certificate was already in place before the pod restarted. MirrorMaker2 came back up with a truststore that already trusted the new CA. Not a single handshake error in the logs — no SSLHandshakeException, no PKIX path building failed, no unable to find valid certification path. The connector reconnected and kept replicating.

Worth being precise about the division of labor, because it is easy to give Reflector credit for the wrong thing: Reflector distributes, Strimzi rolls. Reflector never restarts a workload. It copies a secret and stops. Strimzi noticed the mounted secret had changed and rotated the pod to reload it. Two tools, two jobs, and the handoff between them is a one-second window that nobody had to orchestrate.

Why Reflector earns its place

The alternatives to Reflector are all worse:

Copy the secret manually. Works exactly once. The next CA rotation silently breaks replication, and you find out from a lag alert at 3am, or worse, from a failover that doesn't have the data you assumed it had.

Run a CronJob that re-copies it. Now you have a race between your copy interval and Strimzi's roll. If the CronJob runs every 15 minutes and Strimzi rolls MirrorMaker2 90 seconds after renewal, the pod comes up with a stale truststore and dies until the next tick.

Put everything in one namespace. Throws away the isolation that made you use separate namespaces in the first place — separate RBAC, separate quotas, separate blast radius.

Use a shared external CA. Reasonable at scale, but it means opting out of Strimzi's automatic cert management and owning that lifecycle yourself. That's a real project, not a config change.

Reflector replaces all of that with four annotations on a secret. It is declarative, it lives with the resource it describes, and it is fast enough that it beats the operator's own roll by a comfortable margin.

Field notes

  • A rolling restart and a real outage look identical to a threshold monitor. Under-replicated partitions spike during any healthy broker roll — upgrades, node drains, cert rotations. If your alert can't tell those apart, it will page you for planned work forever. Set maintenanceTimeWindows on the Kafka CR so renewals land in a known window, and align monitor downtime with it.

  • Check pod age before you check anything else. kubectl get pods showing an age of 17 minutes on a cluster that has been up for months tells you a roll just happened, which reframes the entire investigation in about two seconds. In this incident the contingency cluster's pods were 3 days old, which immediately proved it hadn't been touched.

  • Alert timestamps lag reality. A monitor averaging over 10 minutes fires up to 10 minutes after the condition clears. Always reconstruct the timeline from operator logs, never from when the alert arrived — otherwise you'll go hunting for a problem in a window where nothing was wrong.

  • reflected-version is the field that proves propagation worked. Compare it against the source secret's resourceVersion. If they match, the copy is current. If they don't, Reflector is behind and you have a real problem — this is the single most useful thing to check when debugging cross-namespace cert issues.

  • Grep for the right thing in operator logs. Searching for roll matches every reconciled and Reconciliation line and buries you in noise. Search for KafkaRoller:473 and Ca:1035 — the restart decisions and the CA renewals, nothing else.

  • Distinguish handshake failures from protocol errors. MirrorMaker2's logs had plenty of UnknownTopicOrPartitionException noise from the checkpoint connector, which looks alarming until you realize it proves the client is connected and authenticated — it's getting broker-level responses. A cert problem fails earlier, at the TLS handshake, and looks completely different. Don't let unrelated errors convince you the thing you're investigating is broken.

  • Clusters rotate independently. The contingency cluster's CA was created months after the primary's and was still on generation 0 — it hadn't rotated once. Which means the same roll, and the same alert, is still ahead of it. Knowing that in advance turns a future 2am page into a scheduled maintenance window.

Closing

The incident was a non-incident: a cluster doing exactly what it was configured to do, caught mid-stride by a monitor that measures symptoms instead of causes. The genuinely fragile part — a cross-namespace TLS dependency that breaks silently and shows up as replication lag hours later — held, because a small operator copied a secret one second after it changed.

That's the argument for Reflector in one sentence: it turns a cert rotation from something you have to remember into something you never think about again.