Restarting Kubernetes Pods When There Are New Secrets With Reloader

I will tell you a secret—no, a story. Say, at some point, I had a Kubernetes webhook admission controller that I wrote and deployed, and then the TLS certificate was automatically (nice!) renewed by cert-manager, but the pod wasn’t restarted, so it still had the old certificate, and now all Kubernetes deployments failed. That is indeed a story, perhaps a sad one. I had this shiny new cert, but no one was using it. Say I wanted to fix that. One way would be with Reloader.

Reloader

Reloader can watch changes in ConfigMap and Secret and do rolling upgrades on Pods with their associated DeploymentConfigs, Deployments, Daemonsets Statefulsets and Rollouts. - Reloader

Install Reloader

First add the repo.

$ helm repo add stakater https://stakater.github.io/stakater-charts
$ helm repo update

Create a namespace.

$ k create ns reloader
namespace/reloader created
$ kn reloader 
✔ Active namespace is "reloader"

Install reloader.

$ helm install reloader stakater/reloader
NAME: reloader
LAST DEPLOYED: Thu Nov 23 09:36:22 2023
NAMESPACE: reloader
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
- For a `Deployment` called `foo` have a `ConfigMap` called `foo-configmap`. Then add this annotation to main metadata of your `Deployment`
  configmap.reloader.stakater.com/reload: "foo-configmap"

- For a `Deployment` called `foo` have a `Secret` called `foo-secret`. Then add this annotation to main metadata of your `Deployment`
  secret.reloader.stakater.com/reload: "foo-secret"

- After successful installation, your pods will get rolling updates when a change in data of configmap or secret will happen.

Now we’ve got pods.

$ k get pods
NAME                                 READY   STATUS    RESTARTS   AGE
reloader-reloader-64df699b8d-tm5rn   1/1     Running   0          3m4s

Nice and easy. Thanks Helm!

Simple Test

Create a secret.

kubectl create secret generic foo-secret --from-literal=key1=bar

Create a cert-manager certificate. (Of course you need cert-manager installed.)

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: foo-certs
  namespace: foo
spec:
  secretName: foo-certs
  issuerRef:
    name: kubeadm-ca
    kind: ClusterIssuer
  duration: 24h  # Validity period of the certificate
  renewBefore: 12h 
  commonName: foo.foo.svc.cluster.local
  dnsNames:
    - foo.foo.svc.cluster.local
    - foo.foo.svc
EOF

Use that secret in a deployment. Note the annotation for Reloader. We’re mounting the secret in /etc/foo and certificates /etc/certs.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo
  labels:
    app: foo
  annotations:
    secret.reloader.stakater.com/reload: "foo-secret,foo-certs"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: foo
  template:
    metadata:
      labels:
        app: foo
    spec:
      containers:
      - name: my-container
        image: nginx
        volumeMounts:
        - name: secret-volume
          mountPath: "/etc/foo"
          readOnly: true
        - name: certs
          mountPath: "/etc/certs"
          readOnly: true
      volumes:
      - name: secret-volume
        secret:
          secretName: foo-secret
      - name: certs
        secret:
          secretName: foo-certs
EOF

Recreate the secret and check the logs of reloader.

$ kubectl create secret generic foo-secret --from-literal=key1=foo --dry-run=client -o yaml | kubectl apply -f -
Warning: resource secrets/foo-secret is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
secret/foo-secret configured

Reloader logs. It has noticed the secret update and restarted the pod.

$ k logs -n reloader reloader-reloader-64df699b8d-tm5rn 
time="2023-11-23T14:36:25Z" level=info msg="Environment: Kubernetes"
time="2023-11-23T14:36:25Z" level=info msg="Starting Reloader"
time="2023-11-23T14:36:25Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2023-11-23T14:36:25Z" level=info msg="created controller for: configMaps"
time="2023-11-23T14:36:25Z" level=info msg="Starting Controller to watch resource type: configMaps"
time="2023-11-23T14:36:25Z" level=info msg="created controller for: secrets"
time="2023-11-23T14:36:25Z" level=info msg="Starting Controller to watch resource type: secrets"
time="2023-11-23T15:18:53Z" level=info msg="Changes detected in 'foo-secret' of type 'SECRET' in namespace 'foo', Updated 'foo' of type 'Deployment' in namespace 'foo'"

New pod should be starting.

$ k get pods
NAME                   READY   STATUS        RESTARTS   AGE
foo-5c67d96557-s6cj2   1/1     Running       0          18s
foo-75cb458f7d-xcszx   1/1     Terminating   0          2m30s

Now it’s got the new secret.

$ k exec -it foo-5c67d96557-s6cj2 -- cat /etc/foo/key1
foo

Boom.

Certificates

Above we crated a certificate with only 24 hours of validity that should renew after 12 hours. So when it’s renewed, there will be a new version of the secret, and reloader will restart the pod. Let’s see.

$ k logs -n reloader reloader-reloader-7f4859f649-6cvqt 
time="2023-11-23T16:03:57Z" level=info msg="Environment: Kubernetes"
time="2023-11-23T16:03:57Z" level=info msg="Starting Reloader"
time="2023-11-23T16:03:57Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2023-11-23T16:03:57Z" level=info msg="created controller for: configMaps"
time="2023-11-23T16:03:57Z" level=info msg="Starting Controller to watch resource type: configMaps"
time="2023-11-23T16:03:57Z" level=info msg="created controller for: secrets"
time="2023-11-23T16:03:57Z" level=info msg="Starting Controller to watch resource type: secrets"
time="2023-11-23T16:06:18Z" level=info msg="Changes detected in 'foo-secret' of type 'SECRET' in namespace 'foo', Updated 'foo' of type 'Deployment' in namespace 'foo'"
time="2023-11-24T04:44:56Z" level=info msg="Changes detected in 'foo-certs' of type 'SECRET' in namespace 'foo', Updated 'foo' of type 'Deployment' in namespace 'foo'"

Looking at cert-manager logs we see:

I1124 04:44:56.006536       1 trigger_controller.go:194] "cert-manager/certificates-trigger: Certificate must be re-issued" key="foo/foo-certs" reason="Renewing" message="Renewing certificate as renewal was scheduled at 2023-11-24 04:44:56 +0000 UTC"
SNIP!
I1124 04:44:56.636293       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "foo-certs-jk5sq" condition "Ready" to 2023-11-24 04:44:56.636261134 +0000 UTC m=+4380108.430326366

Right, so the secret was updated. Let’s see if the pod was restarted.

$ k get pods
NAME                  READY   STATUS    RESTARTS   AGE
foo-746699dd7-kr99d   1/1     Running   0          6h43m
$ k describe pod foo-746699dd7-kr99d | grep -i started
      Started:      Thu, 23 Nov 2023 23:44:59 -0500

That time converts to 04:44:59 UTC, which is when the secret was updated. So it was restarted. This is great, so when a new certificate is issued, the pod will be restarted and mount the new secret and have access to the new certificate and key.

There’s a reloader annotation as well.

$ k get pods -oyaml | grep reloader
      reloader.stakater.com/last-reloaded-from: '{"type":"SECRET","name":"foo-certs","namespace":"foo","hash":"94af434fda756e922affdd1c43d723b26f196f3e","containerRefs":["my-container"],"observedAt":1700801096}'

Conclusion

Personally, I would think that this kind of thing would be automatic, but it’s not. So this is a good way to make sure that your pods are restarted when there are new secrets.

Kubernetes is a framework, and you have to pull in a lot of “libraries,” such as Reloader.