Kubernetes webhooks and certs for those webhooks

Jamie Gaskins — Sun, 05 Feb 2023 23:07:44 +0000

I'm experimenting with running ARM-based workloads on GKE so I need to set nodeSelector: {kubernetes.io/arch: arm64} on every pod, including ones I don't own (various operators like Cert Manager and Nginx Ingress Controller). I've had to manually go in and add that field to every Deployment and StatefulSet so far manually.

Someone recommended using a MutatingWebhook to achieve that, so I'm learning about those and I have one deployed to my GKE cluster, but the request isn't making it to my HTTP server. The service reference docs don't specifically mention a requirement for TLS, but the URL-based webhook docs do. I imagine the service reference does require TLS, though, so maybe it's failing TLS verification? I wasn't using TLS at all, just plain HTTP. Does this seem right?

Also, if that's the case, I'm spoiled by Cert Manager and I never want to go back to dealing with CSRs manually, and every K8s webhook tutorial I can find seems to tell you to do just that with the openssl req CLI. So since I've already got Cert Manager running in the cluster, does it make sense to provision a Certificate resource (presumably something like my-webhook-service.my-webhook-namespace.svc.cluster.local for the TLS name?), then mount the resulting secret as a volume and load the .key file in the app? Or is there something easier?

[EDIT] Got it working! 🎉

Turns out, Cert Manager actually has a documentation page for this very specific use case. I'd always wondered what the cert-manager-cainjector deployment was for, tbh.

Anyway, here's what I did, with comments:

---
# Everything in a namespace so I could just delete
# it any time I needed
apiVersion: v1
kind: Namespace
metadata:
  name: webhook-test
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: pod-mutating-webhook.jgaskins.dev
  namespace: webhook-test
  annotations:
    # This is what tells CertManager to inject
    # the `caBundle` for certs we generate below
    cert-manager.io/inject-ca-from: webhook-test/pod-mutating-webhook
webhooks:
  - name: pod-mutating-webhook.jgaskins.dev
    admissionReviewVersions: [v1]
    rules:
      # We want all pods in the cluster to be passed
      # through this webhook just in case
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods]
    # If you're mutating Pod resources, you'll want to
    # Uncomment this so if you screw up so you can
    # deploy it again! This took a LOT of trial
    # and error!
    # failurePolicy: Ignore
    clientConfig:
      service:
        namespace: webhook-test
        name: pod-mutating-webhook
        path: /pods
        port: 3000
    sideEffects: None
---
# The service we wire up to our webhook handler
apiVersion: v1
kind: Service
metadata:
  namespace: webhook-test
  name: pod-mutating-webhook
spec:
  selector:
    app.kubernetes.io/name: pod-mutating-webhook
  ports:
  - port: 3000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: webhook-test
  name: pod-mutating-webhook
spec:
  selector:
    matchLabels: &labels
      app.kubernetes.io/name: pod-mutating-webhook
  template:
    metadata:
      labels: *labels
    spec:
      # Of course I had to add the nodeSelector
      # here, too 😂
      nodeSelector:
        kubernetes.io/arch: arm64
      containers:
      - name: web
        image: jgaskins/kubernetes-examples:mutating-webhooks
        imagePullPolicy: Always
        env:
        - name: LOG_LEVEL
          value: DEBUG
        ports:
        - protocol: TCP
          name: http
          containerPort: 3000
        # Mount the certs created by Cert Manager into
        # the app
        volumeMounts:
        - name: tls
          mountPath: /certs
          readOnly: true
      volumes:
      - name: tls
        secret:
          secretName: pod-mutating-webhook-tls

####### CERT STUFF ######
---
# The self-signed cert issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: webhook-test
spec:
  # Man, self-signed Issuers sure are simple, huh?
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: pod-mutating-webhook
  namespace: webhook-test
spec:
  # This must be the name of the secret that
  # you mount into your Deployment above!
  secretName: pod-mutating-webhook-tls
  # This must be `${serviceName}.${namespace}.svc`
  dnsNames:
  - pod-mutating-webhook.webhook-test.svc
  issuerRef:
    # Set this to your Issuer name above.
    # Must be in the same namespace if you're
    # not using a ClusterIssuer.
    name: selfsigned-issuer

The code for this example (written in Crystal) is here — the AdmissionReview response is generated in this method. When I deployed it and removed all of the manual kubernetes.io/arch: arm64 node selectors I added to every Deployment and StatefulSet, my MutatingWebhook server began properly adding them to the pods:

2023-02-06T06:59:56.451183Z   INFO - app: Listening on port 3000...
2023-02-06T07:03:13.916140Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (2.54ms)
2023-02-06T07:03:21.900212Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (357.68µs)
2023-02-06T07:03:24.440362Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (345.84µs)
2023-02-06T07:03:25.537122Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (301.28µs)
2023-02-06T07:03:26.530624Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (347.08µs)
2023-02-06T07:03:28.076697Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (321.72µs)
2023-02-06T07:03:28.520480Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (396.52µs)
2023-02-06T07:03:29.507916Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (381.16µs)
2023-02-06T07:03:44.978362Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (341.52µs)
2023-02-06T07:03:49.315628Z   INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (483.32µs)

The Ops Community ⚙️: Jamie Gaskins

Kubernetes webhooks and certs for those webhooks

[EDIT] Got it working! 🎉