I'm experimenting with running ARM-based workloads on GKE so I need to set nodeSelector: {kubernetes.io/arch: arm64} on every pod, including ones I don't own (various operators like Cert Manager and Nginx Ingress Controller). I've had to manually go in and add that field to every Deployment and StatefulSet so far manually.
Someone recommended using a MutatingWebhook to achieve that, so I'm learning about those and I have one deployed to my GKE cluster, but the request isn't making it to my HTTP server. The service reference docs don't specifically mention a requirement for TLS, but the URL-based webhook docs do. I imagine the service reference does require TLS, though, so maybe it's failing TLS verification? I wasn't using TLS at all, just plain HTTP. Does this seem right?
Also, if that's the case, I'm spoiled by Cert Manager and I never want to go back to dealing with CSRs manually, and every K8s webhook tutorial I can find seems to tell you to do just that with the openssl req CLI. So since I've already got Cert Manager running in the cluster, does it make sense to provision a Certificate resource (presumably something like my-webhook-service.my-webhook-namespace.svc.cluster.local for the TLS name?), then mount the resulting secret as a volume and load the .key file in the app? Or is there something easier?
[EDIT] Got it working! 🎉
Turns out, Cert Manager actually has a documentation page for this very specific use case. I'd always wondered what the cert-manager-cainjector deployment was for, tbh.
Anyway, here's what I did, with comments:
---
# Everything in a namespace so I could just delete
# it any time I needed
apiVersion: v1
kind: Namespace
metadata:
name: webhook-test
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: pod-mutating-webhook.jgaskins.dev
namespace: webhook-test
annotations:
# This is what tells CertManager to inject
# the `caBundle` for certs we generate below
cert-manager.io/inject-ca-from: webhook-test/pod-mutating-webhook
webhooks:
- name: pod-mutating-webhook.jgaskins.dev
admissionReviewVersions: [v1]
rules:
# We want all pods in the cluster to be passed
# through this webhook just in case
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
# If you're mutating Pod resources, you'll want to
# Uncomment this so if you screw up so you can
# deploy it again! This took a LOT of trial
# and error!
# failurePolicy: Ignore
clientConfig:
service:
namespace: webhook-test
name: pod-mutating-webhook
path: /pods
port: 3000
sideEffects: None
---
# The service we wire up to our webhook handler
apiVersion: v1
kind: Service
metadata:
namespace: webhook-test
name: pod-mutating-webhook
spec:
selector:
app.kubernetes.io/name: pod-mutating-webhook
ports:
- port: 3000
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: webhook-test
name: pod-mutating-webhook
spec:
selector:
matchLabels: &labels
app.kubernetes.io/name: pod-mutating-webhook
template:
metadata:
labels: *labels
spec:
# Of course I had to add the nodeSelector
# here, too 😂
nodeSelector:
kubernetes.io/arch: arm64
containers:
- name: web
image: jgaskins/kubernetes-examples:mutating-webhooks
imagePullPolicy: Always
env:
- name: LOG_LEVEL
value: DEBUG
ports:
- protocol: TCP
name: http
containerPort: 3000
# Mount the certs created by Cert Manager into
# the app
volumeMounts:
- name: tls
mountPath: /certs
readOnly: true
volumes:
- name: tls
secret:
secretName: pod-mutating-webhook-tls
####### CERT STUFF ######
---
# The self-signed cert issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned-issuer
namespace: webhook-test
spec:
# Man, self-signed Issuers sure are simple, huh?
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: pod-mutating-webhook
namespace: webhook-test
spec:
# This must be the name of the secret that
# you mount into your Deployment above!
secretName: pod-mutating-webhook-tls
# This must be `${serviceName}.${namespace}.svc`
dnsNames:
- pod-mutating-webhook.webhook-test.svc
issuerRef:
# Set this to your Issuer name above.
# Must be in the same namespace if you're
# not using a ClusterIssuer.
name: selfsigned-issuer
The code for this example (written in Crystal) is here — the AdmissionReview response is generated in this method. When I deployed it and removed all of the manual kubernetes.io/arch: arm64 node selectors I added to every Deployment and StatefulSet, my MutatingWebhook server began properly adding them to the pods:
2023-02-06T06:59:56.451183Z INFO - app: Listening on port 3000...
2023-02-06T07:03:13.916140Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (2.54ms)
2023-02-06T07:03:21.900212Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (357.68µs)
2023-02-06T07:03:24.440362Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (345.84µs)
2023-02-06T07:03:25.537122Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (301.28µs)
2023-02-06T07:03:26.530624Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (347.08µs)
2023-02-06T07:03:28.076697Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (321.72µs)
2023-02-06T07:03:28.520480Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (396.52µs)
2023-02-06T07:03:29.507916Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (381.16µs)
2023-02-06T07:03:44.978362Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (341.52µs)
2023-02-06T07:03:49.315628Z INFO - http.server: 10.76.3.5 - POST /pods?timeout=10s HTTP/1.1 - 200 (483.32µs)
Top comments (2)
Awesome first post @jgaskins! Welcome the The Ops Community.
@jgaskins has arrived! Thanks for kicking things off with this helpful guide to how you solved your own problem, Jamie.