This article introduces a technique that helps you with debugging running Pods in production.
By changing labels, you can detach Pods from the Service (no traffic), and you troubleshoot them live.
Here's how it works.
Imagine you have a Deployment with three replicas.
Each Pod has an app=hello label.
A Service routes the traffic to your Pods using the selector app=hello
If you want to isolate a Pod you can overwrite the existing label with: kubectl label pod <pod-name> app=debug --overwrite
Two things happen next.
First, the Service stops routing traffic to the Pod because the Service's selector doesn't match the label.
Then, the ReplicaSet notices that there are only two replicas, but you asked for 3.
The ReplicaSet creates a new Pod.
You end up with 4 pods:
- 3 Pods running production traffic.
- A single isolated pod that used to receive traffic, but doesn't anymore.
The pod is still running and you can inspect its state.
What tool should you use?
For simple tasks, you could use:
-
kubectl exec, to attach to the container. -
kubectl port-forward, to tunnel the traffic. -
kubectl debugto run a sidecar container alongside the existing one.
But there are more options.
Inspector gadget is a tool designed to introspect and debug Kubernetes applications using eBPF.
If you use VS Code, you might like the official Kubernetes extension.
In this tutorial, you'll learn how to use it to attach the debugger to the application in the Pod.
If you feel at home in the command line, you should check out netshoot — a Docker + Kubernetes network trouble-shooting swiss-army container.
Which, of course, you can combine it with kubectl debug.
If you need more help to troubleshoot your deployments in Kubernetes, you should check out my flowchart:
And finally, if you've enjoyed this thread, you might also like the Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training or this collection of past Twitter threads https://twitter.com/danielepolencic/status/1298543151901155330
Top comments (0)