The Ops Community ⚙️

Cover image for Isolating Kubernetes pods for debugging
Daniele Polencic
Daniele Polencic

Posted on

Isolating Kubernetes pods for debugging

This article introduces a technique that helps you with debugging running Pods in production.

By changing labels, you can detach Pods from the Service (no traffic), and you troubleshoot them live.

Isolating pods for troubleshooting

Here's how it works.

Imagine you have a Deployment with three replicas.

Each Pod has an app=hello label.

A Service routes the traffic to your Pods using the selector app=hello

A deployment with three replicas and a service

If you want to isolate a Pod you can overwrite the existing label with: kubectl label pod <pod-name> app=debug --overwrite

Two things happen next.

Changing the label to a Pod in a deployment

First, the Service stops routing traffic to the Pod because the Service's selector doesn't match the label.

The service stops routing traffic to the pod

Then, the ReplicaSet notices that there are only two replicas, but you asked for 3.

The ReplicaSet creates a new Pod.

The ReplicaSet creates a new Pod

You end up with 4 pods:

  • 3 Pods running production traffic.
  • A single isolated pod that used to receive traffic, but doesn't anymore.

The pod is still running and you can inspect its state.

What tool should you use?

For simple tasks, you could use:

  • kubectl exec, to attach to the container.
  • kubectl port-forward, to tunnel the traffic.
  • kubectl debug to run a sidecar container alongside the existing one.

But there are more options.

Inspector gadget is a tool designed to introspect and debug Kubernetes applications using eBPF.

If you use VS Code, you might like the official Kubernetes extension.

In this tutorial, you'll learn how to use it to attach the debugger to the application in the Pod.

If you feel at home in the command line, you should check out netshoot — a Docker + Kubernetes network trouble-shooting swiss-army container.

Which, of course, you can combine it with kubectl debug.

If you need more help to troubleshoot your deployments in Kubernetes, you should check out my flowchart:

Troubleshooting Kubernetes deployments flowchart

And finally, if you've enjoyed this thread, you might also like the Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training or this collection of past Twitter threads https://twitter.com/danielepolencic/status/1298543151901155330

Top comments (0)