The Ops Community ⚙️

Cover image for Tracing pod to pod network traffic in Kubernetes
Daniele Polencic
Daniele Polencic

Posted on

Tracing pod to pod network traffic in Kubernetes

How does Pod to Pod communication work in Kubernetes?

How does the traffic reach the pod?

In this article, you will dive into how low-level networking works in Kubernetes.

Let's start by focusing on the pod and node networking.

When you deploy a Pod, the following things happen:

  1. The pod gets its own network namespace.
  2. An IP address is assigned.
  3. Any containers in the pod share the same networking namespace and can see each other on localhost.

Network namespaces in a Kubernetes node

A pod must first have access to the node's root namespace to reach other pods.

This is achieved using a virtual eth pair connecting the 2 namespaces: pod and root.

The bridge allows traffic to flow between virtual pairs and traverse through the common root namespace.

Bridge that connects all containers in the node

So what happens when Pod-A wants to send a message to Pod-B?

Since the destination isn't one of the containers in the namespace Pod-A sends out a packet to its default interface eth0.

This interface is tied to the veth pair and packets are forwarded to the root namespace.

Tracing the flow: starting from the container

The ethernet bridge, acting as a virtual switch, has to somehow resolve the destination pod IP (Pod-B) to its MAC address.

The packet reaches the cni0 bridge

The ARP protocol comes to the rescue.

When the frame reaches the bridge, an ARP broadcast is sent to all connected devices.

The bridge shouts "Who has Pod-B IP address?"

ARP queries

A reply is received with the interface's MAC address that connects Pod-B, which is stored in the bridge ARP cache (lookup table).

ARP reply

Once the IP and MAC address mapping is stored, the bridge looks up in the table and forwards the packet to the correct endpoint.

The packet reaches Pod-B veth in the root namespace, and from there, it quickly reaches the eth0 interface inside the Pod-B namespace.

The packet reaches the other pod

With this, the communication between Pod-A and Pod-B has been successful.

An additional hop is required for pods to communicate across different nodes, as the packets have to travel through the node network to reach their destination.

Tracing pod traffic across nodes

This is the "plain" networking version.

How does this change when you install a CNI plugin that uses an overlay network?

Let's take Flannel as an example.

Flannel installs a new interface between the node's eth0 and the container bridge cni0.

All traffic flowing through this interface is encapsulated (e.g. VXLAN, Wireguard, etc.).

The Flannel interface encapsulates the traffic

The new packets don't have pods' IP addresses as source and destination, but nodes' IPs.

So the wrapper packet will exit from the node and travel to the destination node.

Packets are encpsulated using different backends

Once on the other side, the flannel.1 interface unwraps the packet and lets the original pod-to-pod packet reach its destination.

The packet is unwrapped and forwarded to the interface

How does Flannel know where all the Pods are located and their IP addresses?

On each node, the Flannel daemon syncs the IP addresses allocations in a distributed database.

Other instances can query this database to decide where to send those packets.

The flannel deaemons sync IP addresses to a (distributed) database

Here are a few links if you want to learn more:

And finally, if you've enjoyed this thread, you might also like:

Top comments (0)