The Ops Community ⚙️

Cover image for IP and pod allocations in EKS
Daniele Polencic
Daniele Polencic

Posted on

IP and pod allocations in EKS

When running an EKS cluster, you might face two issues:

  • Running out of IP addresses assigned to pods.
  • Low pod count per node (due to ENI limits).

In this article, you will learn how to overcome those.

Before we start, here is some background on how intra-node networking works in Kubernetes.

When a node is created, the kubelet delegates:

  1. Creating the container to the Container Runtime.
  2. Attaching the container to the network to the CNI.
  3. Mounting volumes to the CSI.

The kubelet delegates tasks to the CRI, CNI and CSI

Let's focus on the CNI part.

Each pod has its own isolated Linux network namespace and is attached to a bridge.

The CNI is responsible for creating the bridge, assigning the IP and connecting veth0 to the cni0.

In most cases, all containers on a node are connected to a network bridge

This usually happens, but different CNIs might use other means to connect the container to the network.

As an example, there might not be a cni0 bridge.

The AWS-CNI is an example of such a CNI.

Not all CNI use a bridge to connect the containers on the same node

In AWS, each EC2 instance can have multiple network interfaces (ENIs).

You can assign a limited number of IPs to each ENI.

For example, an m5.large can have up to 10 IPs for ENI.

Of those 10 IPs, you have to assign one to the network interface.

The rest you can give away.

Elastic Network interfaces and IP addresses

Previously, you could use the extra IPs and assign them to Pods.

But there was a big limit: the number of IP addresses.

Let's have a look at an example.

With an m5.large, you have up to 3 ENIs with 10 IP private addresses each.

Since one IP is reserved, you're left with 9 per ENI (or 27 in total).

That means that your m5.large could run up to 27 Pods.

Not a lot.

You can have up to 27 pods in a m5.large

But AWS released a change to EC2 that allows "prefixes" to be assigned to network interfaces.

Prefixes what?!

In simple words, ENIs now support a range instead of a single IP address.

If before you could have 10 private IP addresses, now you can have 10 slots of IP addresses.

And how big is the slot?

By default, 16 IP addresses.

With 10 slots, you could have up to 160 IP addresses.

That's a rather significant change!

Let's have a look at an example.

Addresses prefix in EC2: before and after

With an m5.large, you have 3 ENIs with 10 slots (or IPs) each.

Since one IP is reserved for the ENI, you're left with 9 slots.

Each slot is 16 IPs, so 9*16=144 IPs.

Since there are 3 ENIs, 144x3=432 IPs.

You can have up to 432 Pods now (vs 27 before).

You can have up to 432 pods in a m5.large

The AWS-CNI support slots and caps the max number of Pods to 110 or 250, so you won't be able to run 432 Pods on an m5.large.

It's also worth pointing out that this is not enabled by default — not even in newer clusters.

Perhaps because only nitro instances support it.

Assigning slots it's great until you realize that the CNI gives 16 IP addresses at once instead of only 1, which has the following implications:

  • Quicker IP space exhaustion.
  • Fragmentation.

Let's review those.

Issue with prefixes in EC2 and EKS

A pod is scheduled to a node.

The AWS-CNI allocates 1 slot (16 IPs), and the pod uses one.

Now imagine having 5 nodes and a deployment with 5 replicas.

What happens?

The Kubernetes scheduler prefers to spread the pods across the cluster.

Likely, each node receives 1 pod, and the AWS-CNI allocates 1 slot (16 IPs).

You allocated 5*15=75 IPs from your network, but only 5 are used.

IP allocations with the AWS CNI

But there's more.

Slots allocate a contiguous block of IP addresses.

If a new IP is assigned (e.g. a node is created), you might have an issue with fragmentation.

How can you solve those?

Relevant links:

And finally, if you've enjoyed this thread, you might also like:

Top comments (1)

derlin profile image
Lucy Linder

Really interesting, and I love the illustrations ! Thank you for the sharing