Prerequisites:

  • Familiarity with:
    • IP addresses and CIDR blocks
    • GKE & VPC in GCP

Context:

So, you want to host your workloads in GKE. You have two options:

  • Create a GKE cluster keeping the default values of IP CIDR for nodes, service and pods
  • Spend some time and understand how IP allocation works in GKE and create a scalable cluster

If you choose the first option, you might hit a scalability bottleneck as your application grows. If you choose the second option, you might still hit the bottleneck, but you'll know what went wrong.

Background:

  • A cluster in GKE is created within a subnet in a VPC network. IP address is allocated to Nodes, Pods & Services as following:

    • Node: A node is given an IP from the subnet's primary IP address range
    • Pod: A pod is given an IP from the subnet's secondary IP address range
    • Service: A service is given an IP from the subnet's secondary IP address range

  • The size of the IP address range determines how many nodes, pods and services can be created in a GKE cluster.

  • If there is no IP available in any of the IP address ranges, then your cluster has hit a rock bottom! ie, you now have to deal with IP address exhaustion issues.

Problems:

While working with GKE, you might face IP exhaustion issues of a few types:

  • Service IP range is exhausted
  • Pod IP range is exhausted
  • Node IP range is exhausted
  • Node IP range is available and so is pod IP range, but still new nodes are not getting created

Let's discuss each one of them.

Service IP range is exhausted

Service's subnet secondary range is immutable in a GKE cluster, so if it is exhausted, then you need to create a new GKE cluster!

Pod IP range is exhausted

You can add additional secondary IP address range for pods.

Node IP range is exhausted

You can expand the subnet's primary address range.

Node IP range is available and so is pod IP range, but still new nodes are not getting created

For example, if the subnet's primary IP address range is /24, then you will expect 252 nodes to be easily created. But you see that your cluster can't scale beyond (let's say) 64 nodes. You wonder why this happened!

The maximum number of pods and services that can be created in a GKE cluster depends on the subnet's secondary IP address range. But the maximum number of nodes that can be created depends on the following three factors:

  • Subnet's primary IP address range
  • Pod IP address range
  • Maximum pods per node

Let's take a look at how GKE allocates IP addresses to a node:

  • Based on the 'Maximum pods per node', GKE assigns the smallest subnet (from pod IP address range) that can fit double that number of IP addresses to the node.
  • In a standard GKE cluster, the default value of maximum pods per node is set to 110, and the smallest subnet mask that can accommodate 220 (110 X 2) IP addresses is /24. So, GKE carves out /24 from the pod IP address range and assigns it to the node.
  • This means that - if there is no available /24 slice in the pod IP address range and even if there is enough IP address in the primary IP address range, a node cannot be created in GKE.
Example:

Let's say a GKE cluster has following IP configuration:

  • Subnet primary IP range: 10.106.0.0/16
  • Pod IP range: 10.107.0.0/16
  • Service IP range: 10.108.0.0/16

Each CIDR block has 65536 IP addresses. To an uninformed, it'll look like ~65K nodes/pods/services can be created in the GKE. But as discussed above, GKE will reserve /24 slice for each node it creates, ie, it will reserve 256 pod IP addresses for each node. You can easily calculate the maximum total number of nodes that can be created with this configuration - it is 256 (65536/256 = 256)

Note: The reason why GKE allocates twice as many IP addresses as the maximum pods per node is that it reduces IP address reuse while pods gets added/removed from a node
Solutions:

Following are the possible solutions to this problem:

1. Add an additional secondary pod IP address range

It gives enough room for the new nodes to get their /24 slice from (based on the above example) and hence new nodes can be created

2. Reduce max-pod-per-node config

In the above example, what if you reduce the default max pod per node config to let's say 64, so GKE will now reserve /25, ie, it will reserve 128 pod IP addresses per node. This makes the maximum total number of nodes count to be 512 (65536/128).

Note: Max pod per node config is immutable at cluster and node pool level, but you can create a new node pool and define a custom value, and it will override the cluster level value.

Ref: