Kubernetes and Networking from first principles

Its been some time since I wrote the Kubernetes Resource Management Deep dive. In this post I wanted to reduce Kubernetes to it very bare bones ignoring all the scaffolding introduced by various controllers and higher level abstractions and focus on the building blocks with specific focus on how networking works without networking being totally within the purview of core Kubernetes.

What is the core of Kubernetes once you strip everything way. It is just the few basic constructs in etcd

Managed and accessed by

Quick overview

etcd is used to create and update Pods, Endpoints, Services and Nodes. Everything else else is built on top of these core objects. etcd is always accessed via the apiserver making etcd pluggable/replaceable. Networking is not part of core Kubernetes which shows how flexible and modular it is.

Examining the Pods created with specific fields of interest

Now that a Pod has an IP assigned to it independent of Kubernetes; how do all the Pods in the cluster and external entities access this Pod.

Kube-proxy is optional and pluggable and ensures traffic to and from the Pod is sent to the right Pod. Note, I mentioned Pod hence PodIP. All other network addresses are constructs out the scope of CNI.

It is also decoupled from Coredns as well as CNI. As seen above the Pod’s IP and labels are registered in etcd. This forms the foundation of everything networking.

The user/controllers normally create services which are just abstractions. The controller or user creates endpoints in etcd based on the service selector. The controller matches pods based on the service selector and creates and endpoints, and populates it using the IP’s found in the Pod’s in etcd.

In order to ensure that a Service does not expose Pods that are not ready it will wait till the Readiness and Liveness probe for the Pod to report that the Pod is ready.

Now that we know how endpoints are populated its time to see how traffic is sent to the pod, without Kubernetes really handling networking.

Coredns and Services and Endpoints

Kube-proxy and endpoints

Kube-proxy watches etcd to look for endpoints and services. Whenever a service or endpoint is created or modified it updates the iptables/ipvs rules or any other method such that any traffic send to the Service IP is sent in a specified load balanced manner to the set of Pod IPs in the endpoint.

This also shows how Kube-proxy can be decoupled from CNI. In some cases for highly optimized traffic flows the two can be tightly coupled and share state.

Loadbalancers/NodePort

By default a service is created as Type ClusterIP. The other types allows routing of traffic to Pod using Loadbalancers or NodePorts.

This shows how just Nodes, Pods, Services and Endpoints are used to completely achieve everything every higher level concept in Kubernetes. We also saw how the simple Pod probes are used to actively manage traffic.

Because of how loosely coupled networking is in Kubernetes it is possible to create a Pod with a CNI or CNI plugin chain to create multiple interfaces with their own IP addresses even though the Pod spec only support a single IP address.

Only the Primary IP associated with eth0 is registered as the PodIP. The secondary IPs associated with the Pod can be stored in additional annotations in the Pod. These annotations can then be used by a plugin controller to create endpoints that map Service IP’s to these secondary IP addresses.

As we have seen that kube-proxy is decoupled completely from CNI and Pods. Kube-proxy will setup rules for secondary IP addresses just like it does for the Primary IP addresses. As long as the networking implementation has setup host routing such that the secondary IP’s are routable either at Pod level or network fabric level traffic to and from these secondary IP’s and interfaces will flow transparently across the cluster.

Related posts:

As he raised it to his lips, my eyes followed his hand. He was maybe ten years younger than me, maybe more. Confident in that understated way of being invisible until you made eye contact. I watched…

When your and your spouse’s assets are about to be divided up in a divorce, it is normal for you to wonder whether your spouse is hiding anything. Namely, if you live in an equitable distribution…

Can executive presence be improved through coaching? How? One can certainly improve one’s executive presence through executive coaching. To put it simply, our executive presence is what people are…