First look at Antrea: a CNI plugin based on Open vSwitch
VMware recently announced Antrea, an open-source Container Network Interface (CNI) plugin based on Open vSwitch (OVS). The project lives under VMware’s Tanzu GitHub organization. You can find the repo here.
This post will go over Antrea’s installation process, its components and how it works.
Installing Antrea
Similar to other CNI plugins, Antrea has a straightforward installation process that involves applying a set of Kubernetes resources onto your cluster. The main prerequisite is that the node must have the OVS kernel module installed (it seems like they are available out-of-the-box in most recent Linux distributions).
The following command installs Antrea v0.1.1:
kubectl apply -f https://github.com/vmware-tanzu/antrea/releases/download/v0.1.1/antrea.yml
Once installed, the Antrea controller and node agent should be running:
$ kubectl get pods -n kube-system | grep antrea
antrea-agent-ff96v 2/2 Running 0 2m
antrea-agent-kx9s7 2/2 Running 0 2m
antrea-agent-zz2sk 2/2 Running 0 2m
antrea-controller-75c858b85c-lg2wm 1/1 Running 0 2m
Antrea Components
As you can probably tell from the output above, Antrea has two main components: a controller and an agent.
The controller is managed by a Deployment on the cluster. At the moment, only a single replica of the controller is supported. The controller is responsible for computing network policies and exposing them to the agents via an API. As the project evolves, the controller might gain additional responsibilities.
The agent is installed as a DaemonSet, which creates an agent pod on each node of the cluster. The node agent pod contains two containers and an init container. The init container installs the CNI binary and loads the Open vSwitch kernel module. Once the init container is done, the Antrea agent container and the OVS daemon container start up. These containers work together to provide networking connectivity to the pods running on the node.
If you are curious, the architecture document is an excellent resource to learn about the Antrea components in depth.
How Does It Work?
I don’t have much experience with OVS, so I wanted to learn more about how Antrea works. I installed Antrea on a cluster and explored its functionality.
Node Bridge
Antrea installs an OVS bridge called br-int on each node. Initially, the
bridge has a tunnel port and a gateway port. The tunnel port routes packets to
other nodes in the cluster using encapsulation, while the gateway port routes
traffic that is destined to external IPs.
The following command shows the Open vSwitch configuration on a node that has no pods running:
$ kubectl exec -it antrea-agent-ff96v -c antrea-agent ovs-vsctl show
4cabe643-5179-483f-87cb-83cdd4db186c
Bridge br-int
Port "tun0"
Interface "tun0"
type: vxlan
options: {key=flow, remote_ip=flow}
Port "gw0"
Interface "gw0"
type: internal
ovs_version: "2.11.1"
You can see the br-int bridge, which has the tunnel (tun0) and gateway
(gw0) ports.
Pod Network
When a pod is scheduled on a node, the Kubelet executes the Antrea CNI plugin,
which in turn reaches out to the Antrea agent to set up the pod network. At this
point, the agent creates a veth pair for the pod and connects it to the br-int
bridge.
For example, after deploying a sample nginx pod, the OVS configuration of
br-int shows a new port and interface called nginx6db-fd32cf:
$ kubectl exec -it antrea-agent-ff96v -c antrea-agent ovs-vsctl show
4cabe643-5179-483f-87cb-83cdd4db186c
Bridge br-int
Port "nginx6db-fd32cf"
Interface "nginx6db-fd32cf"
Port "tun0"
Interface "tun0"
type: vxlan
options: {key=flow, remote_ip=flow}
Port "gw0"
Interface "gw0"
type: internal
ovs_version: "2.11.1"
Because all pods on a given node are connected to the same bridge, they are in the same Layer-2 broadcast domain. To see this in action, I created three replicas of the netshoot container in my cluster. Notice how two of them are on the same node.
$ kubectl get pods -o custom-columns='name:.metadata.name,pod ip:.status.podIPs[0].ip,node:.spec.nodeName' --sort-by='.spec.nodeName'
name pod ip node
netshoot-86669b4fd6-4m26w 192.168.1.43 ip-10-0-0-105.us-east-2.compute.internal
netshoot-86669b4fd6-vbds2 192.168.1.42 ip-10-0-0-105.us-east-2.compute.internal
netshoot-86669b4fd6-jmvfc 192.168.2.23 ip-10-0-0-247.us-east-2.compute.internal
Using arping, I can ping - using ARP packets - pods running on the same node:
$ kubectl exec -it netshoot-86669b4fd6-4m26w -- arping -c 4 192.168.1.42
ARPING 192.168.1.42 from 192.168.1.43 eth0
Unicast reply from 192.168.1.42 [06:3E:37:BC:79:79] 0.841ms
Unicast reply from 192.168.1.42 [06:3E:37:BC:79:79] 0.709ms
Unicast reply from 192.168.1.42 [06:3E:37:BC:79:79] 0.545ms
Unicast reply from 192.168.1.42 [06:3E:37:BC:79:79] 0.545ms
Sent 4 probes (1 broadcast(s))
Received 4 response(s)
On the other hand, I am unable to arping a pod running on another node:
$ kubectl exec -it netshoot-86669b4fd6-4m26w -- arping -c 4 192.168.2.23
ARPING 192.168.2.23 from 192.168.1.43 eth0
Sent 4 probes (4 broadcast(s))
Received 0 response(s)
command terminated with exit code 1
This is because they lack Layer-2 connectivity. Pod traffic that needs to go from one node to another must be routed through a tunnel to the destination.
The Antrea documentation has a great diagram that describes the different packet walks. Check it out for more details on the data path.
Network Policy
Antrea implements NetworkPolicy using OVS Flows. Flows are organized in tables, and they are applied on each node by the Antrea agent. After going through the source code, I found that the ingress rule table is flow table 90.
Before applying a network policy, flow table 90 on a given node is almost empty:
$ kubectl -n kube-system exec -it antrea-agent-kx9s7 ovs-ofctl dump-flows br-int | grep table=90
cookie=0x0, duration=261738.430s, table=90, n_packets=36327088, n_bytes=597418196451, priority=210,ct_state=-new+est,ip actions=resubmit(,110)
cookie=0x0, duration=261738.793s, table=90, n_packets=89, n_bytes=7527, priority=80,ip actions=resubmit(,100)
After applying a network policy with an ingress rule, we can see new entries in the flow table. Because these are ingress rules, Antrea installs the flows on the nodes that are hosting the pods selected by the network policy.
$ kubectl -n kube-system exec -it antrea-agent-kx9s7 ovs-ofctl dump-flows br-int | grep table=90
cookie=0x0, duration=262007.710s, table=90, n_packets=36327088, n_bytes=597418196451, priority=210,ct_state=-new+est,ip actions=resubmit(,110)
cookie=0x0, duration=162.476s, table=90, n_packets=0, n_bytes=0, priority=200,ip,nw_src=192.168.2.1 actions=conjunction(2,1/2)
cookie=0x0, duration=0.277s, table=90, n_packets=0, n_bytes=0, priority=200,ip,nw_src=192.168.2.31 actions=conjunction(2,1/2)
cookie=0x0, duration=162.432s, table=90, n_packets=0, n_bytes=0, priority=200,ip,reg1=0x1e actions=conjunction(2,2/2)
cookie=0x0, duration=162.507s, table=90, n_packets=0, n_bytes=0, priority=190,conj_id=2,ip actions=resubmit(,110)
cookie=0x0, duration=262008.073s, table=90, n_packets=89, n_bytes=7527, priority=80,ip actions=resubmit(,100)
Antrea installs flows that have conjunction actions. In the output above, the
third line is an action that says “Allow Ingress when the source IP address is
192.168.2.31”. This is the first half of the conjunction, as indicated by
conjunction(2,1/2). The second half of the conjunction is on the fourth line,
which matches the flow with the targeted pod’s MAC address using registers
(reg1=0x1e). This is all fairly new to me, so it is pretty hand-wavy at the
moment. I plan to dig into this more in a follow-up post.
Wrap Up
In this post, we explored Antrea’s high-level architecture and how it works. Antrea is a relatively new project, and it has an exciting roadmap ahead.
If you want to learn more about Antrea or contribute to the project, check out the following resources:
- Announcing Project Antrea - Open Source Kubernetes Networking
- Antrea GitHub repository
- Antrea Architecture
- Antrea Roadmap
Did you find this post useful? Did I get something wrong? I would love to hear from you! Please reach out via @alexbrand.