August 06, 2019

Testing Kubernetes Network Policy Enforcement with Sonobuoy

The NetworkPolicy resource in Kubernetes allows you to define ingress and egress rules on pods. With these rules, you can control how pods communicate with each other and other services on the network.

It is important to understand that Kubernetes does not enforce the NetworkPolicy rules you define. Instead, the enforcement of the policy is the responsibility of the Container Network Interface (CNI) plugin running in your cluster.

If you are deploying clusters with a CNI plugin that enforces NetworkPolicy (you should!), you will want to verify that the plugin is applying the policy correctly. Luckily, the community maintains a set of end-to-end tests that we can leverage to do this.

Using Sonobuoy, it is straightforward to run the NetworkPolicy tests:

sonobuoy run --e2e-focus "NetworkPolicy" --e2e-skip ""

These are the results obtained from a cluster running Flannel as the CNI plugin. As you can see, a number of the tests fail because Flannel does not support NetworkPolicy.

(kubernetes-admin@flannel) $ sonobuoy retrieve
201908061342_sonobuoy_aacee2a3-29be-409d-ac67-fb38a2ea9775.tar.gz
(kubernetes-admin@flannel) $ sonobuoy e2e 201908061342_sonobuoy_aacee2a3-29be-409d-ac67-fb38a2ea9775.tar.gz
failed tests: 10
[sig-network] NetworkPolicy NetworkPolicy between server and client should allow egress access on one named port [Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should allow ingress access on one named port [Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on NamespaceSelector [Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on NamespaceSelector with MatchExpressions[Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on PodSelector [Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on PodSelector and NamespaceSelector [Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on PodSelector or NamespaceSelector [Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on PodSelector with MatchExpressions[Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on Ports [Feature:NetworkPolicy]
[sig-network] NetworkPolicy NetworkPolicy between server and client should support a 'default-deny' policy [Feature:NetworkPolicy]

Contrast the above with the results from a cluster running Calico as the CNI plugin:

(kubernetes-admin@calico) $ sonobuoy retrieve
201908061350_sonobuoy_4d65b9e4-a7ac-48a1-9ac6-8bfcd3b287e2.tar.gz
(kubernetes-admin@calico) $ sonobuoy e2e 201908061350_sonobuoy_4d65b9e4-a7ac-48a1-9ac6-8bfcd3b287e2.tar.gz
failed tests: 0

As you can see, there were zero test failures because Calico does support Kubernetes NetworkPolicy.

When deploying a CNI plugin that enforces NetworkPolicy, it is a good idea to run these tests after installing or upgrading the plugin. This will ensure that the plugin is configured correctly and that the NetworkPolicy enforcement bits are behaving as expected.

If you want to learn more about Kubernetes NetworkPolicy, check out the following resources:

Did you find this post useful? Did I get something wrong? I would love to hear from you! Please reach out via @alexbrand.