Understanding the Role of Cert-manager in Cluster API

The Cluster API team has started cutting release candidates of the latest version of the project: Cluster API v0.3. I have been looking forward to this new version, as it adds a set of powerful capabilities, such as declarative control plane management and cluster upgrades. I plan to explore those features in future posts.

As I started testing the latest release candidate, I noticed that I had to install cert-manager into my management cluster. I wasn't quite sure why this was the case, so I decided to learn more about how Cluster API leverages cert-manager.

What is cert-manager?

Cert-manager is a controller and set of Custom Resource Definitions (CRDs) that enable automated certificate management on Kubernetes. Some of the CRDs that cert-manager installs are Issuers, CertificateRequests, and Certificates. Using these CRDs, you can request a certificate via the Kubernetes API. Once you create the request, the cert-manager controller creates and manages the certificate for you.

The most common scenario I have run into with cert-manager is issuing certificates for Ingress resources. Cluster API, however, does not use cert-manager for this.

Cert-manager and Cluster API

After installing the latest release candidate of cluster API in a fresh management cluster, I looked for certificate requests:

$ kubectl get certificaterequests --all-namespaces
NAMESPACE             NAME                                                 READY   AGE
capi-webhook-system   capi-kubeadm-bootstrap-serving-cert-1303918272       True    109m
capi-webhook-system   capi-kubeadm-control-plane-serving-cert-1399157480   True    109m
capi-webhook-system   capi-serving-cert-3321106290                         True    109m

I noticed that all certificate requests live in the capi-webhook-system namespace. The next thing I wanted to know was the issuer used to create these certificates:

$ kubectl get issuer -n capi-webhook-system
NAME                                           AGE
capi-kubeadm-bootstrap-selfsigned-issuer       109m
capi-kubeadm-control-plane-selfsigned-issuer   109m
capi-selfsigned-issuer                         109m

There are three issuers, one for each of the certificates that Cluster API requests during installation. Judging by the name, the issuers create self-signed certificates.

Finally, I listed the Certificates in the capi-webhook-system to verify that cert-manager had fulfilled the Certificate Requests:

$ kubectl get certificates -n capi-webhook-system
NAME                                      READY   SECRET                                            AGE
capi-kubeadm-bootstrap-serving-cert       True    capi-kubeadm-bootstrap-webhook-service-cert       108m
capi-kubeadm-control-plane-serving-cert   True    capi-kubeadm-control-plane-webhook-service-cert   108m
capi-serving-cert                         True    capi-webhook-service-cert                         108m

After exploring the certificaterequests, issuers and certificates created during the Cluster API installation, it is evident that Cluster API leverages cert-manager to create self-signed certificates. But how are these certificates used?

Cluster API Webhooks

Starting with version 0.3, Cluster API configures validating and mutating webhooks in the management cluster. It also uses webhooks for converting custom resources between API versions. These webhooks are admission controllers or plugins that the API server calls when Cluster API resources are created or modified.

The validating webhooks validate v1alpha3 Cluster API resources. For example, if I try to create a v1alpha3 Cluster that has an InfrastructureRef in another namespace, I get a validation error. This validation is implemented in the validating webhook.

$ kubectl apply -f invalid-cluster.yaml
Error from server (Cluster.cluster.x-k8s.io "capi-quickstart" is invalid:
spec.infrastructureRef.namespace: Invalid value: "foo": must match metadata.namespace):
error when creating "sample.yaml": admission webhook "validation.cluster.cluster.x-k8s.io" denied the request:
Cluster.cluster.x-k8s.io "capi-quickstart" is invalid: spec.infrastructureRef.namespace:
Invalid value: "foo": must match metadata.namespace

Cluster API uses the mutating webhooks to set default values in v1alpha3 custom resources. For example, the Cluster resource defaults the namespace of the InfrastructureRef and ControlPlaneRef to the same namespace as the Cluster. The defaulting of these values improve the UX of these custom resources.

Finally, Cluster API specifies conversion webhooks in the CRD specifications. The conversion webhooks implement the conversion of custom resources from v1alpha2 to v1alpha3 and vice-versa. The conversion allows Cluster API users to upgrade their existing management clusters from Cluster API 0.2 to 0.3.

This is great, but what does it have to do with certificates?

When configuring admission webhooks, Kubernetes requires the webhooks to expose an HTTPS service. And this is precisely where the certificates come into play—three certificates for three webhooks. The webhook services use the certificates to expose the webhooks over HTTPS.

# Three certificates
$ kubectl get certificate
NAME                                      READY   SECRET                                            AGE
capi-kubeadm-bootstrap-serving-cert       True    capi-kubeadm-bootstrap-webhook-service-cert       47h
capi-kubeadm-control-plane-serving-cert   True    capi-kubeadm-control-plane-webhook-service-cert   47h
capi-serving-cert                         True    capi-webhook-service-cert                         47h

# Three webhook services
$ kubectl get service
NAME                                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
capi-kubeadm-bootstrap-webhook-service       ClusterIP   <none>        443/TCP   47h
capi-kubeadm-control-plane-webhook-service   ClusterIP   <none>        443/TCP   47h
capi-webhook-service                         ClusterIP    <none>        443/TCP   47h


I hope this post sheds light on why Cluster API v0.3 needs cert-manager in the management cluster.

In brief, Cluster API v0.3 installs admission webhooks in the management cluster to validate, mutate, and convert Cluster API resources. Kubernetes requires these webhooks to be HTTPS services, thus requiring serving certificates. Instead of creating the certificates in a one-shot manner, Cluster API leverages the cert-manager project to create and manage the webhook certificates throughout the lifetime of the management cluster.

If you are looking to test the latest Cluster API release candidates, check out the master version of the Cluster API Quick Start (docs are still under development, so you might run into sharp edges).

Additional reading:

Did you find this post useful? Did I get something wrong? I would love to hear from you! Please reach out via @alexbrand.