cert-manager and trust-manager concept guide

cert-manager and trust-manager are open-source CNCF projects that provide for automated management of X.509 certificates.

• cert-manager is a Kubernetes controller that automates creation and rotation of X.509 certificates.

• trust-manager is a Kubernetes controller that helps to manage trust bundles on Kubernetes clusters.

Together, they’re extremely useful tools for managing the trust anchor and identity issuer for Linkerd (and they can also manage certificates for other components of your system as well). There’s a lot of documentation out there on the specifics of using both of these projects, so we’re going to focus on the underlying concepts here.

cert-manager certificates and issuers

The point of cert-manager is issuing certificates (remember, rotating a certificate amounts to issuing a new certificate under the hood). Issuing certificates is technically easy, but organizationally complex: every security organization has its own rules and policies about what rules they want for certificates.

• One organization, for example, might generate a root certificate by hand, then use it to issue certificates for all of its workloads.

• A different organization might choose to have all their certificates provided by Let’s Encrypt (effectively using Let’s Encrypt as their root certificate authority).

• A large enterprise might run an on-premise instance of a commercial certifying authority, using that to provide all the certificates used inside the enterprise.

All of these are valid choices, and cert-manager can support all of them. To make this possible, cert-manager is configured in terms of certificates and issuers.

A cert-manager certificate is just a description of an X.509 certificate that you want cert-manager to manage for you. It’s always represented by a Certificate CRD.

A cert-manager issuer is a description of a some certifying authority from which cert-manager can request the certificates it manages. There are two different CRDs for issuers: the Issuer resource and the ClusterIssuer resource. These two resources are identical except for the namespace(s) from which they can be accessed, as we’ll discuss below.

cert-manager Certificates

A cert-manager Certificate resource describes an X.509 certificate. These really are almost as straightforward as they sound: they have fields for the certificate’s Subject, for various metadata like the validity period, and - importantly - for the issuer that should be used to issue the certificate. The only complex bit about a Certificate is that cert-manager doesn’t always use the same terminology as X.509 itself to refer to the various bits of metadata to be placed in the certificate.

Internally, cert-manager creates a unique private key for every certificate it generates. It uses this private key to generate a Certificate Signing Request (CSR) which it sends to the Certificate’s issuer; the issuer will then respond with a signed certificate. cert-manager will then store the private key and the signed certificate in a Kubernetes Secret.

In a perfect world, we would not store the private key in the cluster, in order to limit the ways in which the private key could be exposed to attackers. In the real world, though, this simply isn’t always possible: for example, Linkerd needs access to the private key of its identity issuer certificate in order to issue workload certificates.

Instead, the recommended best practice here is to be careful about namespaces. When cert-manager creates a certificate using a Certificate resource, the certificate will be written into a Secret in the same namespace as the Certificate resource. Therefore, you should be careful to put the Certificate resource in a namespace that makes sense:

• The Certificate for the Linkerd identity issuer should go in the linkerd namespace, so that the control plane has access to the resulting Secret.

• The Certificate for the Linkerd trust anchor should not go in the linkerd namespace, since Linkerd only needs access to its public key. Instead, put the Certificate in another namespace (often the cert-manager namespace) and use trust-manager to copy its public key into the linkerd namespace, as described below.

Certificate rotation

Once cert-manager generates a certificate, it will take responsibility for rotating it: periodically updating the certificate so that it never goes outside its validity period. This is a critical part of security when using X.509 certificates, since private keys become more valuable to attackers the longer they’re used.

However, by default, cert-manager will not generate a new private key when it rotates a certificate! You can force private key rotation by setting rotationPolicy: Always in the privateKey stanza of a Certificate resource, and we recommend that you always do so.

cert-manager Issuers and ClusterIssuers

cert-manager Issuers and ClusterIssuers, as mentioned above, describe a certifying authority that cert-manager can use to issue certificates. They are identical except for one critical difference:

• The Issuer resource can only be used to issue certificates in a single namespace.

• The ClusterIssuer resource can be used to issue certificates in any namespace.

Both the Issuer and ClusterIssuer are namespaced resources. The difference is whether or not they can be accessed from any namespace other than their own namespace.

There are many different kinds of issuers. Some of the common ones:

• SelfSigned issuers use a self-signed root certificate generated by cert-manager to issue certificates.

• Vault issuers use a HashiCorp Vault instance to issue certificates.

• ACME issuers use the ACME protocol to issue certificates from a public certifying authority like Let’s Encrypt.

• CA issuers use a certificate stored in a Kubernetes Secret to issue certificates. They’re useful for chaining issuers together, as described below.

The specifics of how to configure an issuer, and of whether you should use an Issuer or a ClusterIssuer resource, depend on the issuer you’re using and your organizational policies. (The cert-manager documentation on Issuer Configuration is a good place to start.) However, all issuers end up being used in the same way: you reference the issuer from a Certificate to tell cert-manager to generate a certificate using that issuer.

Chaining issuers

The CA issuer is worth a little bit of extra discussion, since it unlocks a particularly powerful capability within cert-manager: the ability to chain issuers together.

When using cert-manager to automate certificate management for Linkerd, we want cert-manager to first some issuer to obtain Linkerd’s trust anchor, then use the trust anchor to generate an identity issuer. The way we recommend doing this is covered in detail in the Linkerd documentation about automating certificate rotation, but the overall pattern is:

1. Create an Issuer in the cert-manager namespace telling cert-manager where it should get the trust anchor.

2. Create a Certificate in the cert-manager namespace that uses the trust anchor issuer from step 1 to generate the trust anchor certificate and store it into a Secret in the cert-manager namespace.

3. Create a CA ClusterIssuer in the cert-manager namespace referencing the Secret from step 2.

4. Create a Certificate in the linkerd namespace that uses the ClusterIssuer from step 3 to generate the identity issuer certificate and store it into a Secret in the linkerd namespace.

5. Finally, configure trust-manager to copy only the public part of the trust anchor certificate into a ConfigMap in the linkerd namespace.

You can see this pattern in the Linkerd documentation about automating certificate rotation.

trust-manager

As noted in the discussion about Certificates, when cert-manager generates a certificate, it will write it into a Secret in the same namespace as the Certificate resource. In many cases, though, the certificate also needs to be added to a trust bundle to be useful: this is trust-manager’s job.

Specifically, trust-manager knows how to take a certificate’s public key from a Secret and add it to a trust bundle contained in a Kubernetes ConfigMap resource (since trust bundles only contain public keys, there’s no need for them to be stored in Secrets). In the Linkerd documentation about automating certificate rotation, you can see trust-manager used to copy the public key of the trust anchor certificate into the Linkerd trust bundle.

The trust-manager configuration documentation is fairly straightforward, but there are a few critical things to remember:

1. trust-manager uses the Bundle resource to control how the trust bundle is created. The Bundle resource must have the same name and be in the same namespace as the ConfigMap to be created. For Linkerd, this implies that the Bundle resource must be in the linkerd namespace.

2. trust-manager can only read Secrets in its “trust namespace”, which you configure when installing trust-manager. A reasonable practice when using trust-manager with Linkerd is to install trust-manager in the cert-manager namespace and also configure it to use the cert-manager namespace as the trust namespace: that makes it relatively easy to restrict access to Secrets in the cert-manager namespace while still allowing trust-manager to do its job. access by namespace,

3. The sources stanza of the Bundle resource defines where trust-manager will look for certificates to copy into the bundle – and if you want trust-manager to include multiple certificates in the bundle, you must use multiple sources and manage them carefully. You can see more about this in the Linkerd documentation about automating certificate rotation.