# cert-manager and trust-manager concept guide

cert-manager and trust-manager are open-source CNCF projects that provide for
automated management of X.509 certificates.

- [cert-manager](https://cert-manager.io/docs/) is a Kubernetes controller that
  automates creation and rotation of X.509 certificates.

- [trust-manager](https://cert-manager.io/docs/trust/trust-manager/) is a
  Kubernetes controller that helps to manage trust bundles on Kubernetes
  clusters.

Together, they're extremely useful tools for managing the trust anchor and
identity issuer for Linkerd (and they can also manage certificates for other
components of your system as well). There's a lot of documentation out there on
the specifics of using both of these projects, so we're going to focus on the
underlying concepts here.

> [!NOTE]
> If you're not already familiar with X.509 certificates, you might want to read
> our [(m)TLS concept guide](/buoyant-enterprise-linkerd/2.18/guides/tls-concepts/). Certificates are a critical part of
> the security infrastructure Linkerd, and understanding how they work is
> extremely helpful when operating Linkerd.

## cert-manager certificates and issuers

The point of cert-manager is _issuing certificates_ (remember, rotating a
certificate amounts to issuing a new certificate under the hood). Issuing
certificates is technically easy, but organizationally complex: every security
organization has its own rules and policies about what rules they want for
certificates.

- One organization, for example, might generate a root certificate by hand, then
  use it to issue certificates for all of its workloads.

- A different organization might choose to have all their certificates provided
  by Let's Encrypt (effectively using Let's Encrypt as their root certificate
  authority).

- A large enterprise might run an on-premise instance of a commercial certifying
  authority, using that to provide all the certificates used inside the
  enterprise.

All of these are valid choices, and cert-manager can support all of them. To
make this possible, cert-manager is configured in terms of _certificates_ and
_issuers_.

A cert-manager _certificate_ is just a description of an X.509 certificate that
you want cert-manager to manage for you. It's always represented by a
Certificate CRD.

A cert-manager _issuer_ is a description of a some certifying authority from
which cert-manager can request the certificates it manages. There are two
different CRDs for issuers: the Issuer resource and the ClusterIssuer resource.
These two resources are identical except for the namespace(s) from which they
can be accessed, as we'll discuss
[below](/buoyant-enterprise-linkerd/2.18/guides/cert-manager-concepts/#cert-manager-issuers-and-clusterissuers).

### cert-manager Certificates

A cert-manager Certificate resource describes an X.509 certificate. These really
are almost as straightforward as they sound: they have fields for the
certificate's Subject, for various metadata like the validity period, and -
importantly - for the issuer that should be used to issue the certificate. The
only complex bit about a Certificate is that cert-manager doesn't always use the
same terminology as X.509 itself to refer to the various bits of metadata to be
placed in the certificate.

Internally, cert-manager creates a unique private key for every certificate it
generates. It uses this private key to generate a [Certificate Signing Request
(CSR)][csr-wikipedia] which it sends to the Certificate's issuer; the issuer
will then respond with a signed certificate. cert-manager will then store the
private key and the signed certificate in a Kubernetes Secret.

In a perfect world, we would _not_ store the private key in the cluster, in
order to limit the ways in which the private key could be exposed to attackers.
In the real world, though, this simply isn't always possible: for example,
Linkerd needs access to the private key of its identity issuer certificate in
order to issue workload certificates.

Instead, the recommended best practice here is to be careful about namespaces.
When cert-manager creates a certificate using a Certificate resource, the
certificate will be written into a Secret in the same namespace as the
Certificate resource. Therefore, you should be careful to put the Certificate
resource in a namespace that makes sense:

- The Certificate for the Linkerd identity issuer should go in the `linkerd`
  namespace, so that the control plane has access to the resulting Secret.

- The Certificate for the Linkerd trust anchor should _not_ go in the `linkerd`
  namespace, since Linkerd only needs access to its public key. Instead, put the
  Certificate in another namespace (often the `cert-manager` namespace) and use
  trust-manager to copy its public key into the `linkerd` namespace, as
  described below.

[csr-wikipedia]: https://en.wikipedia.org/wiki/Certificate_signing_request

#### Certificate rotation

Once cert-manager generates a certificate, it will take responsibility for
_rotating_ it: periodically updating the certificate so that it never goes
outside its validity period. This is a critical part of security when using
X.509 certificates, since private keys become more valuable to attackers the
longer they're used.

However, by default, cert-manager will _not_ generate a new private key when it
rotates a certificate! You can force private key rotation by setting
`rotationPolicy: Always` in the `privateKey` stanza of a Certificate resource,
and we recommend that you always do so.

> [!WARNING]
> If you do not set `rotationPolicy: Always` in the Certificate's `privateKey`
> section, cert-manager **will not** actually rotate the trust anchor: instead, it
> will update the validity timestamps but **not** generate a new private key.
> **This is definitely not as secure as rotating the private key**.

### cert-manager Issuers and ClusterIssuers

cert-manager Issuers and ClusterIssuers, as mentioned above, describe a
certifying authority that cert-manager can use to issue certificates. They are
identical except for one critical difference:

- The Issuer resource can only be used to issue certificates in a _single_
  namespace.

- The ClusterIssuer resource can be used to issue certificates in _any_
  namespace.

**Both the Issuer and ClusterIssuer are namespaced resources.** The difference
is whether or not they can be accessed from any namespace other than their own
namespace.

There are _many_ different
[kinds of issuers](https://cert-manager.io/docs/configuration/issuers/). Some of
the common ones:

- `SelfSigned` issuers use a self-signed root certificate generated by
  cert-manager to issue certificates.

- `Vault` issuers use a HashiCorp Vault instance to issue certificates.

- `ACME` issuers use the ACME protocol to issue certificates from a public
  certifying authority like Let's Encrypt.

- `CA` issuers use a certificate stored in a Kubernetes Secret to issue
  certificates. They're useful for chaining issuers together, as described
  below.

The specifics of how to configure an issuer, and of whether you should use an
Issuer or a ClusterIssuer resource, depend on the issuer you're using and your
organizational policies. (The cert-manager documentation on
[Issuer Configuration](https://cert-manager.io/docs/configuration/) is a good
place to start.) However, all issuers end up being used in the same way: you
reference the issuer from a Certificate to tell cert-manager to generate a
certificate using that issuer.

### Chaining issuers

The `CA` issuer is worth a little bit of extra discussion, since it unlocks a
particularly powerful capability within cert-manager: the ability to chain
issuers together.

When using cert-manager to automate certificate management for Linkerd, we want
cert-manager to first some issuer to obtain Linkerd's trust anchor, then use the
trust anchor to generate an identity issuer. The way we recommend doing this is
covered in detail in the [Linkerd documentation about automating certificate
rotation][linkerd-rotation-docs], but the overall pattern is:

1. Create an Issuer in the `cert-manager` namespace telling cert-manager where
   it should get the trust anchor.

2. Create a Certificate in the `cert-manager` namespace that uses the trust
   anchor issuer from step 1 to generate the trust anchor certificate and store
   it into a Secret in the `cert-manager` namespace.

3. Create a `CA` ClusterIssuer in the `cert-manager` namespace referencing the
   Secret from step 2.

4. Create a Certificate _in the `linkerd` namespace_ that uses the ClusterIssuer
   from step 3 to generate the identity issuer certificate and store it into a
   Secret in the `linkerd` namespace.

5. Finally, configure trust-manager to copy only the public part of the trust
   anchor certificate into a ConfigMap in the `linkerd` namespace.

You can see this pattern in the Linkerd documentation about automating
certificate rotation.

## trust-manager

As noted in the discussion about Certificates, when cert-manager generates a
certificate, it will write it into a Secret in the same namespace as the
Certificate resource. In many cases, though, the certificate also needs to be
added to a trust bundle to be useful: this is trust-manager's job.

Specifically, trust-manager knows how to take a certificate's public key from a
Secret and add it to a trust bundle contained in a Kubernetes ConfigMap resource
(since trust bundles only contain public keys, there's no need for them to be
stored in Secrets). In the [Linkerd documentation about automating certificate
rotation][linkerd-rotation-docs], you can see trust-manager used to copy the
public key of the trust anchor certificate into the Linkerd trust bundle.

The trust-manager configuration documentation is fairly straightforward, but
there are a few critical things to remember:

1. trust-manager uses the Bundle resource to control how the trust bundle is
   created. The Bundle resource must have the same name and be in the same
   namespace as the ConfigMap to be created. For Linkerd, this implies that the
   Bundle resource must be in the `linkerd` namespace.

2. trust-manager can only read Secrets in its "trust namespace", which you
   configure when installing trust-manager. A reasonable practice when using
   trust-manager with Linkerd is to install trust-manager in the `cert-manager`
   namespace and also configure it to use the `cert-manager` namespace as the
   trust namespace: that makes it relatively easy to restrict access to Secrets
   in the `cert-manager` namespace while still allowing trust-manager to do its
   job. access by namespace,

3. The `sources` stanza of the Bundle resource defines where trust-manager will
   look for certificates to copy into the bundle -- and if you want
   trust-manager to include multiple certificates in the bundle, you _must_ use
   multiple `sources` and manage them carefully. You can see more about this in
   the [Linkerd documentation about automating certificate
   rotation][linkerd-rotation-docs].

[linkerd-rotation-docs]: https://linkerd.io/2.18/tasks/automatically-rotating-control-plane-tls-credentials/
