# Full BEL installation guide

This guide walks you through installing Buoyant Enterprise for Linkerd (BEL) on
a Kubernetes cluster.

## Before you begin

### Choose an install method

| Method                                                              | Recommended for              | Notes                                                                                                        |
| ------------------------------------------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [Lifecycle operator](#option-a-install-with-the-lifecycle-operator) | Production                   | Upgrades are declarative and work well with GitOps workflows.                                                |
| [Helm](#option-b-install-with-helm)                                 | Production                   | You manage BEL's Helm releases directly.                                                                     |
| [CLI](#option-c-install-with-the-cli)                               | Demos and non-production use | Quickest to set up. TLS certificates expire after one year, and the trust-rotation operator isn't supported. |

### Choose your options

| Option                             | Description                                                                           | Methods        |
| ---------------------------------- | ------------------------------------------------------------------------------------- | -------------- |
| **High availability (HA)**         | Runs the control plane in HA mode. Recommended for production.                        | All            |
| **CNI plugin**                     | Uses the Linkerd CNI plugin instead of an init container to configure pod networking. | All            |
| **Trust-rotation operator** (beta) | Automates trust anchor rotation using cert-manager.                                   | Operator, Helm |
| **FIPS modules**                   | Installs FIPS-validated builds. Requires a plan that includes FIPS.                   | All            |
| **Buoyant Cloud**                  | Connects your cluster to Buoyant Cloud. Requires a plan that includes Buoyant Cloud.  | All            |

## Common setup

Complete these steps whichever install method you chose.

### Install the Gateway API

Linkerd uses [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/)
resources to configure certain features, so the Gateway API has to be installed
first. To install the latest supported version, run:

```bash
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml
```

If you already have the Gateway API CRDs on your cluster, you can likely keep
using them as they are. For details, see
[Managing Gateway API resources](../../guides/managing-gateway-api/).

### Get your license key

BEL requires a valid license key, which you get from the
[Buoyant portal](https://enterprise.buoyant.io/). Follow the instructions there
and you'll end up with an environment variable like this:

```bash
export BUOYANT_LICENSE=[LICENSE]
```

**Option: Buoyant Cloud**

The portal also gives you API credentials. Export them as well:

```bash
export API_CLIENT_ID=[ID]
export API_CLIENT_SECRET=[SECRET]
export BUOYANT_LICENSE=[LICENSE]
```

**End of option: Buoyant Cloud**

### Install the BEL CLI

Download and install the BEL CLI:

```bash
curl --proto '=https' --tlsv1.2 -sSfL https://enterprise.buoyant.io/install | sh
```

**Option: FIPS modules**

Install the FIPS build of the CLI instead:

```bash
curl --proto '=https' --tlsv1.2 -sSfL https://enterprise.buoyant.io/install | FIPS=1 sh
```

**End of option: FIPS modules**

Follow the instructions to add the `linkerd` CLI to your system path, then
confirm that it's running the expected version:

```bash
linkerd version --client
```

You should see:

```bash
Client version: enterprise-2.20.6
```

**Option: FIPS modules**

The version ends with `-fips` (`enterprise-2.20.6-fips`).

**End of option: FIPS modules**

### Validate your cluster

Check that your cluster is ready for installation:

```bash
linkerd check --pre
```

Next, follow the section for the install method you chose:
[lifecycle operator](#option-a-install-with-the-lifecycle-operator),
[Helm](#option-b-install-with-helm), or [CLI](#option-c-install-with-the-cli).

## Option A: Install with the lifecycle operator

### A1. Install the Linkerd CNI plugin

**Option: CNI plugin**

The `linkerd-cni` DaemonSet must be running on your cluster before you install
the BEL control plane:

```bash
helm repo add linkerd-edge https://helm.linkerd.io/edge
helm repo update
helm search repo linkerd2-cni
helm install linkerd-cni -n linkerd-cni --create-namespace linkerd-edge/linkerd2-cni
# ensure the plugin is installed and ready
linkerd check --pre --linkerd-cni-enabled
```

**End of option: CNI plugin**

### A2. Install the lifecycle operator

Add or update the `linkerd-buoyant` Helm repo:

```bash
helm repo add linkerd-buoyant https://helm.buoyant.cloud
helm repo update
```

Install the BEL lifecycle automation operator:

```bash
helm install linkerd-buoyant \
  --create-namespace \
  --namespace linkerd-buoyant \
  --set buoyantCloudEnabled=false \
  --set license=$BUOYANT_LICENSE \
  linkerd-buoyant/linkerd-buoyant
```

**Option: Buoyant Cloud**

Use this command instead. Replace `[YOUR_CLUSTER_NAME]` with the name that will
identify this cluster in Buoyant Cloud.

```bash
helm install linkerd-buoyant \
  --create-namespace \
  --namespace linkerd-buoyant \
  --set metadata.agentName=[YOUR_CLUSTER_NAME] \
  --set api.clientID=$API_CLIENT_ID \
  --set api.clientSecret=$API_CLIENT_SECRET \
  linkerd-buoyant/linkerd-buoyant
```

**End of option: Buoyant Cloud**

### A3. Create the TLS infrastructure

Choose one approach:

- **Generate certificates manually.** Follow A3a below.
- **Trust-rotation operator.** Follow A3b below. The trust-rotation operator
  requires cert-manager.

#### A3a. Generate certificates manually

> [!NOTE]
> TLS key management is a complex topic. In this guide we create the
> certificates by hand. [Automated rotation](https://linkerd.io/2.20/tasks/automatically-rotating-control-plane-tls-credentials/)
> of these credentials is common in production environments, but it's outside the
> scope of this guide.

You need to generate a trust anchor certificate and key pair, and an issuer
certificate and key pair. Pay attention to the lifetimes of these certificates,
especially the trust anchor, and keep the files somewhere safe.

Generate the trust anchor certificate and key with a tool like
[`step`](https://smallstep.com/docs/step-cli/installation):

```bash
step certificate create root.linkerd.cluster.local ca.crt ca.key \
--profile root-ca --no-password --insecure
```

This creates `ca.crt` and `ca.key`. The `--no-password --insecure` flags leave
the files unencrypted, with no passphrase. For a longer-lived trust anchor, add
`--not-after` with the lifetime you want (for example, `--not-after=87600h`).

Generate the issuer certificate and key, which sign the Linkerd proxies'
certificate signing requests:

```bash
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca.crt --ca-key ca.key
```

This creates `issuer.crt` and `issuer.key`. Store them in the cluster as the
identity issuer secret:

```bash
cat <<EOF > linkerd-identity-secret.yaml
apiVersion: v1
data:
  ca.crt: $(cat ca.crt | base64 | tr -d '\n')
  tls.crt: $(cat issuer.crt | base64 | tr -d '\n')
  tls.key: $(cat issuer.key | base64 | tr -d '\n')
kind: Secret
metadata:
  name: linkerd-identity-issuer
  namespace: linkerd
type: kubernetes.io/tls
EOF

kubectl apply -f linkerd-identity-secret.yaml
```

#### A3b. Use cert-manager (trust-rotation operator)

**Option: Trust-rotation operator**

> [!NOTE]
> BEL's [trust-rotation operator](../../features/trust-rotation/)
> automates trust anchor rotation. It relies on
> [cert-manager](https://cert-manager.io/) to issue Linkerd's trust anchor and
> identity issuer certificates. For details, see
> [Configuring BEL's trust-rotation operator](../../guides/configuring-trust-rotation/).

If cert-manager isn't already installed on your cluster, install it:

```bash
helm repo add jetstack https://charts.jetstack.io --force-update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true \
  --wait
```

Install the `linkerd-cert-manager` chart. It creates the cert-manager `Issuer`
and `Certificate` resources that produce Linkerd's trust anchor and identity
issuer secrets:

```bash
helm install linkerd-cert-manager \
  --namespace linkerd \
  --create-namespace \
  oci://ghcr.io/linkerd/charts/linkerd-cert-manager \
  --wait
```

Linkerd reads its trust roots from the `linkerd-identity-trust-roots` ConfigMap.
That ConfigMap has to be seeded from cert-manager's `linkerd-trust-anchor`
Secret before you install Linkerd. The `linkerd trust bundle` command renders it
for you:

```bash
linkerd trust bundle | kubectl apply -f -
```

**End of option: Trust-rotation operator**

### A4. Configure the lifecycle operator

Create a `ControlPlane` resource that tells the operator how to install and
manage the Linkerd control plane. The command below writes it to
`linkerd-control-plane-config.yaml`.

**If you generated certificates manually (A3a):** run this from the directory
that contains `ca.crt`.

```bash
cat <<EOF > linkerd-control-plane-config.yaml
apiVersion: linkerd.buoyant.io/v1alpha1
kind: ControlPlane
metadata:
  name: linkerd-control-plane
spec:
  components:
    linkerd:
      version: enterprise-2.20.6
      controlPlaneConfig:
        license: $BUOYANT_LICENSE
        identityTrustAnchorsPEM: |
$(cat ca.crt | sed 's/^/          /')
        identity:
          issuer:
            scheme: kubernetes.io/tls
EOF
```

**If you used cert-manager (A3b):** this version tells Linkerd to use the trust
bundle and identity issuer that cert-manager manages.

```bash
cat <<EOF > linkerd-control-plane-config.yaml
apiVersion: linkerd.buoyant.io/v1alpha1
kind: ControlPlane
metadata:
  name: linkerd-control-plane
spec:
  components:
    linkerd:
      version: enterprise-2.20.6
      controlPlaneConfig:
        license: $BUOYANT_LICENSE
        identity:
          externalCA: true
          issuer:
            scheme: kubernetes.io/tls
EOF
```

**Option: CNI plugin**

Before you run the command, add a `crdsConfig` block, and add `cniEnabled: true`
under `controlPlaneConfig`:

```yaml
linkerd:
  version: ...
  crdsConfig:
    cniEnabled: true
  controlPlaneConfig:
    cniEnabled: true
    license: $BUOYANT_LICENSE
```

**End of option: CNI plugin**

**Option: FIPS modules**

Before you run the command, change the version to
`enterprise-2.20.6-fips`.

**End of option: FIPS modules**

**Option: High availability (HA)**

After you create the file, append the recommended HA settings to it:

```bash
tmp=$(mktemp -d)
helm pull linkerd-buoyant/linkerd-enterprise-control-plane --untar --untardir "$tmp"
sed 's/^/        /' "$tmp/linkerd-enterprise-control-plane/values-ha.yaml" \
  >> linkerd-control-plane-config.yaml
```

**End of option: High availability (HA)**

**Option: GKE 1.29+**

GKE 1.29 and later needs a custom `clusterNetworks` value. Add this to your
`ControlPlane` configuration:

```yaml
spec:
  components:
    linkerd:
      controlPlaneConfig:
        clusterNetworks: "34.118.224.0/20,10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16,fd00::/8"
```

For more information, see the
[GKE docs](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#-managed_secondary_ranges_default).

**End of option: GKE 1.29+**

### A5. Install BEL

Apply the configuration. The lifecycle operator then installs the Linkerd
control plane:

```bash
kubectl apply -f linkerd-control-plane-config.yaml
```

**Option: Trust-rotation operator**

Continue to
[Install the trust-rotation operator](#install-the-trust-rotation-operator).

**End of option: Trust-rotation operator**

Otherwise, skip to [Verify your installation](#verify-your-installation).

## Option B: Install with Helm

### B1. Install the Linkerd CNI plugin

**Option: CNI plugin**

The `linkerd-cni` DaemonSet must be running on your cluster before you install
the BEL control plane:

```bash
helm repo add linkerd-edge https://helm.linkerd.io/edge
helm repo update
helm search repo linkerd2-cni
helm install linkerd-cni -n linkerd-cni --create-namespace linkerd-edge/linkerd2-cni
# ensure the plugin is installed and ready
linkerd check --pre --linkerd-cni-enabled
```

**End of option: CNI plugin**

### B2. Create the TLS infrastructure

Choose one approach:

- **Generate certificates manually.** Follow B2a below.
- **Trust-rotation operator.** Follow B2b below. The trust-rotation operator
  requires cert-manager.

#### B2a. Generate certificates manually

> [!NOTE]
> TLS key management is a complex topic. In this guide we create the
> certificates by hand. [Automated rotation](https://linkerd.io/2.20/tasks/automatically-rotating-control-plane-tls-credentials/)
> of these credentials is common in production environments, but it's outside the
> scope of this guide.

You need to generate a trust anchor certificate and key pair, and an issuer
certificate and key pair. Pay attention to the lifetimes of these certificates,
especially the trust anchor, and keep the files somewhere safe.

Generate the trust anchor certificate and key with a tool like
[`step`](https://smallstep.com/docs/step-cli/installation):

```bash
step certificate create root.linkerd.cluster.local ca.crt ca.key \
--profile root-ca --no-password --insecure
```

This creates `ca.crt` and `ca.key`. The `--no-password --insecure` flags leave
the files unencrypted, with no passphrase. For a longer-lived trust anchor, add
`--not-after` with the lifetime you want (for example, `--not-after=87600h`).

Generate the issuer certificate and key, which sign the Linkerd proxies'
certificate signing requests:

```bash
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca.crt --ca-key ca.key
```

This creates `issuer.crt` and `issuer.key`. You'll pass them to Helm in the next
step.

#### B2b. Use cert-manager (trust-rotation operator)

**Option: Trust-rotation operator**

> [!NOTE]
> BEL's [trust-rotation operator](../../features/trust-rotation/)
> automates trust anchor rotation. It relies on
> [cert-manager](https://cert-manager.io/) to issue Linkerd's trust anchor and
> identity issuer certificates. For details, see
> [Configuring BEL's trust-rotation operator](../../guides/configuring-trust-rotation/).

If cert-manager isn't already installed on your cluster, install it:

```bash
helm repo add jetstack https://charts.jetstack.io --force-update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true \
  --wait
```

Install the `linkerd-cert-manager` chart. It creates the cert-manager `Issuer`
and `Certificate` resources that produce Linkerd's trust anchor and identity
issuer secrets:

```bash
helm install linkerd-cert-manager \
  --namespace linkerd \
  --create-namespace \
  oci://ghcr.io/linkerd/charts/linkerd-cert-manager \
  --wait
```

Linkerd reads its trust roots from the `linkerd-identity-trust-roots` ConfigMap.
That ConfigMap has to be seeded from cert-manager's `linkerd-trust-anchor`
Secret before you install Linkerd. The `linkerd trust bundle` command renders it
for you:

```bash
linkerd trust bundle | kubectl apply -f -
```

**End of option: Trust-rotation operator**

### B3. Install BEL

Add or update the `linkerd-buoyant` Helm repo:

```bash
helm repo add linkerd-buoyant https://helm.buoyant.cloud
helm repo update
```

Install the BEL CRDs and control plane.

**If you generated certificates manually (B2a):** run this from the directory
that contains `ca.crt`, `issuer.crt`, and `issuer.key`.

```bash
helm install linkerd-crds \
  --create-namespace \
  --namespace linkerd \
  linkerd-buoyant/linkerd-enterprise-crds

helm install linkerd-control-plane \
  --namespace linkerd \
  --set license=$BUOYANT_LICENSE \
  --set-file identityTrustAnchorsPEM=ca.crt \
  --set-file identity.issuer.tls.crtPEM=issuer.crt \
  --set-file identity.issuer.tls.keyPEM=issuer.key \
  linkerd-buoyant/linkerd-enterprise-control-plane
```

**If you used cert-manager (B2b):** this version tells Linkerd to use the trust
bundle and identity issuer that cert-manager manages.

```bash
helm install linkerd-crds \
  --create-namespace \
  --namespace linkerd \
  linkerd-buoyant/linkerd-enterprise-crds

helm install linkerd-control-plane \
  --namespace linkerd \
  --set license=$BUOYANT_LICENSE \
  --set identity.externalCA=true \
  --set identity.issuer.scheme=kubernetes.io/tls \
  linkerd-buoyant/linkerd-enterprise-control-plane
```

**Option: High availability (HA)**

Before you install the control plane, download the chart so you have its HA
values file:

```bash
helm fetch --untar linkerd-buoyant/linkerd-enterprise-control-plane
```

Then add this flag to the `linkerd-control-plane` install:

```bash
--values linkerd-enterprise-control-plane/values-ha.yaml
```

**End of option: High availability (HA)**

**Option: CNI plugin**

Add this flag to the `linkerd-control-plane` install:

```bash
--set cniEnabled=true
```

**End of option: CNI plugin**

**Option: FIPS modules**

Add these flags to the `linkerd-control-plane` install:

```bash
--set linkerdVersion=enterprise-2.20.6-fips
--set proxyInit.image.version=enterprise-2.20.6-fips
```

**End of option: FIPS modules**

**Option: GKE 1.29+**

GKE 1.29 and later needs a custom `clusterNetworks` value. Add this flag to the
`linkerd-control-plane` install:

```bash
--set clusterNetworks=34.118.224.0/20\,10.0.0.0/8\,100.64.0.0/10\,172.16.0.0/12\,192.168.0.0/16\,fd00::/8
```

For more information, see the
[GKE docs](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#-managed_secondary_ranges_default).

**End of option: GKE 1.29+**

### B4. Install the Buoyant Cloud agent

**Option: Buoyant Cloud**

Replace `[YOUR_CLUSTER_NAME]` with the name that will identify this cluster in
Buoyant Cloud:

```bash
helm install linkerd-buoyant \
  --create-namespace \
  --namespace linkerd-buoyant \
  --set metadata.agentName=[YOUR_CLUSTER_NAME] \
  --set api.clientID=$API_CLIENT_ID \
  --set api.clientSecret=$API_CLIENT_SECRET \
  linkerd-buoyant/linkerd-buoyant
```

**End of option: Buoyant Cloud**

**Option: Trust-rotation operator**

Continue to
[Install the trust-rotation operator](#install-the-trust-rotation-operator).

**End of option: Trust-rotation operator**

Otherwise, skip to [Verify your installation](#verify-your-installation).

## Option C: Install with the CLI

> [!WARNING]
> Installing BEL with the CLI is quick and easy, but the result is
> optimized for non-production use. The TLS certificates it generates expire after
> one year, and the trust-rotation operator isn't supported. For production
> traffic, use the
> [lifecycle operator](#option-a-install-with-the-lifecycle-operator) or
> [Helm](#option-b-install-with-helm).

### C1. Install the Linkerd CNI plugin

**Option: CNI plugin**

```bash
linkerd install-cni | kubectl apply -f -
# ensure the plugin is installed and ready
linkerd check --pre --linkerd-cni-enabled
```

**End of option: CNI plugin**

### C2. Install BEL

```bash
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
```

**Option: High availability (HA)**

Add `--ha` to the second `linkerd install` command.

**End of option: High availability (HA)**

**Option: CNI plugin**

Add `--linkerd-cni-enabled` to the second `linkerd install` command.

**End of option: CNI plugin**

For example, with both HA and CNI:
`linkerd install --ha --linkerd-cni-enabled | kubectl apply -f -`

**Option: GKE 1.29+**

GKE 1.29 and later needs a custom `clusterNetworks` value. Add this flag to the
second `linkerd install` command:

```bash
--set clusterNetworks=34.118.224.0/20\,10.0.0.0/8\,100.64.0.0/10\,172.16.0.0/12\,192.168.0.0/16\,fd00::/8
```

For more information, see the
[GKE docs](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#-managed_secondary_ranges_default).

**End of option: GKE 1.29+**

### C3. Install the Buoyant Cloud agent

**Option: Buoyant Cloud**

Replace `[YOUR_CLUSTER_NAME]` with the name that will identify this cluster in
Buoyant Cloud:

```bash
BUOYANT_CLOUD_CLIENT_ID=$API_CLIENT_ID \
BUOYANT_CLOUD_CLIENT_SECRET=$API_CLIENT_SECRET \
linkerd buoyant install --cluster-name=[YOUR_CLUSTER_NAME] | kubectl apply -f -
```

**End of option: Buoyant Cloud**

Continue to [Verify your installation](#verify-your-installation).

## Install the trust-rotation operator

**Option: Trust-rotation operator**

Lifecycle operator and Helm methods only.

The trust-rotation operator is published as two Helm charts: a CRD chart and the
operator chart. Install the CRD chart first, because the operator chart expects
the `TrustAnchorRotation` CRD to exist already:

```bash
helm install trust-rotation-operator-crds \
  --namespace linkerd \
  oci://ghcr.io/buoyantio/charts/trust-rotation-operator-crds \
  --devel --wait

helm install trust-rotation-operator \
  --namespace linkerd \
  oci://ghcr.io/buoyantio/charts/trust-rotation-operator \
  --devel --wait
```

By default, the operator chart creates a single `TrustAnchorRotation` resource
named `cluster`. To learn how to rotate the trust anchor, see
[Performing a trust-anchor rotation](../../guides/configuring-trust-rotation/#performing-a-trust-anchor-rotation).

**End of option: Trust-rotation operator**

## Verify your installation

When the installation finishes, check Linkerd's health and configuration:

```bash
linkerd check
```

**Option: Trust-rotation operator**

The `linkerd trust` CLI extension registers itself with `linkerd check`, so this
also confirms that cert-manager, the Linkerd PKI resources, and the
trust-rotation operator are installed and healthy.

**End of option: Trust-rotation operator**

## That's it!

You've installed the enterprise distribution of Linkerd on your Kubernetes
cluster.

- **Lifecycle operator:** Future upgrades are simple and can be managed
  declaratively, in a GitOps workflow.
- **CLI:** Your installation is optimized for non-production traffic, such as
  demos and experiments.
