# Enabling HAZL

Buoyant Enterprise for Linkerd provides an opt-in
[High Availability Zonal Load balancer](/buoyant-enterprise-linkerd/2.15/features/hazl/) option that
intelligently balances HTTP and gRPC traffic with built-in zone awareness in
environments with multiple availability zones.

This feature is disabled by default and must be explicitly enabled in the
configuration. This guide demonstrates the before and after effects of enabling
HAZL.

## Prerequisites

- Kubernetes cluster with three zone-labeled nodes (zone-0|zone-1|zone-2)
- [Buoyant Enterprise for Linkerd](/buoyant-enterprise-linkerd/2.15/installation/) installed
- The `BUOYANT_LICENSE` environment variable set, with functioning BEL CLI

## Caveats

- For preserving zone locality in multi-cluster situations, HAZL currently
  requires [pod-to-pod multicluster](https://linkerd.io/2.15/tasks/pod-to-pod-multicluster/) and does not work with
  gateway-based multicluster deployments.
- HAZL-enabled workloads will dynamically shift traffic out of the current zone
  any time the in-zone endpoint's capability to handle the incoming request load
  is exceeded and threatens to impact the success rate of the requests.

## Step 1: Getting started

Your cluster will need a minimum of three nodes, with the following labels
applied:

- `topology.kubernetes.io/zone=zone-0`
- `topology.kubernetes.io/zone=zone-1`
- `topology.kubernetes.io/zone=zone-2`

You will need a minimum of one node with each of the labels for the `colorz`
application to deploy properly.

If you need a cluster with Buoyant Enterprise for Linkerd installed, you can use
`k3d` to deploy one using the assets and instructions in the
[Deploying BEL with HAZL](https://github.com/BuoyantIO/service-mesh-academy/tree/main/deploying-bel-with-hazl)
GitHub repository. Following the instructions in the repository, the cluster
will have properly labeled nodes for the three zones, with Buoyant Enterprise
for Linkerd deployed.

## Step 2: Install sample app

If you do not already have meshed workloads on your cluster, you can install and
mesh the sample `colorz` app to test HAZL:

```bash
kubectl apply -k https://github.com/BuoyantIO/service-mesh-academy/deploying-bel-with-hazl/colorz/
```

Confirm all pods are running, meshed, and distributed across different nodes by
running the following command:

```bash
kubectl get pods -n colorz -o wide
```

You should see the color pods distributed one-per-node, with the brush pod
sharing a node with one of the color pods.

## Step 3: Gather baseline metrics

Before we enable HAZL we need to gather baseline zone distribution metrics to
compare with metrics after HAZL activation. To do so, run the following command:

```bash
linkerd dg proxy-metrics deploy/brush -n colorz | \
grep 'request_total' | \
awk -F'[ ,}]' '{
    pod="";
    zone="";
    requests=0;
    for(i=1;i<=NF;i++) {
        if($i ~ /^dst_pod=/) pod=substr($i, 9);
        else if($i ~ /^dst_zone=/) zone=substr($i, 10);
        else if($i ~ /^[0-9]+$/) requests=$i
    }
    if($0 ~ /direction="outbound"/ && $0 ~ /tls="true"/ && pod != "" && zone != "")
        print "Pod: " pod " | Zone: " zone ": " requests
}'
```

You should see output similar to this:

```bash
Pod: "green-8f8d79cb8-wmnr4" | Zone: "zone-1": 72483
Pod: "blue-657dcc787c-kgpsb" | Zone: "zone-2": 69010
Pod: "red-6d6b7c4f8f-ktg65"  | Zone: "zone-0": 66588
```

**Note:** If you're using your own application for testing you will need to
adjust the deployment name and namespace in the command above to reference your
client's deployment name and namespace.

## Step 4: Enable HAZL

**Lifecycle operator:**

To enable HAZL you will need to modify your `ControlPlane` CRD to set the
`ext-endpoint-zone-weights` flag, and apply it to your cluster:

```yaml
controlPlaneConfig:
  destinationController:
    additionalArgs:
      - -ext-endpoint-zone-weights
```

> [!NOTE]
> Note the intentional double dash on the
> `ext-endpoint-zone-weights` flag.

**Helm:**

To enable HAZL you will need to modify your Helm values file to set the
`ext-endpoint-zone-weights` flag:

```bash
helm upgrade linkerd-control-plane -n linkerd --reuse-values --set "destinationController.additionalArgs[0]=-ext-endpoint-zone-weights" linkerd-buoyant/linkerd-enterprise-control-plane
```

**CLI:**

To enable HAZL you will need to run `linkerd upgrade` to set the
`ext-endpoint-zone-weights` flag:

```bash
linkerd upgrade --set "destinationController.additionalArgs[0]=-ext-endpoint-zone-weights" | kubectl apply -f -
```

After applying the new configuration you will want to wait for the updated
control plane to come up, then run the following to restart the application
proxies and reset metrics:

```bash
kubectl rollout restart deployment blue red green -n colorz && sleep 30 && kubectl rollout restart deploy brush -n colorz
```

> [!NOTE]
> You will not have to restart your application proxies in an
> actual production environment after the HAZL flag is enabled and the control
> plane is rolled, unless the default load bands have been changed. We are
> restarting them here and in a specific order to more clearly and visibly
> demonstrate the difference in request distribution via the metrics we're
> comparing, but it is not a requirement for HAZL itself.

## Step 5: Verify HAZL enablement and functionality

To confirm the settings have been applied confirm that new control-plane
components have been created, and check the running configuration for the
`ext-endpoint-zone-weights` flag like so:

```bash
kubectl get configmap linkerd-config -n linkerd -o yaml | grep 'ext-endpoint-zone-weights'
```

After confirming the configuration has been successfully applied we will need to
confirm our new application pod distribution with the following command:

```bash
kubectl get pods -n colorz -o wide
```

Note which color pod shares a node with the new brush pod this time.

```bash
NAME                     READY   STATUS    RESTARTS   AGE    IP           NODE                       NOMINATED NODE   READINESS GATES
blue-5bbc87784c-bt4nd    2/2     Running   0          100s   10.42.2.11   k3d-demo-cluster-agent-2   <none>           <none>
red-6774947bbc-kvjn9     2/2     Running   0          100s   10.42.0.16   k3d-demo-cluster-agent-0   <none>           <none>
green-f4dccb54b-rpxpv    2/2     Running   0          100s   10.42.1.15   k3d-demo-cluster-agent-1   <none>           <none>
brush-6dd6db689f-x4474   2/2     Running   0          70s    10.42.0.17   k3d-demo-cluster-agent-0   <none>           <none>
```

We can now dump the metrics again and compare with the pre-HAZL ones we gathered
previously:

```bash
linkerd dg proxy-metrics deploy/brush -n colorz | \
grep 'request_total' | \
awk -F'[ ,}]' '{
    pod="";
    zone="";
    requests=0;
    for(i=1;i<=NF;i++) {
        if($i ~ /^dst_pod=/) pod=substr($i, 9);
        else if($i ~ /^dst_zone=/) zone=substr($i, 10);
        else if($i ~ /^[0-9]+$/) requests=$i
    }
    if($0 ~ /direction="outbound"/ && $0 ~ /tls="true"/ && pod != "" && zone != "")
        print "Pod: " pod " | Zone: " zone ": " requests
}'
```

You should see output similar to this:

```bash
Pod: "red-6774947bbc-kvjn9" | Zone: "zone-0": 3749
```

With HAZL enabled all requests are now routing to in-zone backend endpoints and
we've completely eliminated all cross-zone traffic.

## Walkthrough

[Video](https://vimeo.com/938377441#t=319s)

## Learning more

For more information about using this feature, see the
[HAZL reference](/buoyant-enterprise-linkerd/2.15/reference/hazl/) page.
