# HAZL reference

Buoyant Enterprise for Linkerd's
[High Availability Zonal Load balancer (HAZL)](/buoyant-enterprise-linkerd/latest/features/hazl/) is a dynamic
request-level load balancer that can dramatically reduce cloud spend by
minimizing cross-zone traffic. Unlike Kubernetes's native Topology Aware Routing
feature, HAZL never sacrifices reliability to achieve this cost reduction.

## Load band

HAZL's behavior is configured with a _load band_. This load band determines the
nominal load for a system. When system load is within this band, HAZL will not
change its behavior. When system performance falls outside this band, HAZL will
change its behavior:

- When system load exceeds the **high** value of the load band, HAZL will start
  adding endpoints from non-local zones.
- When system load falls below the **low** value of the load band, HAZL will
  start removing endpoints from non-local zones.

System load is defined as the _product of latency and throughput for a single
endpoint_. For example, given a client sending 50 requests per second to an
endpoint that has a median latency of 0.02 seconds, the load for that endpoint
is _50 \* 0.02 = 1.0_. If a Kubernetes service with 10 endpoints has an overall
median latency of 0.02 seconds, at 50 RPS we would expect load per endpoint to
be 0.1.

Note that the load corresponds roughly to the number of in-flight and queued
requests we would expect at any moment for a single endpoint. This can serve as
a useful shorthand when configuring HAZL.

Note also that factors such as retry configuration, circuit breaking, and
ongoing long-lived requests can all influence the exact sequence of changes to
the load balancing pool.

## Configuring HAZL

**Lifecycle operator:**

HAZL's default load bands are configured within the `additionalEnv` block within
the `proxy` block in the control plane's `controlPlaneConfig` configuration. For
example:

```yaml
controlPlaneConfig:
  destinationController:
    additionalArgs:
      - -ext-endpoint-zone-weights # Flag to enable HAZL
  proxy:
    additionalEnv:
      - name: BUOYANT_BALANCER_LOAD_LOW
        value: "0.8"
      - name: BUOYANT_BALANCER_LOAD_HIGH
        value: "2.0"
```

**Helm:**

HAZL's default load bands are configured by setting `additionalEnv` values on
the proxy configuration. For example:

```bash
helm upgrade linkerd-control-plane -n linkerd --reuse-values \
  --set "destinationController.additionalArgs[0]=-ext-endpoint-zone-weights" \
  --set "proxy.additionalEnv[0].name=BUOYANT_BALANCER_LOAD_LOW" \
  --set-string "proxy.additionalEnv[0].value=0.8" \
  --set "proxy.additionalEnv[1].name=BUOYANT_BALANCER_LOAD_HIGH" \
  --set-string "proxy.additionalEnv[1].value=0.2" \
  linkerd-buoyant/linkerd-enterprise-control-plane
```

**CLI:**

HAZL's default load bands are configured by setting `additionalEnv` values on
the proxy configuration. For example:

```bash
linkerd upgrade \
  --set "destinationController.additionalArgs[0]=-ext-endpoint-zone-weights" \
  --set "proxy.additionalEnv[0].name=BUOYANT_BALANCER_LOAD_LOW" \
  --set-string "proxy.additionalEnv[0].value=0.8" \
  --set "proxy.additionalEnv[1].name=BUOYANT_BALANCER_LOAD_HIGH" \
  --set-string "proxy.additionalEnv[1].value=0.2" | kubectl apply -f -
```

The values are defined as follows:

| Variable                     | Description                           | Default |
| ---------------------------- | ------------------------------------- | ------- |
| `BUOYANT_BALANCER_LOAD_LOW`  | The low value of the HAZL load band.  | 0.8     |
| `BUOYANT_BALANCER_LOAD_HIGH` | The high value of the HAZL load band. | 2.0     |

**Note:** If the default load bands are changed, you will have to restart your
application proxies for the new values to take effect.

HAZL's load bands can also be configured on a per-workload basis by setting the
`balancer.buoyant.io/load-band` annotation on the pod spec template. In this
mode, the specified load band will apply to all outbound traffic from that
workload to any service. (In the future, specifying load bands for individual
services will also be possible.) This annotation may take any of the following
values:

- `low,high` where `low` and `high` are non-negative floating point numbers
  representing the low and high ends of the load band.
  - e.g. `balancer.buoyant.io/load-band: "0.8,10.0"`
- `disabled` which indicates that HAZL behavior should be disabled by setting
  the low and high ends of the load band to 0. This is equivalent to `"0,0"`.

For example:

```yaml
kind: Deployment
apiVersion: apps/v1
metadata:
  # ...
spec:
  template:
    metadata:
      annotations:
        balancer.buoyant.io/load-band: "0.8,10.0"
```

The annotation can be set on pods or namespaces. The value set on a pod will be
used if present. Otherwise, the value on the namespace will be used if present.
Otherwise, the cluster default will be used.