# Tuning HAZL

In this guide we'll walk through how to tune [HAZL](/buoyant-enterprise-linkerd/2.17/features/hazl/) in a
production environment.

The goal of tuning is to find a load band min and max such that:

- The load average is within the min and max under normal operation;
- The load average exceeds the max when you want to add out-of-zone endpoints;
- The load average drops below the min when you want to remove out-of-zone
  endpoints.

Since this is currently a global config, we recommend starting with your most
costly / important service and getting that those work, then potentially trying
to also tune so that secondary services behave how you want as well. (This may
not always be possible.)

## Tuning goals

When HAZL is restricting traffic to the same zone, let’s call that its
**throttled** state. When it’s allowing cross-zone traffic, let’s call that its
**unthrottled** state. (Note that HAZL really operates as a gradient, not in a
binary manner, but for tuning simplicity we’ll treat it as binary.)

Similarly, when the workload is operating in a healthy manner, let’s call that
its **normal** state. When it’s not operating healthily, let’s call that its
**stressed** state.

Given that, our basic goal here is to find:

1. A band max that causes HAZL to add endpoints when the system is in a
   throttled (single-zone) state but stressed; (this is our **throttled stressed
   loadavg** number)
2. A band min that causes HAZL to remove endpoints when the system is in an
   unthrottled (multi-zone) state but with normal load; (our **unthrottled
   normal loadavg** number)

## Critical things to understand before tuning

**Each proxy maintains its own view of the universe and makes its own
decisions**. In high-traffic systems we expect all proxies for a given workload
to behave the same way since they have the same config and see the same set of
endpoints. But they don’t directly share information, and in low-traffic systems
especially, individual proxies state can diverge.

**HAZL changes client-side behavior only.** Each proxy that makes an outbound
call to a workload is deciding where to send that traffic, which is what HAZL
controls. The proxy on the server side receiving the traffic does not use HAZL.
(Corollary: HAZL works regardless of whether the destination service is meshed
or unmeshed, or even whether it’s in-cluster or off-cluster.)

**HAZL works by adding and removing endpoints from a pool.** An endpoint is an
IP/port combination. A pool is a set of endpoints that are eligible for traffic.
Once an endpoint is in the pool, Linkerd’s regular load balancing (called
variously “p2c” and “EWMA” in the docs and metrics) takes it from there,
selecting a specific endpoint from the pool based on observed latency.

**HAZL uses a metric called “load”**. This is basically the number of
outstanding requests to an endpoint. A system at rest has a load of 0. The
normal load for a workload is a function of many factors, but typical values
would be <5.

**HAZL makes decisions based on the “average load” for the pool as a whole.**
Specifically, it adds or removes endpoints based on the average load number
across all _active_ endpoints in the pool (where _active_ means that it is in
the current pool and eligible for traffic, as opposed to _inactive_ endpoints
that HAZL knows about but hasn’t added to the pool.) If the average load is
higher than the load band max, HAZL will add endpoints to the pool. If it falls
below the load band min, HAZL will remove endpoints from the pool.

**HAZL can only be configured at the global (cluster) level.** This is a known
limitation that will be addressed in future BEL releases.

## Tuning approach

![Tuning HAZL](/buoyant-enterprise-linkerd/2.17/guides/tuning-hazl/tuning-hazl.png)

In this tuning, we’ll look purely at the empirical load numbers and build out a
min and max from there. It may be helpful to start with min set to 0 and max set
to 1000, to avoid accidentally triggering a state change before we’re ready.

1. **(A)** Start with the workload in a “normal” state — taking traffic and
   returning successful responses with expected latency.
2. Confirm that HAZL is enabled. There should be no cross-zone traffic
   happening.
3. Observe the current `outbound_http_balancer_adaptive_load_average` for
   traffic to the workload. This is our **throttled normal loadavg**. Update the
   band max so it is comfortably above this loadavg.
4. Put the system in a “stressed” state—add enough traffic or induce latency to
   the system the you would want to pull in endpoints from other zones.
5. **(B)** Observe the current `outbound_http_balancer_adaptive_load_average`
   for traffic to the workload. This is our **throttled stressed loadavg**. It
   should be higher than the throttled normal loadavg, since the system is
   stressed. Update the load band max so that it is _below_ this value, but
   still higher than the throttled nomal loadavg—we want HAZL adding endpoints
   in this situation.
6. After updating the band max, HAZL should now allow cross-zone traffic. Ensure
   this happens. (Note that you can see the band min and max as proxy metrics,
   so you can double-check that they have picked up the updates from the control
   plane.)
7. **(C)** Observe the current `outbound_http_balancer_adaptive_load_average`
   for traffic to the workload. This is our **unthrottled stressed loadavg**. It
   should be lower than the unthrottled stressed loadavg, because HAZL has
   pulled in out-of-cluster endpoints to handle extra load. Ensure the band min
   is _below_ this value. We don’t want HAZL removing endpoints in this
   situation.
8. Return the system to a “normal” state by removing the excess traffic,
   latency, etc.
9. **(D)** Observe the current `outbound_http_balancer_adaptive_load_average`
   for traffic to the workload. This is our **unthrottled normal loadavg**. It
   should be lower than the unthrottled stressed loadavg, because the system is
   no longer stressed. (This should probably be the lowest loadavg seen, since
   the system is unstressed _and_ cross-zone endpoints are available.) Update
   the band min so that it is _above_ this value, but still below the
   unthrottled stressed loadavg from the previous step—we want HAZL removing
   endpoints in this situation.
10. After updating the load band min, HAZL should now remove cross-zone traffic.
    Ensure this happens.
