Tuning HAZL

In this guide we’ll walk through how to tune HAZL in a production environment.

The goal of tuning is to find a load band min and max such that:

• The load average is within the min and max under normal operation;
• The load average exceeds the max when you want to add out-of-zone endpoints;
• The load average drops below the min when you want to remove out-of-zone endpoints.

Since this is currently a global config, we recommend starting with your most costly / important service and getting that those work, then potentially trying to also tune so that secondary services behave how you want as well. (This may not always be possible.)

Tuning goals

When HAZL is restricting traffic to the same zone, let’s call that its throttled state. When it’s allowing cross-zone traffic, let’s call that its unthrottled state. (Note that HAZL really operates as a gradient, not in a binary manner, but for tuning simplicity we’ll treat it as binary.)

Similarly, when the workload is operating in a healthy manner, let’s call that its normal state. When it’s not operating healthily, let’s call that its stressed state.

Given that, our basic goal here is to find:

1. A band max that causes HAZL to add endpoints when the system is in a throttled (single-zone) state but stressed; (this is our throttled stressed loadavg number)
2. A band min that causes HAZL to remove endpoints when the system is in an unthrottled (multi-zone) state but with normal load; (our unthrottled normal loadavg number)

Critical things to understand before tuning

Each proxy maintains its own view of the universe and makes its own decisions. In high-traffic systems we expect all proxies for a given workload to behave the same way since they have the same config and see the same set of endpoints. But they don’t directly share information, and in low-traffic systems especially, individual proxies state can diverge.

HAZL changes client-side behavior only. Each proxy that makes an outbound call to a workload is deciding where to send that traffic, which is what HAZL controls. The proxy on the server side receiving the traffic does not use HAZL. (Corollary: HAZL works regardless of whether the destination service is meshed or unmeshed, or even whether it’s in-cluster or off-cluster.)

HAZL works by adding and removing endpoints from a pool. An endpoint is an IP/port combination. A pool is a set of endpoints that are eligible for traffic. Once an endpoint is in the pool, Linkerd’s regular load balancing (called variously “p2c” and “EWMA” in the docs and metrics) takes it from there, selecting a specific endpoint from the pool based on observed latency.

HAZL uses a metric called “load”. This is basically the number of outstanding requests to an endpoint. A system at rest has a load of 0. The normal load for a workload is a function of many factors, but typical values would be <5.

HAZL makes decisions based on the “average load” for the pool as a whole. Specifically, it adds or removes endpoints based on the average load number across all active endpoints in the pool (where active means that it is in the current pool and eligible for traffic, as opposed to inactive endpoints that HAZL knows about but hasn’t added to the pool.) If the average load is higher than the load band max, HAZL will add endpoints to the pool. If it falls below the load band min, HAZL will remove endpoints from the pool.

HAZL can only be configured at the global (cluster) level. This is a known limitation that will be addressed in future BEL releases.

Tuning approach

In this tuning, we’ll look purely at the empirical load numbers and build out a min and max from there. It may be helpful to start with min set to 0 and max set to 1000, to avoid accidentally triggering a state change before we’re ready.

1. (A) Start with the workload in a “normal” state — taking traffic and returning successful responses with expected latency.
2. Confirm that HAZL is enabled. There should be no cross-zone traffic happening.
3. Observe the current outbound_http_balancer_adaptive_load_average for traffic to the workload. This is our throttled normal loadavg. Update the band max so it is comfortably above this loadavg.
4. Put the system in a “stressed” state—add enough traffic or induce latency to the system the you would want to pull in endpoints from other zones.
5. (B) Observe the current outbound_http_balancer_adaptive_load_average for traffic to the workload. This is our throttled stressed loadavg. It should be higher than the throttled normal loadavg, since the system is stressed. Update the load band max so that it is below this value, but still higher than the throttled nomal loadavg—we want HAZL adding endpoints in this situation.
6. After updating the band max, HAZL should now allow cross-zone traffic. Ensure this happens. (Note that you can see the band min and max as proxy metrics, so you can double-check that they have picked up the updates from the control plane.)
7. (C) Observe the current outbound_http_balancer_adaptive_load_average for traffic to the workload. This is our unthrottled stressed loadavg. It should be lower than the unthrottled stressed loadavg, because HAZL has pulled in out-of-cluster endpoints to handle extra load. Ensure the band min is below this value. We don’t want HAZL removing endpoints in this situation.
8. Return the system to a “normal” state by removing the excess traffic, latency, etc.
9. (D) Observe the current outbound_http_balancer_adaptive_load_average for traffic to the workload. This is our unthrottled normal loadavg. It should be lower than the unthrottled stressed loadavg, because the system is no longer stressed. (This should probably be the lowest loadavg seen, since the system is unstressed and cross-zone endpoints are available.) Update the band min so that it is above this value, but still below the unthrottled stressed loadavg from the previous step—we want HAZL removing endpoints in this situation.
10. After updating the load band min, HAZL should now remove cross-zone traffic. Ensure this happens.