HAZL reference

Buoyant Enterprise for Linkerd’s High Availability Zonal Load balancer (HAZL) is a dynamic request-level load balancer that can dramatically reduce cloud spend by minimizing cross-zone traffic. Unlike Kubernetes’s native Topology Aware Routing feature, HAZL never sacrifices reliability to achieve this cost reduction.

Load band

HAZL’s behavior is configured with a load band. This load band determines the nominal load for a system. When system load is within this band, HAZL will not change its behavior. When system performance falls outside this band, HAZL will change its behavior:

• When system load exceeds the high value of the load band, HAZL will start adding endpoints from non-local zones.
• When system load falls below the low value of the load band, HAZL will start removing endpoints from non-local zones.

System load is defined as the product of latency and throughput for a single endpoint. For example, given a client sending 50 requests per second to an endpoint that has a median latency of 0.02 seconds, the load for that endpoint is 50 * 0.02 = 1.0. If a Kubernetes service with 10 endpoints has an overall median latency of 0.02 seconds, at 50 RPS we would expect load per endpoint to be 0.1.

Note that the load corresponds roughly to the number of in-flight and queued requests we would expect at any moment for a single endpoint. This can serve as a useful shorthand when configuring HAZL.

Note also that factors such as retry configuration, circuit breaking, and ongoing long-lived requests can all influence the exact sequence of changes to the load balancing pool.

Configuring HAZL

controlPlaneConfig:
  destinationController:
    additionalArgs:
      - -ext-endpoint-zone-weights # Flag to enable HAZL
  proxy:
    additionalEnv:
      - name: BUOYANT_BALANCER_LOAD_LOW
        value: "0.8"
      - name: BUOYANT_BALANCER_LOAD_HIGH
        value: "2.0"

helm upgrade linkerd-control-plane -n linkerd --reuse-values \
  --set "destinationController.additionalArgs[0]=-ext-endpoint-zone-weights" \
  --set "proxy.additionalEnv[0].name=BUOYANT_BALANCER_LOAD_LOW" \
  --set "proxy.additionalEnv[0].value=0.8" \
  --set "proxy.additionalEnv[1].name=BUOYANT_BALANCER_LOAD_HIGH" \
  --set "proxy.additionalEnv[1].value=0.2" \
  linkerd-buoyant/linkerd-enterprise-control-plane

linkerd upgrade \
  --set "destinationController.additionalArgs[0]=-ext-endpoint-zone-weights" \
  --set "proxy.additionalEnv[0].name=BUOYANT_BALANCER_LOAD_LOW" \
  --set "proxy.additionalEnv[0].value=0.8" \
  --set "proxy.additionalEnv[1].name=BUOYANT_BALANCER_LOAD_HIGH" \
  --set "proxy.additionalEnv[1].value=0.2" | kubectl apply -f -

The values are defined as follows:

Note: If the default load bands are changed, you will have to restart your application proxies for the new values to take effect.