HAZL reference

Buoyant Enterprise for Linkerd’s High Availability Zonal Load balancer (HAZL) is a dynamic request-level load balancer that can dramatically reduce cloud spend by minimizing cross-zone traffic. Unlike Kubernetes’s native Topology Aware Routing feature, HAZL never sacrifices reliability to achieve this cost reduction.

Load band

HAZL’s behavior is configured with a load band. This load band determines the nominal load for a system. When system load is within this band, HAZL will not change its behavior. When system performance falls outside this band, HAZL will change its behavior:

• When system load exceeds the high value of the load band, HAZL will start adding endpoints from non-local zones.
• When system load falls below the low value of the load band, HAZL will start removing endpoints from non-local zones.

System load is defined as the product of latency and throughput for a single endpoint. For example, given a client sending 50 requests per second to an endpoint that has a median latency of 0.02 seconds, the load for that endpoint is 50 * 0.02 = 1.0. If a Kubernetes service with 10 endpoints has an overall median latency of 0.02 seconds, at 50 RPS we would expect load per endpoint to be 0.1.

Note that the load corresponds roughly to the number of in-flight and queued requests we would expect at any moment for a single endpoint. This can serve as a useful shorthand when configuring HAZL.

Note also that factors such as retry configuration, circuit breaking, and ongoing long-lived requests can all influence the exact sequence of changes to the load balancing pool.

Configuring HAZL

controlPlaneConfig:
  destinationController:
    additionalArgs:
      - -ext-endpoint-zone-weights # Flag to enable HAZL
  proxy:
    additionalEnv:
      - name: BUOYANT_BALANCER_LOAD_LOW
        value: "0.8"
      - name: BUOYANT_BALANCER_LOAD_HIGH
        value: "2.0"

helm upgrade linkerd-control-plane -n linkerd --reuse-values \
  --set "destinationController.additionalArgs[0]=-ext-endpoint-zone-weights" \
  --set "proxy.additionalEnv[0].name=BUOYANT_BALANCER_LOAD_LOW" \
  --set-string "proxy.additionalEnv[0].value=0.8" \
  --set "proxy.additionalEnv[1].name=BUOYANT_BALANCER_LOAD_HIGH" \
  --set-string "proxy.additionalEnv[1].value=0.2" \
  linkerd-buoyant/linkerd-enterprise-control-plane

linkerd upgrade \
  --set "destinationController.additionalArgs[0]=-ext-endpoint-zone-weights" \
  --set "proxy.additionalEnv[0].name=BUOYANT_BALANCER_LOAD_LOW" \
  --set-string "proxy.additionalEnv[0].value=0.8" \
  --set "proxy.additionalEnv[1].name=BUOYANT_BALANCER_LOAD_HIGH" \
  --set-string "proxy.additionalEnv[1].value=0.2" | kubectl apply -f -

The values are defined as follows:

Note: If the default load bands are changed, you will have to restart your application proxies for the new values to take effect.

HAZL’s load bands can also be configured on a per-workload basis by setting the balancer.buoyant.io/load-band annotation on the pod spec template. In this mode, the specified load band will apply to all outbound traffic from that workload to any service. (In the future, specifying load bands for individual services will also be possible.) This annotation may take any of the following values:

• low,high where low and high are non-negative floating point numbers representing the low and high ends of the load band.
  • e.g. balancer.buoyant.io/load-band: "0.8,10.0"
• disabled which indicates that HAZL behavior should be disabled by setting the low and high ends of the load band to 0. This is equivalent to "0,0".

For example:

kind: Deployment
apiVersion: apps/v1
metadata:
  # ...
spec:
  template:
    metadata:
      annotations:
        balancer.buoyant.io/load-band: "0.8,10.0"

The annotation can be set on pods or namespaces. The value set on a pod will be used if present. Otherwise, the value on the namespace will be used if present. Otherwise, the cluster default will be used.