# Linkerd resource request and limits

This guide describes our recommendations for setting Kubernetes resource limits
and requests for Buoyant Enterprise for Linkerd.

This is a general guide, intended to provide a safe starting point for systems
that handle moderate load. Systems expected to handle high load (or small load)
may wish to alter these values. Additionally, as you develop an operational
history with your applications, you may wish to tune these values for more
efficient allocation of resources and to achieve specific QoS goals.

See _Real-world Examples_ below for concrete examples of Linkerd's resource
consumption in practice.

## General strategy and further tuning

The effects of resource requests and limits can be nuanced in Kubernetes, and
there are many factors to consider including desired QoS levels and application
behaviors. In this guide, we attempt to provide a safe framework that minimizes
disruptions and allows Linkerd to scale in most cases. These values are a
starting point. Once an operational history has been established with Linkerd,
there are many ways to can improve upon them, including:

- You can _lower_ the requests and limits to capture the empirical resource
  consumption, allowing you to more efficiently schedule Linkerd workloads.

- You can set CPU limits when unspecified below, allowing your pods to achieve
  Guaranteed QoS rather than Burstable QoS.

- You can use automated systems to continually tune these values.

In short: this guide aims to provide a safe basic setup, but there is plenty of
opportunity for tailoring.

## Control plane

We recommend the following resource configuration for the Linkerd control plane
as a safe starting point for a system designed for moderate load. These values
are built from our empirical observations of the "moderate traffic" system
described below, plus plenty of headroom.

| Component                     | CPU request | CPU limit | Memory request | Memory limit |
| ----------------------------- | ----------- | --------- | -------------- | ------------ |
| destination                   | 100m        | 100m      | 500 MB         | 500 MB       |
| sp-validator                  | 10m         | 10m       | 100 MB         | 100 MB       |
| policy                        | 10m         | 10m       | 100 MB         | 100 MB       |
| identity                      | 10m         | 10m       | 100 MB         | 100 MB       |
| proxy-injector                | 20m         | 20m       | 200 MB         | 200 MB       |
| license                       | 20m         | 20m       | 200 MB         | 200 MB       |
| autoregistration (if enabled) | 20m         | 20m       | 100 MB         | 100 MB       |

See [Real-World Examples](/buoyant-enterprise-linkerd/latest/guides/resource-request-and-limits/#real-world-examples) below for a demonstration of how
memory and CPU consumption can grow at scale.

## Data plane proxy

In contrast to the control plane, which is in many ways a standard Kubernetes
application, the resource consumption by data plane proxies is a function of the
application it sits in front of. Thus, resource usage in practice can vary
wildly across proxies within the same cluster.

As a starting point, we recommend the following resource configuration. For new
Linkerd deployments, we suggest starting with the _Flexible_ profile until an
operational history has been established. If you know your application will
handle particularly small or large amounts of traffic, you may wish to start
with the lightweight or heavyweight configurations instead.

| Profile            | CPU request | CPU limit | Proxy CPU limit | Memory request | Memory limit |
| ------------------ | ----------- | --------- | --------------- | -------------- | ------------ |
| Flexible / generic | 100m        | 1000m     | unset           | 20 MB          | 250 MB       |
| Lightweight        | 10m         | 10m       | unset           | 20 MB          | 20 MB        |
| Heavyweight        | 1000m       | 2000m     | 2000m           | 1000 MB        | 1000 MB      |

Important notes for further tuning:

- Linkerd has its own _proxy-cpu-limit_ configuration value. When unset (the
  default), the proxy will run a single worker thread and will be unable to
  scale beyond one core. Thus, when setting either a CPU request or limit above
  1000m, you should also set the proxy-cpu-limit configuration to match the
  higher value so that the proxy is able to make use of multiple CPU cores.
- The Linkerd proxy-init container uses the same requests and limits as the
  proxy (because resources are shared) and thus does not need explicit
  configuration.
- If you want to achieve Guaranteed QoS for application pods, you must set equal
  CPU and memory limits and requests for all containers in the pod, i.e. both
  proxy and application containers.

For very high-scale proxies that need to consume multiple cores, there is a
relationship between worker threads, CPU cores, and the way that Kubernetes
handles concurrency. We suggest reviewing [Configuring Proxy
Concurrency](https://linkerd.io/2.20/tasks/configuring-proxy-concurrency/) for complex cases.

## Configuring these limits

These limits can be configured in Linkerd by setting the corresponding values
during control plane installation (or upgrade) time.

This includes control plane configuration: (all container-level resources)

- destinationResources
- spValidatorResources
- policyController.resources
- identityResources
- proxyInjectorResources
- heartbeatResources
- licenseResources
- autoregistrationResources (if enabled)

And control plane configuration of the proxy resources which run in the control
plane: (pod-level)

- destinationProxyResources
- identityProxyResources
- proxyInjectorProxyResources
- enterpriseProxyResources
- autoregistrationProxyResources (if enabled)

And finally, global proxy configuration (which are overridden by the section
above, for proxies that run in the control plane specifically):

- proxy.resources

Finally, the global proxy configuration above may be overridden by namespace and
pod-level annotations, to allow per-workload tuning of the data plane:

- config.linkerd.io/proxy-cpu-limit
- config.linkerd.io/proxy-cpu-request
- config.linkerd.io/proxy-memory-limit
- config.linkerd.io/proxy-memory-request

See the
[Linkerd control plane configuration reference](/buoyant-enterprise-linkerd/latest/reference/bel-configuration/#install=install-helm&tab=linkerd-enterprise-control-plane)
for details on these variables and their CLI equivalents.

## Real-world examples

Below are some real-world examples of control plane usage in production in
recent versions of Buoyant Enterprise for Linkerd. The numbers in these tables
represent the _maximum_ measured CPU and memory usage for a single pod in that
service.

### Light traffic system (12 RPS)

**Control plane:**

| Deployment             | Container name | CPU (mC) | Memory |
| ---------------------- | -------------- | -------- | ------ |
| linkerd-destination    | destination    | 1        | 100 MB |
| linkerd-proxy-injector | proxy-injector | 1        | 72 MB  |
| linkerd-identity       | identity       | 1        | 36 MB  |
| linkerd-destination    | sp-validator   | 1        | 31 MB  |
| linkerd-destination    | policy         | 1        | 28 MB  |

**Data plane**: The most memory consumed by a single proxy in this environment
was 50 MB and the most CPU consumed was 35 millicores, with the majority of
proxies consuming significantly fewer resources.

### Moderate traffic system (200 RPS)

**Control plane:**

| Deployment               | Container name   | CPU (mC) | Memory |
| ------------------------ | ---------------- | -------- | ------ |
| linkerd-destination      | destination      | 24       | 205 MB |
| linkerd-destination      | policy           | 4        | 23 MB  |
| linkerd-destination      | sp-validator     | 2        | 41 MB  |
| linkerd-proxy-injector   | proxy-injector   | 3        | 72 MB  |
| linkerd-enterprise       | license          | 1        | 72 MB  |
| linkerd-identity         | identity         | 1        | 42 MB  |
| linkerd-autoregistration | autoregistration | 1        | 40 Mb  |

**Data plane**: The most memory consumed by a single proxy in this environment
was 180 MB and the most CPU consumed was 250 millicores, with the majority of
proxies consuming significantly fewer resources.

### High-traffic system (3,000 RPS)

**Control plane:**

| Deployment             | Container name | CPU (mC) | Memory |
| ---------------------- | -------------- | -------- | ------ |
| linkerd-destination    | destination    | 250      | 1.6 GB |
| linkerd-destination    | policy         | 7        | 166 Mb |
| linkerd-destination    | sp-validator   | 2        | 45 Mb  |
| linkerd-proxy-injector | proxy-injector | 15       | 291 Mb |
| linkerd-identity       | identity       | 3        | 56 Mb  |

**Data plane**: the most memory consumed by a single proxy in this environment
was 700 MB and the most CPU consumed was 650 millicores, with the majority of
proxies consuming significantly fewer resources.