# enterprise-2.19.0

Linkerd 2.19 is a new major release that adds support for Windows containers and
post-quantum cryptography, improves Linkerd's supply chain security, and
optionally supports FIPS 140-3 validated encryption.

See our
[Linkerd 2.19 announcement blog post](https://buoyant.io/blog/linkerd-enterprise-2-19-windows-service-mesh-post-quantum-cryptography-supply-chain-security-fips-140-3-and-a-new-on-cluster-dashboard)
for more details.

> [!WARNING]
> This is a new
> [major version](/buoyant-enterprise-linkerd/latest/overview/versions/#stable-release-versions)
> of Linkerd. There are [breaking changes](/release-notes/buoyant-enterprise-linkerd/enterprise-2.19.0/#breaking-changes) in this release.
> Please read these release notes carefully before upgrading.

> [!WARNING]
> FIPS users who are upgrading from 2.18 to 2.19 must
> follow a specific sequence to preserve zero-downtime upgrades. You must first
> upgrade to [2.18.7](/release-notes/buoyant-enterprise-linkerd/enterprise-2.18.7/) or later, and then to
> [2.19.4](/release-notes/buoyant-enterprise-linkerd/enterprise-2.19.4/) or later.

## Who should upgrade?

This is a feature release. We recommend upgrading to Linkerd 2.19 for customers
who want to make use of the new features introduced.

## Supported Kubernetes and Gateway API versions

BEL 2.19 supports Kubernetes versions 1.29 through 1.35, and Gateway API
versions v1.1.1 through v1.2.1.

## Control plane operator requirements

To upgrade with [BEL’s lifecycle automation operator](/buoyant-enterprise-linkerd/latest/features/operator/), you will need Buoyant Extension [v0.37.2](/release-notes/buoyant-extension/v0.37.2/) or later.

## Upgrade guidance

This release includes several breaking changes, as well as several non-breaking
changes that may require action on your part. Please read these release notes
carefully.

## Breaking changes

### Admin port names have changed to be unique per component

The Linkerd control plane components expose "admin" ports that serve liveness
and readiness probes, as well as metrics traffic. In previous releases, these
ports were named `admin-http` regardless of component. However, this duplication
can be ambiguous in certain circumstances, and Kubernetes 1.33 and beyond warn
about duplicate container port names.

In Linkerd 2.19, the names are now unique per component: `dest-admin`,
`policy-admin`, `ident-admin`, etc.

**What you need to do:** If you reference Linkerd control plane port names in
Services, probes, or monitoring rules (search for “admin-http”), update these
names after upgrading. See the Control plane port name reference for details on
the new names.

### Linkerd now blocks connections to ports of ClusterIP Services not defined in the Service resource

For ClusterIP Services, Linkerd will now block connections to ports that are not
defined in the Service spec. This behavior now matches the behavior of
kube-proxy, and avoids spurious false alarms with port scanners.

**What you need to do:** Prior to upgrading to Linkerd 2.19, audit your
ClusterIP services, and ensure that all ports for ClusterIP services are defined
in the Service manifest.

### The default iptables mode has been changed to nftables

Some major Kubernetes platforms have removed support for "legacy" iptables (for
example, Kubernetes 1.33 on EKS). To reflect this, as of Linkerd 2.19, the
default iptables mode used by Linkerd is now nftables. Customers who require
legacy iptables support may use the `proxyInit.iptablesMode` configuration (and,
for the Linkerd CNI plugin, the `iptablesMode=legacy` flag) to use this mode.

**What you need to do:** Determine whether your environment supports nftables.
If it **does not**, ensure the configuration settings above are used to force
legacy iptables usage.

### ARMv7 environments are no longer supported

Linkerd 2.19 no longer ships binaries for ARMv7 architectures, which have been
largely superseded.

**What you need to do:** No action is required.

## Notable changes

### Decoupling of Gateway API ownership

Continuing the transition we started in 2.18, in 2.19 Linkerd will no longer
install or manage the Gateway API CRDs. This change improves reliability by
isolating ownership of shared cluster resources and reduces upgrade risk by
preventing Linkerd from modifying CRDs that other components may depend on. You
now control when and how Gateway API is installed or upgraded, independent of
Linkerd releases. See our
[Gateway API doc](/buoyant-enterprise-linkerd/2.19/guides/managing-gateway-api/).

Note that Linkerd does not require the Gateway API types to run, but several
Linkerd features require Gateway API types such as `HTTPRoute` and `GRPCRoute`
before they can be used.

**What you need to do:**

1. Verify that the installed version of the Gateway API is compatible with
   Linkerd 2.19. (See our
   [Gateway API doc](/buoyant-enterprise-linkerd/2.19/guides/managing-gateway-api/)
   for how to do this.)
2. Ensure you have a process for keeping the Gateway API CRDs up-to-date in your
   clusters. For managed clusters, this may be done by your cluster provider.

### Deprecation of hotpatch releases

In Linkerd 2.19, we've updated the build pipeline significantly (see _New supply
chain security_ below). One such change is that hotpatch releases are no longer
required. The original motivation for hotpatches was to address issues in
underlying base container images (i.e. not in executables produced by Buoyant)
that are detected by container image security scanners. Hotpatches allowed us to
create new container images with updated base image files containing
remediations to CVEs. Many of these CVEs were present in executables or
libraries that are not in the execution path of any part of Linkerd, but were
flagged by security scanners.

Starting in 2.19, we've made improvements to our build pipeline infrastructure
to produce container images that contain only the minimum necessary files to
support Linkerd executables. This should remove the need for hotpatches, and we
will address CVEs through point releases instead.

Note that this does not affect our SLAs around CVE remediation.

**What you need to do:** No action required.

### The Linkerd-Jaeger extension is now deprecated

The past few Linkerd releases have been modernizing our distributed tracing
infrastructure. In Linkerd 2.18, we moved the default tracing protocol from
OpenCensus to OpenTelemetry. In this release, we have deprecated the
Linkerd-jaeger extension, which is no longer necessary. If you are running this
extension, it will continue to work in 2.19 but may not work in future releases,
and you should migrate. See the
[Linkerd-Jaeger migration doc](https://linkerd.io/2.19/tasks/jaeger-extension-migration/).

Note that OpenTelemetry tracing can be enabled in Linkerd without the use of
this extension. See our
[Guide to distributed tracing with Linkerd](https://linkerd.io/2.19/tasks/distributed-tracing/).

**What you need to do:** If you are using the Linkerd-jaeger extension, after
installing Linkerd 2.19, please following the guide above to remove it.

### Tracing Semantic conventions

When tracing is enabled in the proxy, the proxy now follows the
[OpenTelemetry HTTP semantic conventions](https://opentelemetry.io/docs/specs/semconv/http/http-spans/)
for labels and attributes populated in each emitted span.

**What you need to do:** No action required.

### Native sidecar support graduates to beta

Linkerd has supported native sidecars since version 2.15. Reflecting the
widespread availability and adoption of native sidecars, support has been
officially graduated to beta. The
`config.alpha.linkerd.io/proxy-enable-native-sidecar` annotation is now
deprecated, and should be replaced with
`config.beta.linkerd.io/proxy-enable-native-sidecar`.

**What you need to do:** If you are using native sidecars, after installing
Linkerd 2.19, update any references to
`config.alpha.linkerd.io/proxy-enable-native-sidecar` to
`config.beta.linkerd.io/proxy-enable-native-sidecar`.

### The proxy inbound connection pool is now limited to 10K connections by default

To prevent unbounded growth and reduce the risk of memory/resource exhaustion
under extreme concurrency, the proxy now enforces a default cap of 10,000
entries in the inbound connection pool. In very high concurrency scenarios,
users should observe more predictable memory usage. (This change was first
shipped in BEL 2.18.3.)

Users with extreme concurrency requirements may raise this limit by setting
`LINKERD2_PROXY_MAX_IDLE_CONNS_PER_ENDPOINT` (for inbound) and
`LINKERD2_PROXY_OUTBOUND_MAX_IDLE_CONNS_PER_ENDPOINT` (for outbound)
configuration variables in the proxy.

**What you need to do**: Prior to installing Linkerd 2.19, evaluate your running
application for individual meshed pods that handle more than 10,000 connections
each. For any such pods, update the configuration described above to raise the
limit appropriately.

## New features

### Windows container support

This release adds support for running Linkerd's dataplane microproxies on
Kubernetes Windows nodes, allowing containerized Windows applications on those
nodes to join the mesh. Containerized Windows applications marked for injection
will be automatically meshed by Linkerd, and all TCP traffic to and from these
applications will be transparently handled by the Linkerd dataplane microproxy
in that pod. See the
[Windows doc](/buoyant-enterprise-linkerd/2.19/features/windows-containers/).

### Post-quantum cryptography and FIPS 140-3 support

Linkerd's internal TLS infrastructure has been overhauled to prepare for a
potential post-quantum future. We've updated the core cryptographic module in
the proxy from `ring` to `aws-lc`, and added support for the AES_128_GCM
ciphersuite and the post-quantum ML-KEM-768 key exchange algorithm, which are
used by default for all communication between meshed pods. The TLS cipher, key
exchange, and signature algorithms are now exported as part of the standard
metrics suite. Finally, in FIPS builds, Buoyant Enterprise for Linkerd now uses
FIPS 140-3 validated cryptographic modules with AES_256_GCM. No action is
required to take advantage of this upgrade.

### New supply chain security

While we've been publishing full SBOMs for BEL container images since Linkerd
2.15, Linkerd 2.19 introduces a new, state-of-the-art container release pipeline
with native OCI 1.1 referrer support and end-to-end image signing.

In Linkerd 2.19, all stable BEL images now ship with SBOM and SLSA v0.2
provenance attestations, published as first-class OCI 1.1 referrers and signed
by digest. This means that BEL supply chain metadata is now linked to its
corresponding images directly within the registry and is cryptographically
signed, allowing registry-native verification of both image authenticity and
integrity.

You can use this supply-chain information to verify provenance and authenticity
of BEL images in a fully programmatic way at deploy time, using tools such as
`oras`, `cosign` and `skopeo`. See the
[Verifying Signed Artifacts](/buoyant-enterprise-linkerd/2.19/guides/verifying-signed-artifacts/)
doc.

### Linkerd Dashboard

Alongside this release, we've also shipped a new on-cluster Linkerd web
dashboard designed to replace the ailing `linkerd-viz` web UI. This new Linkerd
dashboard is under rapid iteration, but is stable and ready for usage. This new
Linkerd dashboard includes the majority of the functionality of the older
`linkerd-viz` dashboard, plus additional views of live TLS traffic that allow
you to audit real-time and historical use of TLS, including (when enabled) use
of FIPS validated cryptography. This includes:

- Easy identification of plaintext traffic
- Filtering by source, destination, port, and traffic type
- Identification of FIPS 140-3 and 140-2 cryptographic modules
- And more!

The dashboard is compatible with Buoyant Enterprise for Linkerd 2.17. See the
[Dashboard doc](/linkerd-dashboard/getting-started/).

## Fine-grained Changelog

### Control plane

- **Breaking change:** Linkerd will now block connections to ports of a
  ClusterIP Service which are not defined in the Service spec. This now matches
  the behavior of kube-proxy.
  ([fix(policy-controller)!: Disallow requests to undefined service ports linkerd/linkerd2#14149](https://github.com/linkerd/linkerd2/pull/14149))
- **Breaking change:** Changed port names in the Linkerd control plane to be
  unique within a Pod
  ([refactor(charts)!: disambiguate container port names for Kubernetes 1.33 linkerd/linkerd2#14111](https://github.com/linkerd/linkerd2/pull/14111))
- Fixed discovery staleness when targeting the linkerd-admin port in
  native-sidecar mode
  ([fix(destination): discovery staleness when targeting the linkerd-admin port in native-sidecar mode linkerd/linkerd2#14468](https://github.com/linkerd/linkerd2/pull/14468))
- Fixed a potential panic in the control plane when processing discovery
  requests with invalid hostnames
  ([fix(destination): Replace With with GetMetricWith linkerd/linkerd2#14428](https://github.com/linkerd/linkerd2/pull/14428))
- Consolidated the policy-control and destination controller into a single
  docker image
  ([build(controller)!: eliminate policy-controller image linkerd/linkerd2#14348](https://github.com/linkerd/linkerd2/pull/14348))
- Fixed an issue where Prometheus wasn't able to scrape metrics on proxies
  injected as native sidecar containers in workloads with a restrictive inbound
  policy
  ([fix(policy-controller): properly index named ports in native sidecar containers linkerd/linkerd2#14144](https://github.com/linkerd/linkerd2/pull/14144))
- Added httproutes.gateway.networking.k8s.io as a valid targetRef kind for
  AuthorizationPolicies
  ([fix(policy): allow gateway api httproutes as authorizationpolicy target type linkerd/linkerd2#13962](https://github.com/linkerd/linkerd2/pull/13962))
- Added support for windows workloads that work in CNI only mode and use a
  Windows-specific security context
- Introduced explicit treatment of workload as Windows specific for the purpose
  of injection

### Tracing

- Allow tracing of the Linkerd proxy even when the linkerd-jaeger extension is
  not installed
  ([feat(tracing): Add proxy tracing configuration to control plane helm chart linkerd/linkerd2#13994](https://github.com/linkerd/linkerd2/pull/13994))
- Added pod IP as a attribute in traces
  ([Add pod ip to via downward API to Trace Attributes linkerd/linkerd2#13981](https://github.com/linkerd/linkerd2/pull/13981))
- **Breaking change:** The linkerd-jaeger extension has been removed.
  ([feat(jaeger)!: Remove linkerd-jaeger extension linkerd/linkerd2#14558](https://github.com/linkerd/linkerd2/pull/14558))
- **Breaking change:** controlPlaneTracing and controlPlaneTracingNamespace has
  been removed from the control plane values
  ([feat(tracing)!: Improve control plane tracing configuration linkerd/linkerd2#14557](https://github.com/linkerd/linkerd2/pull/14557))

### Multicluster

- Fixed an issue where timeouts during linkerd multicluster check were being
  misreported
  ([fix(multicluster): Do not ignore timeout and gateway metrics errors during multicluster check linkerd/linkerd2#14418](https://github.com/linkerd/linkerd2/pull/14418))

### Helm Charts

- Corrected the value of the linkerd.io/created-by to accurately reflect if the
  resource was created by Helm or the Linkerd CLI
  ([fix(helm): Fix cli version string in extensions linkerd/linkerd2#14309](https://github.com/linkerd/linkerd2/pull/14309))
- Removed the deprecated preserveUnknownFields field from CRD manifests
  ([fix(helm): Remove preserveUnknownFields from service profile template in linkerd-crds chart linkerd/linkerd2#14179](https://github.com/linkerd/linkerd2/pull/14179))
- Fixed an issue where the caBundle value was being ignored when externalSecret
  is set to false
  ([fix(helm): Allow setting caBundle with caPEM and keyPEM linkerd/linkerd2#14109](https://github.com/linkerd/linkerd2/pull/14109))

### Proxy

- **Breaking change:** Dropped support for arm/v7
  ([feat!(ci): Remove arm/v7 support linkerd/linkerd2-proxy#4037](https://github.com/linkerd/linkerd2-proxy/pull/4037))
- Added a inbound_requests_total metric
  ([feat(app/inbound): count inbound requests linkerd/linkerd2-proxy#4127](https://github.com/linkerd/linkerd2-proxy/pull/4127))
- Added inbound_http_response_frame_size histogram metrics to for inbound HTTP
  body frame size
  ([feat(app/inbound): response body frame size metrics linkerd/linkerd2-proxy#4165](https://github.com/linkerd/linkerd2-proxy/pull/4165))
- Removed an unintended HTTP/1 header read timeout
  ([fix(proxy/http): remove http/1 header read timeout linkerd/linkerd2-proxy#3985](https://github.com/linkerd/linkerd2-proxy/pull/3985))
- The proxy now uses the AES_256_GCM cipher suite and ML-KEM-768 key exchange
  for mesh TLS
