Troubleshooting “linkerd-buoyant”

This page provides resolution steps for common problems reported with the linkerd check command.

The “linkerd-buoyant” checks

These checks only run when the Buoyant extension is installed. This check validates that the linkerd-buoyant CLI and Buoyant extension are configured correctly, including the Linkerd lifecycle automation operator and the Buoyant Cloud agent when present.

linkerd-buoyant can determine the latest version

Warning

‼ linkerd-buoyant can determine the latest version
    Get "https://enterprise.buoyant.io/version.json": dial tcp: lookup enterprise.buoyant.io: no such host

Remediation

Ensure you can connect to the Buoyant version check endpoint from the environment where the linkerd CLI is running:

$ curl https://enterprise.buoyant.io/version.json
{"linkerd-buoyant":"v0.27.0"}

linkerd-buoyant CLI is up-to-date

Warning

‼ linkerd-buoyant cli is up-to-date
    CLI version is v0.27.1 but the latest is v0.27.4

Remediation

To update to the latest version of the linkerd-buoyant CLI, run:

curl https://enterprise.buoyant.io/install | sh

linkerd-buoyant Namespace exists

Problem

× linkerd-buoyant Namespace exists
    namespaces "linkerd-buoyant" not found

Remediation

Ensure the linkerd-buoyant namespace exists:

kubectl get ns/linkerd-buoyant

If the namespace does not exist, the linkerd-buoyant installation may be missing or incomplete. Follow the installation guide to fix.

agent-metadata ConfigMap exists

Problem

× agent-metadata ConfigMap exists
    configmaps "agent-metadata" not found

Remediation

Ensure that the config map exists:

$ kubectl -n linkerd-buoyant get cm/agent-metadata
NAME             DATA   AGE
agent-metadata   2      18d

If it does not exist, the linkerd-buoyant installation may be missing or incomplete. Visit the Clusters page in your Buoyant Cloud workspace. Select the “View install instructions” menu item for this cluster and follow the instructions to reinstall.

buoyant-cloud-org-credentials Secret exists

Problem

× buoyant-cloud-org-credentials Secret exists
    secrets "buoyant-cloud-org-credentials" not found

Remediation

Ensure that the secret exists:

$ kubectl -n linkerd-buoyant get secret/buoyant-cloud-org-credentials
NAME                            TYPE     DATA   AGE
buoyant-cloud-org-credentials   Opaque   2      18d

buoyant-cloud-org-credentials Secret has correct labels

Warning

‼ buoyant-cloud-org-credentials Secret has correct labels
    missing app.kubernetes.io/part-of label

Remediation

If you’ve configured your agent to rely on an externally-provided credentials secret, then it’s safe to ignore this warning. If that’s not the case, then the linkerd-buoyant installation may be missing or incomplete. Visit the Clusters page in your Buoyant Cloud workspace. Select the “View install instructions” menu item for this cluster and follow the instructions to reinstall.

buoyant-cloud-agent ClusterRole exists

Problem

× buoyant-cloud-agent ClusterRole exists
    clusterroles.rbac.authorization.k8s.io "buoyant-cloud-agent" not found

Remediation

Ensure that the cluster role exists:

$ kubectl get clusterrole/buoyant-cloud-agent
NAME                  CREATED AT
buoyant-cloud-agent   2024-01-12T02:53:26Z

Also ensure you have permission to create ClusterRoles:

$ kubectl auth can-i create ClusterRoles
yes

buoyant-cloud-agent ClusterRoleBinding exists

Problem

× buoyant-cloud-agent ClusterRoleBinding exists
    clusterrolebindings.rbac.authorization.k8s.io "buoyant-cloud-agent" not found

Remediation

Ensure that the cluster role binding exists:

$ kubectl get clusterrolebinding/buoyant-cloud-agent
NAME                  ROLE                              AGE
buoyant-cloud-agent   ClusterRole/buoyant-cloud-agent   65d

Also ensure you have permission to create ClusterRoleBindings:

$ kubectl auth can-i create ClusterRoleBindings
yes

buoyant-cloud-agent ServiceAccount exists

Problem

× buoyant-cloud-agent ServiceAccount exists
    serviceaccounts "buoyant-cloud-agent" not found

Remediation

Ensure that the service account exists:

$ kubectl -n linkerd-buoyant get serviceaccount/buoyant-cloud-agent
NAME                  SECRETS   AGE
buoyant-cloud-agent   0         83d

Also ensure you have permission to create ServiceAccounts:

$ kubectl -n linkerd-buoyant auth can-i create ServiceAccount
yes

buoyant-cloud-agent Deployment exists

Problem

× buoyant-cloud-agent Deployment exists
    deployments.apps "buoyant-cloud-agent" not found

Remediation

Ensure the buoyant-cloud-agent deployment exists:

$ kubectl -n linkerd-buoyant get deploy/buoyant-cloud-agent
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
buoyant-cloud-agent   1/1     1            1           83d

buoyant-cloud-agent Deployment is running

Problem

× buoyant-cloud-agent Deployment is running
    no running pods for buoyant-cloud-agent Deployment

Remediation

Note, it takes a little bit for pods to be scheduled, images to be pulled and everything to start up. If this is a permanent error, you’ll want to validate the state of the buoyant-cloud-agent Deployment with:

$ kubectl -n linkerd-buoyant get po --selector app=buoyant-cloud-agent
NAME                                   READY   STATUS    RESTARTS   AGE
buoyant-cloud-agent-54b9476cdd-494c6   2/2     Running   0          13h

Check the agent’s logs with:

kubectl -n linkerd-buoyant logs deploy/buoyant-cloud-agent buoyant-cloud-agent

buoyant-cloud-agent Deployment is injected

Warning

‼ buoyant-cloud-agent Deployment is injected
    could not find proxy container for buoyant-cloud-agent-54b9476cdd-494c6 pod

Remediation

Ensure the buoyant-cloud-agent pod is injected; the READY column should show 2/2 containers:

$ kubectl -n linkerd-buoyant get pods --selector app=buoyant-cloud-agent
NAME                                   READY   STATUS    RESTARTS   AGE
buoyant-cloud-agent-54b9476cdd-494c6   2/2     Running   0          34h

Make sure that the linkerd-proxy-injector deployment is working correctly by running linkerd check.

buoyant-cloud-agent Deployment is up-to-date

Warning

‼ buoyant-cloud-agent Deployment is up-to-date
    incorrect app.kubernetes.io/version label: v0.27.1, expected: v0.27.4

Remediation

Check the version with:

$ linkerd-buoyant version
CLI version:   v0.27.4
Agent version: v0.27.4

To update to the latest version, visit the Clusters page in your Buoyant Cloud workspace. Click on the “Upgrade” link for this cluster and follow the instructions.

buoyant-cloud-agent Deployment is running a single pod

Problem

× buoyant-cloud-agent Deployment is running a single pod
    expected 1 buoyant-cloud-agent pod, found 2

Remediation

buoyant-cloud-agent should run as a singleton. Check for other pods:

 $ kubectl get po -A --selector app=buoyant-cloud-agent
NAMESPACE         NAME                                   READY   STATUS    RESTARTS   AGE
linkerd-buoyant   buoyant-cloud-agent-54b9476cdd-494c6   2/2     Running   0          34h

buoyant-cloud-metrics DaemonSet exists

Problem

× buoyant-cloud-metrics DaemonSet exists
    daemonsets.apps "buoyant-cloud-metrics" not found

Remediation

Ensure the buoyant-cloud-metrics daemonset exists:

$ kubectl -n linkerd-buoyant get ds/buoyant-cloud-metrics
NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
buoyant-cloud-metrics   10        10        10      10           10          <none>          83d

buoyant-cloud-metrics DaemonSet is running

Problem

× buoyant-cloud-metrics DaemonSet is running
    no running pods for buoyant-cloud-metrics DaemonSet

Remediation

$ kubectl -n linkerd-buoyant get po --selector app=buoyant-cloud-metrics
NAME                          READY   STATUS    RESTARTS   AGE
buoyant-cloud-metrics-2lkk6   2/2     Running   0          35h
buoyant-cloud-metrics-5ptzn   2/2     Running   0          35h
buoyant-cloud-metrics-6466b   2/2     Running   0          35h

Check the metrics agent’s logs with:

kubectl -n linkerd-buoyant logs ds/buoyant-cloud-metrics buoyant-cloud-metrics

buoyant-cloud-metrics DaemonSet is injected

Warning

‼ buoyant-cloud-metrics DaemonSet is injected
    could not find proxy container for buoyant-cloud-metrics-2lkk6 pod

Remediation

Ensure the buoyant-cloud-metrics pods are injected; the READY column should show 2/2 containers:

$ kubectl -n linkerd-buoyant get po --selector app=buoyant-cloud-metrics
NAME                          READY   STATUS    RESTARTS   AGE
buoyant-cloud-metrics-2lkk6   2/2     Running   0          35h
buoyant-cloud-metrics-5ptzn   2/2     Running   0          35h
buoyant-cloud-metrics-6466b   2/2     Running   0          35h

Make sure that the linkerd-proxy-injector deployment is working correctly by running linkerd check.

buoyant-cloud-metrics DaemonSet is up-to-date

Warning

‼ buoyant-cloud-metrics DaemonSet is up-to-date
    incorrect app.kubernetes.io/version label: v0.27.1, expected: v0.27.4

Remediation

Check the version with:

$ kubectl -n linkerd-buoyant get ds/buoyant-cloud-metrics -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"metrics","app.kubernetes.io/part-of":"linkerd-buoyant","app.kubernetes.io/version":"v0.27.4","helm.sh/chart":"linkerd-buoyant-v0.27.4","linkerd.io/extension":"buoyant"}

To update to the latest version, visit the Clusters page in your Buoyant Cloud workspace. Click on the “Upgrade” link for this cluster and follow the instructions.

linkerd-control-plane-operator Deployment exists

Problem

× linkerd-control-plane-operator Deployment exists
    deployments.apps "linkerd-control-plane-operator" not found

Remediation

Ensure the linkerd-control-plane-operator deployment exists:

$ kubectl -n linkerd-buoyant get deploy/linkerd-control-plane-operator
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
linkerd-control-plane-operator   1/1     1            1           84d

If it does not exist, the linkerd-buoyant installation may be missing or incomplete. Follow the installation guide to fix.

linkerd-control-plane-operator Deployment is running

Problem

× linkerd-control-plane-operator Deployment is running
    no running pods for linkerd-control-plane-operator Deployment

Remediation

Note, it takes a little bit for pods to be scheduled, images to be pulled and everything to start up. If this is a permanent error, you’ll want to validate the state of the linkerd-control-plane-operator Deployment with:

$ kubectl -n linkerd-buoyant get po --selector app.kubernetes.io/component=linkerd-control-plane-operator
NAME                                            READY   STATUS    RESTARTS   AGE
linkerd-control-plane-operator-f594c6f4-n89s8   1/1     Running   0          36h

Check the operator’s logs with:

kubectl -n linkerd-buoyant logs deploy/linkerd-control-plane-operator linkerd-control-plane-operator

linkerd-control-plane-operator Deployment is up-to-date

Warning

‼ linkerd-control-plane-operator Deployment is up-to-date
    incorrect app.kubernetes.io/version label: v0.27.1, expected: v0.27.4

Remediation

Check the version with:

$ kubectl -n linkerd-buoyant get deploy/linkerd-control-plane-operator  -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/component":"linkerd-control-plane-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"linkerd-buoyant","app.kubernetes.io/version":"v0.27.4","helm.sh/chart":"linkerd-buoyant-v0.27.4","linkerd.io/extension":"buoyant"}

To update to the latest version, follow the installation guide.

linkerd-control-plane-operator Deployment is running a single pod

Problem

× linkerd-control-plane-operator Deployment is running a single pod
    expected 1 linkerd-control-plane-operator pod, found 2

Remediation

linkerd-control-plane-operator should run as a singleton. Check for other pods:

$ kubectl get po -A --selector app.kubernetes.io/component=linkerd-control-plane-operator
NAMESPACE         NAME                                            READY   STATUS    RESTARTS   AGE
linkerd-buoyant   linkerd-control-plane-operator-f594c6f4-n89s8   1/1     Running   0          36h

linkerd-control-plane-validator Deployment exists

Problem

× linkerd-control-plane-validator Deployment exists
    deployments.apps "linkerd-control-plane-validator" not found

Remediation

Ensure the linkerd-control-plane-validator deployment exists:

$ kubectl -n linkerd-buoyant get deploy/linkerd-control-plane-validator
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
linkerd-control-plane-validator   1/1     1            1           84d

If it does not exist, the linkerd-buoyant installation may be missing or incomplete. Follow the installation guide to fix.

linkerd-control-plane-validator Deployment is running

Problem

× linkerd-control-plane-validator Deployment is running
    no running pods for linkerd-control-plane-validator Deployment

Remediation

$ kubectl -n linkerd-buoyant get po --selector app.kubernetes.io/component=linkerd-control-plane-validator
NAME                                              READY   STATUS    RESTARTS   AGE
linkerd-control-plane-validator-b977c7654-smj4k   1/1     Running   0          3d2h

Check the validator’s logs with:

kubectl -n linkerd-buoyant logs deploy/linkerd-control-plane-validator linkerd-control-plane-validator

linkerd-control-plane-validator Deployment is up-to-date

Warning

‼ linkerd-control-plane-validator Deployment is up-to-date
    incorrect app.kubernetes.io/version label: v0.27.1, expected: v0.27.4

Remediation

Check the version with:

$ kubectl -n linkerd-buoyant get deploy/linkerd-control-plane-validator  -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/component":"linkerd-control-plane-validator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"linkerd-buoyant","app.kubernetes.io/version":"v0.27.4","helm.sh/chart":"linkerd-buoyant-v0.27.4","linkerd.io/extension":"buoyant"}

To update to the latest version, follow the installation guide.

linkerd-control-plane-validator Deployment is running a single pod

Problem

× linkerd-control-plane-validator Deployment is running a single pod
    expected 1 linkerd-control-plane-validator pod, found 2

Remediation

linkerd-control-plane-validator should run as a singleton. Check for other pods:

$ kubectl get po -A --selector app.kubernetes.io/component=linkerd-control-plane-validator
NAMESPACE         NAME                                              READY   STATUS    RESTARTS   AGE
linkerd-buoyant   linkerd-control-plane-validator-b977c7654-smj4k   1/1     Running   0          3d2h

linkerd-data-plane-operator Deployment exists

Problem

× linkerd-data-plane-operator Deployment exists
    deployments.apps "linkerd-data-plane-operator" not found

Remediation

Ensure the linkerd-data-plane-operator deployment exists:

$ kubectl -n linkerd-buoyant get deploy/linkerd-data-plane-operator
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
linkerd-data-plane-operator   1/1     1            1           84d

If it does not exist, the linkerd-buoyant installation may be missing or incomplete. Follow the installation guide to fix.

linkerd-data-plane-operator Deployment is running

Problem

× linkerd-data-plane-operator Deployment is running
    no running pods for linkerd-data-plane-operator Deployment

Remediation

Note, it takes a little bit for pods to be scheduled, images to be pulled and everything to start up. If this is a permanent error, you’ll want to validate the state of the linkerd-data-plane-operator Deployment with:

$ kubectl -n linkerd-buoyant get po --selector app.kubernetes.io/component=linkerd-data-plane-operator
NAME                                           READY   STATUS    RESTARTS   AGE
linkerd-data-plane-operator-766d8fd469-hrj8p   1/1     Running   0          3d14h

Check the operator’s logs with:

kubectl -n linkerd-buoyant logs deploy/linkerd-data-plane-operator linkerd-data-plane-operator

linkerd-data-plane-operator Deployment is up-to-date

Warning

‼ linkerd-data-plane-operator Deployment is up-to-date
    incorrect app.kubernetes.io/version label: v0.27.1, expected: v0.27.4

Remediation

Check the version with:

$ kubectl -n linkerd-buoyant get deploy/linkerd-data-plane-operator  -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/component":"linkerd-data-plane-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"linkerd-buoyant","app.kubernetes.io/version":"v0.27.4","helm.sh/chart":"linkerd-buoyant-v0.27.4","linkerd.io/extension":"buoyant"}

To update to the latest version, follow the installation guide.

linkerd-data-plane-operator Deployment is running a single pod

Problem

× linkerd-data-plane-operator Deployment is running a single pod
    expected 1 linkerd-data-plane-operator pod, found 2

Remediation

linkerd-data-plane-operator should run as a singleton. Check for other pods:

$ kubectl get po -A --selector app.kubernetes.io/component=linkerd-data-plane-operator
NAMESPACE         NAME                                           READY   STATUS    RESTARTS   AGE
linkerd-buoyant   linkerd-data-plane-operator-766d8fd469-hrj8p   1/1     Running   0          3d14h

controlplanes.linkerd.buoyant.io CRD exists

Problem

× controlplanes.linkerd.buoyant.io CRD exists
    CRD controlplanes.linkerd.buoyant.io does not exist

Remediation

Ensure that the CRD exists:

$ kubectl get crd/controlplanes.linkerd.buoyant.io
NAME                               CREATED AT
controlplanes.linkerd.buoyant.io   2024-01-25T07:38:34Z

Also ensure you have permission to create CRDs:

$ kubectl auth can-i create CustomResourceDefinitions
yes

If it does not exist, the linkerd-buoyant installation may be missing or incomplete. Follow the installation guide to fix.

dataplanes.linkerd.buoyant.io CRD exists

Problem

× dataplanes.linkerd.buoyant.io CRD exists
    CRD dataplanes.linkerd.buoyant.io does not exist

Remediation

Ensure that the CRD exists:

$ kubectl get crd/dataplanes.linkerd.buoyant.io
NAME                            CREATED AT
dataplanes.linkerd.buoyant.io   2024-01-18T23:56:32Z

Also ensure you have permission to create CRDs:

$ kubectl auth can-i create CustomResourceDefinitions
yes

If it does not exist, the linkerd-buoyant installation may be missing or incomplete. Follow the installation guide to fix.