Kubernetes Troubleshooting Decision Tree: Diagnose Pending Pods, CrashLoopBackOff, and Failed Deployments

Overview

Most Kubernetes failures become easier to classify after four questions:

1. Is the workload object healthy, or did the Deployment, StatefulSet, or Job fail to create the expected pods?
2. Were the pods scheduled onto nodes?
3. Did the container start and stay running?
4. Can the running application receive configuration, storage, and network traffic?

Begin with read-only inspection. Set the namespace explicitly and collect evidence before editing a manifest or deleting a pod:

kubectl get deploy,rs,pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --all-containers
kubectl logs <pod-name> -n <namespace> --previous

The --previous option is particularly useful after a container has restarted. If the pod is controlled by a Deployment, also inspect the rollout:

kubectl rollout status deployment/<deployment-name> -n <namespace>
kubectl rollout history deployment/<deployment-name> -n <namespace>
kubectl describe deployment/<deployment-name> -n <namespace>

Command output and available fields can vary by Kubernetes version and resource type. When in doubt, check the command's supported options with kubectl help, and inspect resource fields with kubectl explain pod.spec or kubectl explain deployment.spec. Prefer commands supported by the version of kubectl and the cluster you are operating.

A printable decision tree

Pod not ready or Deployment not progressing?
|
+-- Is the pod Pending?
|   +-- Check Events and scheduler messages
|   +-- Check node capacity, taints, affinity, and PVC binding
|
+-- Is the pod waiting to start?
|   +-- ImagePullBackOff or ErrImagePull: inspect image, registry, and pull secret
|   +-- CreateContainerConfigError: inspect ConfigMaps, Secrets, and references
|
+-- Is the container restarting?
|   +-- CrashLoopBackOff: inspect current and previous logs, exit code, and command
|   +-- Probe failures: inspect probe path, port, timing, and application startup
|
+-- Is the pod Running but not receiving traffic?
|   +-- Check readiness, Service selectors, EndpointSlices, and NetworkPolicy
|
+-- Is the rollout stalled?
    +-- Compare the Deployment, ReplicaSet, and pod templates
    +-- Check unavailable replicas, quotas, scheduling, and recent Events

Checklist by scenario

1. Pods are Pending

A Pending pod has not reached a usable running state. The scheduler's Events usually provide the most direct explanation:

kubectl describe pod <pod-name> -n <namespace>
kubectl get nodes
kubectl describe node <node-name>

Check for insufficient CPU or memory, an unsatisfied node selector or affinity rule, taints without matching tolerations, an unavailable topology requirement, or a PersistentVolumeClaim that has not bound. Compare the pod's requests with allocatable node capacity, not merely the current usage reported by a monitoring tool. A namespace ResourceQuota or a LimitRange can also prevent creation or scheduling.

If storage is involved, inspect the claim and related events:

kubectl get pvc -n <namespace>
kubectl describe pvc <claim-name> -n <namespace>
kubectl get pv

Do not solve a scheduling problem by removing resource requests without understanding the workload's needs. A pod that schedules successfully but is later evicted or throttled may create a different reliability problem.

2. Pods show ImagePullBackOff or ErrImagePull

Inspect the exact image reference and the Events section of describe. Common causes include a misspelled repository or tag, an image that is not available for the node's architecture, registry authentication failure, network access restrictions, or an image-pull secret in the wrong namespace.

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].image}'
kubectl get secrets -n <namespace>
kubectl describe serviceaccount <service-account> -n <namespace>

Confirm that the workload references the intended image, that the pull secret is attached to the relevant ServiceAccount or pod specification, and that the registry policy permits access from the cluster. Avoid relying on a mutable tag when you need reproducible rollouts; use the image versioning approach approved by your platform team.

3. Containers are in CrashLoopBackOff

CrashLoopBackOff describes repeated failed starts with increasing delays; it is a symptom, not the underlying cause. Read both the current and previous container logs, then inspect the termination reason and exit code:

kubectl logs <pod-name> -n <namespace> -c <container-name>
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].state}'

Look for a bad command or argument, missing configuration, failed database or dependency initialization, permission errors, an application that exits after completing its work, or termination caused by resource pressure. Compare the container's command and arguments with the image's expected startup contract. If the process needs time to initialize, distinguish startup behavior from a genuinely unhealthy application before changing liveness settings.

4. Readiness or liveness probes fail

Probe failures can prevent traffic, restart a healthy-but-slow process, or both. Verify the probe type, path, port, scheme, headers, and timing against the application inside the container. A probe port is normally a container port, not automatically a Service port.

kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o yaml

Check whether the endpoint is bound to the expected interface, whether authentication is required, and whether startup time is longer than the configured initial delay and failure window. Use a startup probe when the application has a predictable but lengthy initialization phase, if supported by the cluster version and workload design. Keep readiness separate from liveness: readiness should answer whether the pod can receive traffic, while liveness should answer whether the process needs a restart.

5. The pod is Running but the application is unreachable

Running only confirms that the container process has started. Check readiness, the Service selector, and the Service's endpoints or EndpointSlices:

kubectl get svc <service-name> -n <namespace> -o yaml
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service-name>
kubectl get pods -n <namespace> --show-labels

Verify that the Service selector matches pod labels and that ready addresses exist. Then check the target port, NetworkPolicy rules, DNS name, ingress or gateway configuration, and any cloud load-balancer integration. Test from a permitted client location rather than assuming that a successful request from inside the cluster proves external connectivity.

6. A Deployment rollout fails or stalls

Start with the rollout status and then compare the Deployment, ReplicaSets, and pod template. A new ReplicaSet may exist while its pods cannot schedule, pull an image, pass probes, mount storage, or satisfy a quota:

kubectl rollout status deployment/<deployment-name> -n <namespace>
kubectl get rs -n <namespace>
kubectl describe rs <replicaset-name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Check the rollout strategy, availability limits, termination grace period, and whether old pods can coexist with new pods. If the previous version is known to be safe and the release process permits it, a rollback may reduce impact while the cause is investigated:

kubectl rollout undo deployment/<deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>

Record why the rollback was needed before retrying. Otherwise, the next deployment can reproduce the same failure without improving the manifest or pipeline.

What to double-check before declaring the issue fixed

• Scope: Confirm the namespace, cluster context, and account before running commands that change resources. Use kubectl config current-context and include -n <namespace> explicitly.
• Events: Kubernetes Events are useful evidence but are time-sensitive. Capture relevant output during the incident and correlate it with application and node logs.
• Configuration: Check that referenced ConfigMaps, Secrets, ServiceAccounts, PVCs, and image-pull secrets exist in the same namespace as the workload when namespace scoping applies.
• Resource behavior: Inspect requests, limits, quotas, node capacity, and eviction signals together. Changing one value can move the failure from scheduling to runtime.
• Ownership: Identify whether the object is managed by a deployment controller, Helm, an operator, or a GitOps system. A manual fix may be overwritten by reconciliation.
• Rollout confirmation: Wait for the intended number of replicas to become ready, verify the new pod template, and test the application through its normal Service or ingress path.
• Persistence: Update the source manifest, policy, chart, or platform template that should contain the fix. A direct pod edit is rarely a durable change.

For a repeatable operational process, link this checklist to the team's incident response runbook and CI/CD review steps. Related guidance on incident response runbooks and CI/CD pipeline bottlenecks can help connect cluster symptoms to the deployment process that introduced them.

Common mistakes

• Deleting the pod first: A restart can hide the original Events and may not change a bad image, configuration, probe, or scheduling rule.
• Reading only application logs: Scheduler, kubelet, admission, and controller messages often explain failures that never reach the application.
• Treating CrashLoopBackOff as a Kubernetes bug: Inspect exit codes, previous logs, commands, dependencies, and resource termination reasons before changing cluster settings.
• Changing probes to make dashboards green: A relaxed probe can conceal an unhealthy process or send traffic before the application is ready.
• Using a broad command without a namespace: The output may come from the wrong environment, especially when multiple clusters and contexts are in use.
• Applying an emergency edit without updating source control: Reconciliation or the next CI/CD deployment can restore the failing configuration.
• Assuming Running means available: Always check readiness and the Service's actual ready endpoints before closing the incident.

When to revisit this checklist

Review this guide before seasonal planning cycles, major traffic changes, cluster upgrades, and any redesign of deployment or networking workflows. Revisit it whenever your Kubernetes distribution, kubectl version, ingress or gateway implementation, storage driver, admission policy, observability stack, or deployment controller changes. Command output and supported fields can differ across versions, so validate examples in a non-production context before an emergency.

After a real incident, turn the diagnosis into a short verification test: reproduce the relevant failure safely, deploy the corrected configuration, confirm scheduling and readiness, exercise the Service path, and verify that a future rollout preserves the fix. Add the final cause, useful commands, and ownership location to the team's runbook. That small maintenance step turns Kubernetes troubleshooting from ad hoc command hunting into a reusable platform reliability practice.