Kubernetes CrashLoopBackOff Troubleshooting: Logs, Probes, Configuration, and Recovery Steps
KubernetesTroubleshootingContainersApplication ReliabilityCloud Operations

Kubernetes CrashLoopBackOff Troubleshooting: Logs, Probes, Configuration, and Recovery Steps

CChallenges.pro Editorial Team
2026-11-17
8 min read

A practical Kubernetes CrashLoopBackOff workflow for collecting evidence, checking probes and configuration, and choosing the right recovery.

When a Kubernetes workload enters CrashLoopBackOff, the status describes a restart pattern rather than the underlying failure. This guide provides a repeatable evidence-first workflow: inspect the pod and its previous logs, test probes and configuration, compare resource and dependency signals, then choose between rollback, temporary debugging, and a permanent fix. It also defines the checkpoints worth tracking so recurring container restarts become easier to diagnose over time.

Overview

CrashLoopBackOff means that a container has terminated repeatedly and Kubernetes is delaying the next restart. The delay is intended to avoid restarting a failing container continuously. It does not identify whether the application exited, a probe failed, the image command was invalid, a required setting was missing, or the node terminated the process.

Start by treating the status as an observation. Preserve evidence before editing the Deployment, deleting the pod, or repeatedly restarting it. A replacement pod can remove useful logs and events, while a rushed probe change can hide an application failure rather than resolve it.

The first questions are:

  • Which container is restarting, and what was its last termination reason and exit code?
  • Did the application start and then exit, or fail before its process initialized?
  • Are liveness, readiness, or startup probes involved?
  • Did configuration, permissions, resource availability, image content, or a dependency change?
  • Is the problem isolated to one pod, one node, one version, or the whole workload?

Use the workload's namespace in every command. If you do not know it, begin with kubectl get pods -A, then set a shell variable such as NS=production and POD=orders-7d8f6c9f5b-k2m4q for consistent inspection.

What to track

A useful Kubernetes troubleshooting record contains both the current symptom and the history around it. The following reference table connects common evidence to likely causes and an appropriate next check.

Symptom or evidenceLikely causeNext check
Events report liveness or startup probe failuresIncorrect path, port, scheme, timeout, initial delay, or application startup timeRun the probe manually from a suitable debugging context; compare probe settings with the process's actual listener and startup behavior.
Logs mention a missing environment variable, file, or secretIncorrect ConfigMap or Secret reference, wrong key, missing mount, or a configuration changeInspect the rendered pod specification and confirm references exist without printing secret values.
Termination shows exit code 127, 126, or an executable errorInvalid image entrypoint, missing binary, incorrect command, or permissions on the executableCompare command and args with the image contract and inspect the image or run an interactive diagnostic container.
The process starts, then exits with a normal application errorApplication validation failure, migration issue, bad argument, or dependency initialization failureRead current and previous logs, capture the exit code, and compare the failing revision with the last known-good revision.
Events show OOMKilled or memory-related terminationContainer memory usage exceeded its limit, or the node is under memory pressureReview limits, recent workload changes, memory metrics, and application allocation behavior before changing limits.
Permission-denied errors appear during startupFilesystem ownership, security context, read-only mount, or an incorrectly mounted volumeInspect volume mounts, user and group settings, and the image's expected filesystem permissions.
Connection or name-resolution failures appearDependency unavailable, incorrect Service or port, credentials failure, NetworkPolicy, or DNS problemTest the dependency from the same namespace and network context; use the Kubernetes DNS troubleshooting guide when name resolution is suspect.

Collect the first evidence set

Run these commands before making a change:

kubectl get pod "$POD" -n "$NS" -o wide
kubectl describe pod "$POD" -n "$NS"
kubectl get pod "$POD" -n "$NS" -o yaml
kubectl get events -n "$NS" --field-selector involvedObject.name="$POD" --sort-by=.lastTimestamp

Then inspect logs for the failing container. The --previous option is especially important because it retrieves output from the container instance that just terminated.

kubectl logs "$POD" -n "$NS" -c app --tail=200
kubectl logs "$POD" -n "$NS" -c app --previous --tail=200

Replace app with the actual container name. For multiple containers, inspect each one, including sidecars. In the pod description and YAML, focus on state, lastState, reason, exitCode, startedAt, finishedAt, probe definitions, commands, arguments, mounts, environment references, resource requests and limits, and the pod's node.

Follow a command-by-command diagnostic workflow

  1. Confirm the target. Check that you are inspecting the expected namespace, pod, container, image, and ReplicaSet. A similarly named pod from another rollout can lead to a false diagnosis.
  2. Read the termination record. Use kubectl describe pod and the YAML output to distinguish an application exit, probe-related termination, OOMKilled, eviction, or an image and runtime error.
  3. Read both log streams. Current logs may contain only startup output; previous logs usually contain the actual failure. Add --timestamps when timing matters.
  4. Read events in time order. Events can reveal failed mounts, image pulls, probe failures, scheduling issues, and node-level conditions that are absent from application logs.
  5. Compare the rendered specification. Inspect what reached the pod, not only the source manifest. Check image digest or tag, command, arguments, ports, probes, environment sources, volume mounts, security context, and resources.
  6. Check the owning controller. Use kubectl get pod "$POD" -n "$NS" -o jsonpath='{.metadata.ownerReferences}' and inspect the Deployment, StatefulSet, or Job. A direct pod edit is not a durable fix when a controller recreates it.
  7. Test dependencies and node context. Check Services, endpoints, DNS, policies, mounted volumes, and node conditions. If failures follow one node, compare that node with healthy nodes rather than changing application settings first.

If the container dies too quickly for interactive access, use an ephemeral container where your cluster policy and Kubernetes version support it:

kubectl debug -it "$POD" -n "$NS" --image=busybox:stable --target=app -- sh

An ephemeral container is a diagnostic aid, not a replacement for the failing container. It may not share exactly the same filesystem, tools, user, or runtime behavior. Use it to inspect process visibility, files, DNS, and connectivity, and document the image and target container used. Avoid placing credentials in commands or diagnostic output.

Cadence and checkpoints

CrashLoopBackOff troubleshooting is reactive, but the evidence behind it can support a recurring reliability review. Track the workload, container, image revision, restart count, last termination reason, exit code, probe failures, resource signals, dependency errors, node, and the change that preceded the incident. Record whether the final action was a rollback, configuration correction, resource adjustment, probe correction, image fix, or application change.

Use the following cadence:

  • During an incident: capture logs, events, pod YAML, controller revision, and timestamps before changing the workload. Note every temporary change and its expected expiration.
  • After recovery: confirm that restarts remain stable across more than one reconciliation cycle and, where relevant, across a new pod or rollout. Attach the root-cause evidence to the incident or deployment record.
  • Monthly or quarterly: review recurring restart reasons by service and revision. Look for repeated probe failures, configuration omissions, dependency timeouts, memory terminations, and failures concentrated on a node pool or release process.
  • Before a planned upgrade: verify probe behavior, container runtime messages, resource settings, security context assumptions, and managed-cluster differences. Pair this review with the Kubernetes upgrade checklist.

For broader operational context, connect restart patterns with service-level objectives and alert history. The SLO burn-rate alert guide can help teams decide when a restart symptom represents user-impacting reliability risk rather than an isolated deployment issue.

How to interpret changes

Interpret the timeline, not just the latest status. A CrashLoopBackOff that begins immediately after an image change points toward entrypoints, dependencies, permissions, or application initialization. A pattern that begins after a probe edit points toward endpoint, port, timing, or protocol mismatches. Failures that follow a configuration rollout suggest a missing key, altered format, stale secret, or incorrect mount.

Separate correlation from proof by comparing a failing pod with a healthy pod running the previous revision. Useful comparison points include the image digest, rendered environment references, mounted paths, security context, resource limits, node, and probe definitions. If reverting one controlled change stops the restarts, retain the evidence and reproduce the fix in the source manifest or application code rather than leaving an emergency edit in place.

Do not increase a memory limit, lengthen a probe delay, or remove a liveness check solely to make the status disappear. Those changes can be valid, but each should answer a specific finding. A startup probe may protect a slow-starting application from premature liveness checks; it should not conceal an application that never becomes healthy. Similarly, more memory may address a measured capacity problem, but it will not fix a leak, an invalid configuration, or a dependency timeout.

Classify the response into three categories:

  • Rollback: use when a recent revision is clearly responsible and service restoration is the priority. Preserve the failed revision and evidence for follow-up.
  • Temporary debugging change: use only with an owner, scope, and expiration. Examples include increasing log verbosity, pausing a rollout, or temporarily adjusting a startup probe to collect evidence.
  • Permanent fix: update the image, manifest, probe, configuration validation, resource policy, dependency handling, or runbook, then verify it through the normal delivery path.

When to revisit

Revisit this guide whenever a workload begins restarting repeatedly, a probe or container command changes, a base image or runtime changes, configuration is reorganized, a dependency or NetworkPolicy changes, or a cluster upgrade alters node behavior. It is also worth returning during onboarding for unfamiliar services and during monthly or quarterly application health reviews.

Use this recovery checklist at the end of each incident:

  1. Identify the exact namespace, pod, container, controller, revision, and node.
  2. Save describe output, pod YAML, current logs, previous logs, and ordered events.
  3. Record the last termination reason, exit code, timestamps, and restart pattern.
  4. Check probes, commands, arguments, image content, configuration references, mounts, permissions, resources, and dependencies.
  5. Choose deliberately between rollback, a time-boxed debugging change, and a permanent fix.
  6. Verify recovery after reconciliation and a fresh pod, not only after one manual restart.
  7. Update the runbook, alert context, deployment checks, or application behavior so the next diagnosis starts with better evidence.

Keep the command workflow close to the service's incident response runbook, and review it when Kubernetes versions, managed-cluster behavior, container images, or application startup assumptions change. A consistent evidence trail turns CrashLoopBackOff from a vague pod status into a manageable operational signal.

Related Topics

#Kubernetes#Troubleshooting#Containers#Application Reliability#Cloud Operations
C

Challenges.pro Editorial Team

DevOps and Platform Engineering Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.