A failed CI/CD pipeline is easier to fix when you identify the first broken assumption instead of scanning every log line. This guide provides a repeatable decision tree for checkout, dependency installation, tests, artifact creation, permissions, deployment, runner failures, and rollback, followed by an evidence template and a review cadence for recurring pipeline problems.
Overview
Start with the first stage that reports a failure, not the last stage that appears red in the pipeline interface. Later stages often fail because an earlier stage did not produce the expected file, environment variable, artifact, or deployment output.
A practical CI/CD pipeline failure investigation has four goals:
- Locate the earliest meaningful failure.
- Separate application defects from automation, infrastructure, and configuration problems.
- Collect enough evidence to reproduce or compare the failure.
- Apply the smallest safe correction and verify that the pipeline remains healthy.
Use this decision tree whenever a pipeline fails:
- Did the job start? If not, inspect workflow triggers, runner availability, queueing, branch rules, and permissions.
- Did source checkout complete? If not, investigate repository access, commit references, submodules, and checkout credentials.
- Did the environment initialize? Check the runner image, operating system, runtime versions, required tools, and environment variables.
- Did dependencies install? Verify lockfiles, package registries, network access, credentials, and cache behavior.
- Did tests or static checks run? Distinguish an actual assertion failure from a test process crash, timeout, or missing service.
- Was the artifact created? Confirm build outputs, paths, permissions, naming, and upload steps.
- Did deployment begin and complete? Check target availability, identity, authorization, configuration, health checks, and rollout status.
- Is the failure intermittent? Compare runner, timing, concurrency, dependency, and external-service details before retrying repeatedly.
Do not rerun a failed job as the only response. A retry can identify a transient problem, but it can also hide a deterministic defect and make the failure pattern harder to understand.
What to track
1. Checkout and source resolution
Checkout failures commonly involve an invalid branch or commit, insufficient repository permissions, expired credentials, unavailable submodules, or a shallow clone that does not contain information required by a later command. Record the exact commit SHA, repository, ref, submodule setting, and checkout action or plugin version. Confirm that the pipeline is building the commit you expect, especially when several branches or merge queues are active.
2. Runtime and dependency installation
Capture the runner operating system, architecture, language runtime, package-manager version, lockfile state, and relevant cache keys. A dependency step may fail because a package is unavailable, a private registry cannot be reached, a token is missing, or the lockfile is inconsistent with the manifest. It may also succeed while restoring an unsuitable cache. Compare a failing run with the last successful run rather than changing several versions at once.
When investigating credentials, log the variable names and whether they are present, but never print secret values. Check whether the credential is available in the event context, environment, job, and shell where it is used.
3. Tests, linting, and quality checks
First determine whether the test command returned an assertion failure, crashed before collecting tests, timed out, or could not connect to a required service. Record the test target, selected configuration, parallelism, seed where applicable, and any service containers or databases used by the job.
A single failing test may indicate a code defect. A group of unrelated failures often points to a shared fixture, clock, locale, database state, network dependency, or runner change. A test that passes only on rerun should be tracked as a reliability issue, not treated as resolved.
4. Artifact creation and transfer
Build failures are not limited to compilation. Check whether the expected output directory exists, whether generated files were created, whether the artifact path is relative to the correct working directory, and whether the archive includes the intended files. Confirm that an upload step uses the same name and path that downstream jobs expect. A successful build followed by a missing artifact usually indicates a handoff or path problem.
5. Permissions and deployment
Separate authentication from authorization. Authentication answers whether the pipeline has a valid identity; authorization answers whether that identity can read, write, deploy, or modify the target resource. Inspect the account or workload identity, token audience or scope where relevant, target environment, namespace or project, and approval requirements.
If deployment completed but the service is unhealthy, move from pipeline logs to application and platform signals. Check rollout status, container events, health checks, configuration references, resource availability, and recent changes. For Kubernetes-related failures, a separate Kubernetes troubleshooting decision tree can help narrow pod and deployment symptoms.
6. Runner and infrastructure behavior
Suspect the runner when failures correlate with a specific machine, image, architecture, queue, workspace, or concurrency level. Track disk space, memory pressure, available tools, network reachability, clock settings, and workspace cleanup. Hosted and self-hosted runners may differ in preinstalled software and isolation, so record the runner label or image rather than describing it only as “the build server.”
Error-pattern reference
| Observed pattern | Likely area | First checks |
|---|---|---|
| Job never starts or remains queued | Trigger, runner, capacity, or policy | Workflow event, branch rules, runner labels, queue, concurrency settings |
| Repository or submodule access denied | Checkout credentials or permissions | Ref, token scope, deploy key, submodule URL, repository access |
| Package cannot be found or downloaded | Registry, lockfile, network, or cache | Registry URL, authentication, lockfile, cache key, proxy settings |
| Tests fail only in CI | Environment, timing, service, or test isolation | Runtime versions, timezone, locale, parallelism, service readiness |
| Artifact not found by a later job | Path, naming, upload, or retention | Working directory, artifact name, upload result, download path |
| Deployment returns unauthorized or forbidden | Identity or authorization | Credential source, target environment, scope, role, approval gate |
| Deployment succeeds but health check fails | Application or platform configuration | Logs, rollout events, configuration, secrets, resource limits, probes |
| Failure disappears after retry | Flaky test, runner, network, or external service | Run timing, runner identity, concurrency, dependency and service status |
Evidence-collection template
Attach the following information to the incident, ticket, or pipeline review:
- Run: pipeline URL or identifier, date and time, branch, commit SHA, event type, and failed job.
- Failure: first failing command, exit code, concise error message, and log lines immediately before and after it.
- Environment: runner type, operating system, architecture, runtime versions, relevant tool versions, and workspace details.
- Inputs: changed files, lockfile changes, configuration revisions, feature flags, and non-secret variable names.
- Dependencies: package registry, external service, database, container image, or cloud target involved.
- Comparison: last successful run, first known failing run, and whether a retry changed the result.
- Action: workaround, permanent fix, owner, and verification run.
Cadence and checkpoints
Treat pipeline troubleshooting as both an incident activity and a recurring engineering practice. During an active failure, preserve logs and run metadata before rerunning or editing the workflow. After recovery, document the root cause, the misleading symptoms, and the signal that would have identified it sooner.
On a weekly or biweekly team cycle, review recent failures by stage. Look for repeated dependency errors, flaky tests, long queues, artifact handoff problems, and deployment reversions. The objective is not merely to count red runs; it is to identify work that repeatedly interrupts delivery.
On a monthly or quarterly review, compare pipeline behavior with the previous period. Track failure rate, average recovery time, rerun rate, stage duration, queue time, flaky-test recurrence, and the proportion of failures caused by code, configuration, infrastructure, or external services. Use consistent definitions so that changes reflect real trends rather than altered reporting.
Schedule a deeper checkpoint after runner-image changes, runtime upgrades, dependency-manager changes, CI provider changes, repository migrations, authentication changes, or deployment-platform upgrades. These events can alter defaults, available tools, permission behavior, caching, and network access.
How to interpret changes
A higher failure rate does not automatically mean the application became less reliable. It may reflect more frequent releases, broader test coverage, a new quality gate, or improved reporting. Always compare failure volume with run volume and stage duration.
Use patterns to choose the next investigation:
- Failures concentrated in one stage: inspect that stage's inputs, ownership, and recent configuration changes.
- Failures spread across unrelated repositories: investigate shared runners, images, registries, credentials, or provider changes.
- Duration rises before failure rate rises: check queueing, resource pressure, dependency resolution, and timeouts.
- Reruns frequently pass: classify the instability by test, runner, external dependency, timing, or concurrency before adding retries.
- Deployments pass but rollbacks increase: inspect post-deployment checks, configuration drift, migration ordering, and release verification.
Prefer fixes that improve diagnosis as well as reliability. Examples include pinning important tool versions, validating required variables early, publishing structured test reports, adding explicit artifact checks, and recording runner metadata. Avoid masking a failure with broad retries, ignored exit codes, or disabled tests unless the change is temporary, documented, and actively owned.
Pipeline data can also support broader software delivery reviews. If your team tracks delivery performance or reliability targets, connect pipeline findings to the relevant SLO and error budget review rather than treating CI/CD as an isolated system.
When to revisit
Return to this checklist whenever a pipeline fails, a repository is onboarded, or a deployment process changes. Make the evidence template part of the incident response runbook so that the first responder does not have to reconstruct basic context from memory. The DevOps developer onboarding checklist can also help verify that new repositories have the expected access, secrets, environments, and first-deployment path.
Refresh your troubleshooting notes at least monthly or quarterly, depending on release frequency. Update provider-specific error messages, runner behavior, authentication requirements, action or plugin versions, and example commands after major CI/CD platform releases or significant infrastructure changes. Remove steps that no longer match the organization’s pipeline design.
For the next failure, use this short action sequence:
- Freeze the failing run details and identify the earliest meaningful error.
- Classify the failure as source, environment, dependency, test, artifact, permission, deployment, or runner related.
- Compare it with the last successful run and isolate the smallest changed variable.
- Apply one controlled correction or reproduce the problem in an equivalent environment.
- Verify the fix with a fresh run, record the evidence, and create follow-up work for recurring causes.
Consistent diagnosis turns a failed CI/CD pipeline from a time-consuming interruption into a measurable input for better build automation, deployment safety, and developer productivity.
