Flaky Test Management Workflow: Detection, Quarantine, Ownership, and CI Recovery
CI/CDTestingSoftware QualityDeveloper ProductivityAutomation

Flaky Test Management Workflow: Detection, Quarantine, Ownership, and CI Recovery

CChallenges.pro Editorial Team
2026-11-16
8 min read

A practical workflow to detect flaky tests, quarantine them safely, assign ownership, measure CI impact, and restore them with confidence.

Flaky tests create a difficult CI/CD trade-off: rerunning every failure can hide real regressions, while treating every intermittent result as a hard failure can slow delivery and erode trust in the pipeline. This workflow provides a practical method for detecting flaky tests, confirming the failure pattern, quarantining safely, assigning ownership, and restoring tests without losing quality signals.

Overview

A flaky test is a test that produces different results when the code, test inputs, and relevant environment are intended to remain unchanged. The cause may be timing, shared state, order dependence, unstable external dependencies, resource contention, nondeterministic data, or a defect that appears only under particular conditions.

Flaky test management is not the same as adding retries. A retry can reduce noise for a developer waiting on a pull request, but it does not explain why the test failed or prove that the underlying build is safe. Treat retries as an observation mechanism and a temporary recovery control, not as a permanent repair.

The central workflow is:

  1. Capture the original failure and its execution context.
  2. Re-run the same test under controlled conditions.
  3. Classify the result as a likely product failure, test failure, infrastructure failure, or unconfirmed flake.
  4. Quarantine only when the evidence and risk justify it.
  5. Assign an owner, set a review date, and track remediation.
  6. Restore the test only after it meets explicit exit criteria.

Keep the original failure visible even when a CI system allows a job to pass after a rerun. Reports should distinguish first-attempt failures, successful reruns, repeated failures, and infrastructure interruptions. That distinction is essential for useful software delivery metrics and for deciding whether a change should be merged.

Checklist by scenario

When a test fails once

  • Record the commit, branch, test identifier, runner version, operating system or container image, dependency versions, and relevant environment variables.
  • Preserve logs, screenshots, traces, artifacts, and the first failure stack trace before rerunning.
  • Check whether the failure is deterministic by running the same test again without changing the code.
  • Compare the failure with recent failures in the same test, suite, or environment.
  • Do not label the test flaky solely because a retry passes. A passing retry is evidence for investigation, not a diagnosis.

When the rerun passes

Use a controlled confirmation sequence. Run the isolated test repeatedly, then run it in its normal suite and position. If possible, vary the worker count, test order, and clean versus reused environment. Record each attempt rather than relying on the final status.

A useful decision tree is:

  1. Does the test fail repeatedly under the same conditions? Treat it as a reproducible test or product failure and investigate the assertion, implementation, fixture, or environment.
  2. Does it pass in isolation but fail in a suite? Investigate shared state, order dependence, database cleanup, ports, files, clocks, and parallel execution.
  3. Does it fail only on one runner, operating system, or container image? Compare resource limits, installed dependencies, locale, time zone, networking, and service availability.
  4. Does it fail intermittently across equivalent runs? Mark it as a likely flaky test, attach the evidence, and begin ownership and remediation.
  5. Is there insufficient evidence? Keep the failure visible and collect more data. Do not quarantine simply to make the pipeline green.

When a test blocks a pull request

  • Separate product-risk failures from known test instability in the status message.
  • Allow an approved, bounded rerun policy only if the team can still see the initial failure.
  • Require a link to the test-health record, issue, or ownership entry when a failure is overridden.
  • Escalate immediately if the test covers a high-risk path, security control, migration, data integrity rule, or release gate.
  • Review the change and the test together. A new code path can expose a real race that resembles an existing flake.

When deciding whether to quarantine

Quarantine should be a controlled exception, not a hidden skip. Before suppressing a test, confirm that the team has a reproducible failure pattern, a named owner, a tracking item, a reason, a creation date, and an expiry or review date. Define whether the test is skipped, allowed to fail, moved to a non-blocking job, or run with a separate signal; these behaviors are not equivalent across CI providers or test runners.

Use this policy template:

Test quarantine record

  • Test and suite:
  • Observed failure pattern:
  • Evidence and affected environments:
  • Risk if the test is temporarily non-blocking:
  • Owner and backup owner:
  • Issue or investigation link:
  • Quarantine method and CI job:
  • Created date:
  • Next review date:
  • Exit criteria:

When fixing and restoring the test

  • Reproduce the original failure before changing the test where practical.
  • Fix the underlying cause rather than increasing arbitrary waits or retry counts.
  • Add diagnostics that will make a future failure actionable.
  • Run the test repeatedly in isolation and in its normal suite.
  • Test the relevant execution modes, such as parallel workers, clean environments, and supported operating systems.
  • Remove the quarantine configuration and verify that the test is again reported in the blocking path.
  • Record the repair and retain enough history to compare failure rates before and after the change.

What to double-check

CI reporting fields

A useful flaky test dashboard does not need to be complex, but it must preserve the distinction between attempts and outcomes. Track at least:

FieldPurpose
Test identifierProvides a stable reference even when display names change.
First-attempt failuresShows how often the test interrupts the normal pipeline path.
Rerun countMeasures how often teams depend on retries.
Rerun success rateHighlights failures that disappear on a later attempt.
Failure rateShows the proportion of executions that fail, with the calculation defined clearly.
Quarantine ageShows how long a known problem has remained outside the normal quality gate.
Time to repairMeasures the time from confirmed flake to restored test.
Environment and runnerReveals concentration around a platform, image, dependency, or worker type.
Owner and review dateTurns an alert into an actionable maintenance item.

Define each metric before comparing teams or time periods. For example, failure rate may mean failed executions divided by all executions, while a rerun success rate may mean successful reruns divided by rerun attempts. Mixing these definitions can create misleading trends.

Test runner and CI semantics

Verify how your tools implement retries, reruns, skipped tests, allowed failures, matrix jobs, and job-level versus test-level status. Some systems rerun a complete job; others rerun individual tests or failed cases. Some report the final attempt as the job result while retaining the first failure only in logs. Check the actual configuration and generated report rather than assuming two tools behave alike.

Also verify whether a quarantine marker is applied during test discovery, execution, reporting, or branch protection. A test that is omitted entirely provides less signal than a test that runs and is reported as non-blocking. Use the least suppressive behavior that keeps delivery moving without misrepresenting coverage.

Evidence of common root causes

  • Timing: fixed sleeps, race-sensitive assertions, asynchronous work without a reliable completion condition, or clock-dependent logic.
  • Shared state: records, files, ports, queues, environment variables, or caches left behind by another test.
  • External dependencies: live network calls, third-party services, rate limits, unstable test data, or unavailable local services.
  • Resource contention: CPU, memory, disk, file descriptors, connection pools, or worker limits that vary across runners.
  • Order and parallelism: a test that passes alone but fails after a particular test or under a particular worker count.
  • Data and time: locale, time zone, daylight-saving transitions, random seeds, generated identifiers, and boundary dates.

Common mistakes

  • Making retries the solution: Retries reduce visible failures but can increase the time before a real defect is noticed. Bound them, measure them, and assign follow-up work.
  • Quarantining without an owner: An unowned test becomes permanent technical debt. Every exception needs a responsible team and a review date.
  • Deleting a difficult test: Removal may hide a valuable product-risk check. First determine whether the test can be made deterministic or replaced with a clearer layer of testing.
  • Changing multiple variables at once: Updating timeouts, fixtures, runner images, and application code together makes the repair difficult to validate.
  • Ignoring infrastructure failures: A failed service, exhausted resource, or corrupted workspace should be classified separately from test flakiness.
  • Using one threshold everywhere: A suitable review threshold depends on test criticality, execution volume, pipeline role, and the cost of a false pass. Use thresholds as triggers for investigation, not as proof of causation.
  • Closing the issue when the test passes once: Restoration should require repeated evidence in the environments and execution modes that previously exposed the problem.

When to revisit

Review flaky test health at least during a regular weekly or sprint-based engineering review, and sooner when a high-risk test begins failing intermittently. Revisit the workflow before changing retry policies, migrating CI providers, upgrading test runners, changing container images, increasing parallelism, or reorganizing suites. Tool updates can change rerun semantics, report formats, quarantine configuration, and the meaning of a successful job.

Use this short review checklist:

  1. List new tests with repeated failures and tests whose rerun rate is rising.
  2. Sort quarantined tests by age, risk, and owner.
  3. Confirm that every quarantine still has a valid reason, issue, review date, and exit criteria.
  4. Compare first-attempt failures with final job outcomes so retries are not hiding a trend.
  5. Sample logs from the oldest and most frequently rerun tests.
  6. Restore tests that meet their exit criteria and remove obsolete configuration.
  7. Update the workflow after CI or test-runner changes, and document any change in rerun or reporting semantics.

For related pipeline diagnosis, use the CI/CD Pipeline Failure Troubleshooting decision tree. If scheduled test jobs are part of the workflow, review the CI/CD cron expression guide to verify timing and time-zone assumptions. The practical goal is not a permanently green dashboard; it is a pipeline whose failures remain visible, explainable, owned, and progressively easier to trust.

Related Topics

#CI/CD#Testing#Software Quality#Developer Productivity#Automation
C

Challenges.pro Editorial Team

DevOps and Platform Engineering Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.