A Kubernetes version upgrade is safer when treated as a compatibility project rather than a single maintenance command. This reusable checklist covers planning, deprecated APIs, control planes, nodes, add-ons, storage, networking, workload validation, rollback decisions, and the checks to repeat before every upgrade window.
Overview
A Kubernetes cluster upgrade changes more than the server binary. The control plane, node components, admission policies, controllers, network plugins, storage integrations, observability agents, deployment tooling, and application manifests all have compatibility boundaries. A successful Kubernetes cluster upgrade therefore depends on a tested sequence and clear evidence at each stage.
Start by recording the current state. Capture the Kubernetes version, node versions, cluster provider or installation method, enabled APIs, admission webhooks, installed operators, critical add-on versions, storage classes, ingress implementation, and the workloads that matter most to availability. Exporting configuration and collecting health information gives the team a baseline for comparison after the change.
Do not assume that a control-plane upgrade is reversible. Some managed services, infrastructure providers, and cluster tools support limited downgrade paths or none at all. Define the rollback strategy before execution. In many environments, rollback means restoring workloads or data, repairing an incompatible add-on, or rebuilding a cluster from tested configuration rather than reverting the cluster version.
Version-skew reference table
| Component | What to verify | Why it matters |
|---|---|---|
| Control plane | Target version is supported by the provider or distribution | Upgrade order and available recovery options vary |
| kubelet and nodes | Node versions stay within the supported skew for the control plane | Unsupported combinations can cause registration or scheduling problems |
| kubectl and CI/CD tools | Client versions can communicate with the target cluster | Automation may fail even when workloads appear healthy |
| Controllers and operators | Release notes list compatibility with the target Kubernetes version | Custom resources and reconciliation may break independently |
| Admission webhooks | Webhook API versions, certificates, timeouts, and failure policies | A failing webhook can block creates, updates, or repairs |
This table is a planning aid, not a substitute for the version-skew policy and upgrade documentation for your Kubernetes distribution. Recheck those references for every target version.
Checklist by scenario
Before the upgrade
- Set the target and scope: document the source version, target version, clusters affected, maintenance window, owners, communication channel, and success criteria.
- Confirm support: check the provider or distribution’s supported-version policy, required intermediate versions, node image requirements, and upgrade order.
- Inventory dependencies: list ingress controllers, CSI and CNI components, DNS, metrics agents, autoscalers, policy engines, service meshes, operators, admission webhooks, and backup systems.
- Audit workloads: identify critical services, disruption budgets, stateful workloads, singleton replicas, host-level dependencies, and deployments with tight resource limits.
- Review capacity: confirm that the cluster can drain nodes without making essential pods unschedulable. Revisit requests, limits, affinity, taints, and tolerations using the Kubernetes resource requests and limits guide.
- Back up deliberately: verify recent, restorable backups for application data and cluster configuration. A backup that has never been restored is only an assumption.
- Test a representative path: use a staging or nonproduction cluster with similar add-ons, policies, node types, and workloads. Exercise deployment, scaling, ingress, persistent volumes, scheduled jobs, and a controlled node drain.
- Freeze unrelated changes: pause nonessential platform and application changes during the upgrade window so failures have fewer possible causes.
Deprecated-API audit
Deprecated APIs are a frequent source of upgrade failures because an object may continue working until it is read, updated, or recreated. Search manifests, Helm charts, Kustomize bases, Git repositories, admission rules, and generated resources for APIs removed or changed in the target release.
- List API resources served by the current cluster and compare them with the target release documentation.
- Find manifests using older API groups, including workload, policy, ingress, autoscaling, and authentication-related resources.
- Check rendered output rather than only source templates; chart defaults and overlays can produce an unexpected API version.
- Locate stored objects that will need conversion, not only files that are currently deployed.
- Upgrade or replace operators and controllers that still use removed APIs.
- Apply corrected manifests in a test cluster and verify that reads, updates, rollouts, and deletes work.
Useful audit commands depend on the tools and distribution in use. As a starting point, review API discovery with kubectl api-resources, inspect cluster objects with kubectl get, render Helm output before applying it, and use a deprecated-API scanner that supports your Kubernetes version. Treat scanner output as an investigation list, not proof that every object is safe.
During the upgrade
- Announce the start and record the exact versions and timestamps.
- Recheck cluster health, node readiness, pending pods, recent events, and error rates.
- Upgrade the control plane according to the provider or distribution’s documented sequence.
- Upgrade node pools in controlled batches. Respect disruption budgets and keep enough spare capacity for rescheduling.
- Watch node conditions, kubelet logs, scheduling latency, pod restarts, and image-pull failures after each batch.
- Upgrade add-ons in a dependency-aware order, especially networking, storage, ingress, DNS, policy, and observability components.
- Pause if a defined exit condition is reached. Do not continue because the maintenance window is running short.
After the upgrade
- Confirm all control-plane and worker nodes report the intended versions and are Ready.
- Check system namespaces, controllers, operators, admission webhooks, DNS, metrics, ingress, and storage controllers.
- Run smoke tests for deployment, service discovery, ingress, secrets, configuration, persistent volumes, autoscaling, jobs, and scheduled workloads.
- Inspect application error rates, latency, saturation, logs, traces, and alerts rather than relying only on pod status.
- Perform a controlled rollout and, where appropriate, a node drain test for representative services.
- Compare the post-upgrade baseline with the pre-upgrade record and document exceptions.
If an application is unhealthy after the change, use the Kubernetes troubleshooting decision tree to separate scheduling, image, configuration, storage, and runtime symptoms.
What to double-check
Storage: Verify CSI driver compatibility, volume attachment, snapshot behavior, filesystem permissions, reclaim policies, and recovery procedures. Stateful applications deserve a separate validation plan because a healthy pod does not prove that its data path is healthy.
Networking: Test internal service traffic, external ingress, network policies, load balancer behavior, DNS, and dual-stack assumptions if applicable. Confirm that CNI and ingress components use supported APIs and images.
Admission and security: Test policy enforcement, image verification, secret injection, certificate rotation, and webhook timeouts. A security control that fails closed may prevent routine operations; one that fails open may create a different risk. Record the intended behavior.
Delivery automation: Run the same deployment commands used by CI/CD, not just manual kubectl commands. Check client compatibility, authentication, namespace permissions, manifest rendering, rollout waits, and rollback commands. The production-ready CI/CD checklist can help turn these checks into pipeline gates.
Observability: Make sure dashboards, alerts, log collection, traces, and audit records still receive data. Monitor both platform signals and user-facing service indicators throughout the observation period.
Preflight worksheet
| Question | Evidence to record |
|---|---|
| What is changing? | Source and target versions, clusters, node pools, and add-ons |
| What could block the change? | Deprecated APIs, unsupported integrations, capacity gaps, or failed tests |
| How will success be measured? | Health checks, smoke tests, service indicators, and stable alerts |
| What is the stop condition? | Specific symptoms that trigger a pause or escalation |
| What is recoverable? | Restore points, workload rollback, provider support, and rebuild steps |
| Who decides? | Named technical owner, application owners, and incident lead |
Common mistakes
- Upgrading production first: A small staging cluster may hide capacity, policy, or add-on problems. Make the test environment representative enough to expose them.
- Checking only pod status: Pods can be Running while ingress, storage, DNS, metrics, or application dependencies are failing.
- Ignoring rendered manifests: The API version in a source template may differ from the output generated by Helm, Kustomize, or a deployment controller.
- Draining without capacity planning: Pod disruption budgets and placement rules can make a node drain wait indefinitely or leave critical workloads pending.
- Updating every add-on at once: Bundling unrelated changes makes diagnosis difficult. Group changes by dependency and test them in sequence.
- Assuming rollback is a version downgrade: Confirm the actual recovery path before the window begins, including data consistency and persistent-volume behavior.
- Failing to document exceptions: Temporary compatibility workarounds become permanent risks when they are not assigned an owner and removal date.
When to revisit
Revisit this Kubernetes upgrade checklist before every cluster version change, not only major platform projects. Refresh it when the Kubernetes distribution changes its support policy, when a target release removes or changes APIs, when an operator or cloud integration is upgraded, or when the platform adds new admission, storage, networking, or security controls.
Review it during seasonal planning cycles and before maintenance windows with reduced staffing. Also update the checklist after an incident, failed rehearsal, restore exercise, or significant change to the internal developer platform. Keep one version-controlled copy with the cluster configuration and link each completed upgrade to its evidence: test results, approvals, commands, alerts, exceptions, and follow-up work.
For the next upgrade, begin with three actions: record the current cluster state, run the deprecated-API and dependency audit, and schedule a rehearsal on a representative environment. Confirm the stop conditions and recovery path with the incident owner before touching production. That preparation turns a risky version change into a controlled Kubernetes operations procedure.
