The cgroup v1 Nodes Your Next Kubernetes Upgrade Can't Boot
Four separate floors decide whether a node survives v1.35, and the reward for clearing them is a swap feature with a sharp cliff once working sets spill out of RAM.
Where the v1 nodes are hiding
cgroup v1 rarely survives on the nodes anyone looks at. Start with long-lived base images: Amazon Linux 2, Ubuntu 20.04, and CentOS/RHEL 7- and 8-era hosts. They shipped with v1 defaults, pre-5.8 kernels, or both. Next are custom AMIs and golden images nobody has rebuilt in two years. After those, check the pools that rarely get touched: GPU, batch and on-prem.
A node has to clear four separate floors. Each one can fail an upgrade on its own:
| Requirement | Minimum | How to check |
|---|---|---|
| cgroup mode | v2 unified hierarchy | stat -fc %T /sys/fs/cgroup/ returns cgroup2fs; v1 hosts return tmpfs |
| Linux kernel | 5.8 or later | kubectl get nodes -o wide, KERNEL-VERSION column |
| Container runtime | containerd 1.4+ or CRI-O 1.20+ | Same command, CONTAINER-RUNTIME column |
| cgroup driver | kubelet and runtime must match | Compare kubelet cgroupDriver with the runtime config, such as containerd's SystemdCgroup |
The failCgroupV1: false override exists. Record each use as a dated exception with a named owner, with an expiry. Switching cgroup mode already needs a boot parameter change and a reboot. Rebuilding the image and rolling the pool costs about the same as patching hosts in place. The rebuild also leaves no override behind to forget.
What v2 changes on day one
- Whole-container OOM kills. The OOM killer now kills the entire container, not one process. That ends the zombie pod, where a worker died and the supervisor kept the container looking healthy. Restart patterns in dashboards and on-call runbooks will change.
- PSI metrics on by default. Pressure Stall Information records how long tasks stalled waiting on memory, CPU or I/O. It is a better saturation signal than utilization. It is the instrument to have in place before enabling swap.
- In-place Pod-level resizing without recreating pods. Memory QoS with
memory.min/memory.lowtiering is still alpha in v1.36. Do not design around it yet.
Reading the swap numbers honestly
Node swap works only because cgroup v2 accounts for swap separately. Chris Short highlights GKE benchmarks run on NVMe Local SSDs. Next to the 3x sandbox result, they show the upside and the cliff:
- gVisor headless Chrome went from 80 to 160 pods per node.
- A Linux kernel build with its memory limit halved from 600 MB to 300 MB finished faster: 374s against 433s.
- The same build at 200 MB ran more than 40% slower.
The middle result looks backwards. The source does not explain it. We think this is what happens, though we have not tested it. Without swap, the kernel can reclaim only file-backed pages. A memory-constrained build evicts the page cache it needs and rereads from disk. With swap, cold anonymous pages go out instead and the hot file cache stays resident. At 200 MB nothing cold is left, so the build pages its own working set.
Size node pools on RAM alone and treat swap as headroom for idle and bursty pages. Adding the two together is how a pool reaches the cliff.
The tests ran on local NVMe, the first of three caveats on the headline figure. Network-attached block storage will have much worse swap-in latency. The baseline configuration for the 433s run is not given, so treat the build gain as directional until it is reproduced. Anything in anonymous memory can land on the swap device. Swap encryption and local-disk lifecycle are therefore part of the design.
The smart move
The order is sweep, gate, then harvest. The sweep takes an afternoon with a DaemonSet or kubectl debug node/.... The gate is a pipeline precondition of zero v1 nodes, with one pool canaried through kubeadm preflight before the fleet. After that, prototype swap. Setup is a few lines of KubeletConfiguration with LimitedSwap, applied to Burstable pods on mostly-idle workloads such as agent sandboxes, headless browsers and CI runners. Latency-sensitive services stay on Guaranteed QoS. LimitedSwap does not apply there.
What to do
Run a DaemonSet or kubectl debug node sweep across every cluster this week that reports cgroup mode (stat -fc %T /sys/fs/cgroup/ must return cgroup2fs), kernel 5.8+, containerd 1.4+ or CRI-O 1.20+, and matching cgroup drivers, including GPU, batch and on-prem pools.
Add 'zero cgroup v1 nodes' as a hard precondition in your v1.35 upgrade pipeline this sprint, canary one pool through kubeadm preflight first, and rebuild failing base images instead of setting failCgroupV1: false.
Prototype LimitedSwap this quarter on one local-NVMe pool of Burstable, mostly-idle pods such as agent sandboxes, headless browsers or CI runners, with PSI memory-pressure alerts wired in and rollout gated on p99 latency rather than pod count.