TL;DR: “The node is frozen” is a symptom, not a diagnosis — on this cluster it had three unrelated root causes across three separate incidents: Longhorn iSCSI sessions dropping under NVMe/memory pressure, CIFS mounts blocking in uninterruptible D-state when the NAS disappeared, and kubelet evicting pods because it measured ephemeral-storage against the wrong disk. Each needed a different fix and none of the three fixes overlap. This post is the index: what each failure mode looks like from the outside, how to tell them apart fast, and links to the full postmortem for each.
A frozen Kubernetes node tells you almost nothing about why it’s frozen. No SSH, no ping, kubectl get nodes showing NotReady — that’s the same output whether the cause is a storage driver, a kernel D-state hang, or an eviction cascade nowhere near the actual pressure. Three incidents on the same homelab cluster produced the identical top-level symptom for three causally unrelated reasons. This post exists to shortcut the next round of “which one is this” — a decision tree, not a rehash of the individual investigations.
Three times, on the same cluster, the answer to “why is this node not responding” turned out to be a different subsystem entirely. Each time, the first hour of debugging looked identical: no SSH, no ping, sometimes no SysRq, journald showing the effect but not the cause. Each time, the actual root cause was two or three layers removed from where the investigation started.
This post is for anyone running k3s (or any Kubernetes distribution) with Longhorn, CIFS/SMB mounts, or a data directory split across disks, who has hit an unresponsive node and isn’t sure which failure mode they’re looking at. It assumes you’ve already read — or are about to read — at least one of the three linked postmortems; this is the map, not the territory.
Table of Contents
Open Table of Contents
The three failure modes, side by side
| Longhorn iSCSI + memory pressure | CIFS mount hang | kubelet root-dir eviction | |
|---|---|---|---|
| Symptom | Full node freeze, power-button reboot | Pods stuck Terminating, eventual node panic | Pod evicted despite using almost no disk |
| Trigger | NVMe fdatasync stall or memory overcommit cascading into Longhorn engine liveness | NAS goes offline while a pod is mid-I/O on a CIFS mount | kubelet --root-dir and the real data disk are different filesystems |
| Kernel state | hung_task_panic firing (or failing to fire) | Processes in TASK_UNINTERRUPTIBLE (D-state), immune to SIGKILL | No kernel-level hang — this one is a scheduler decision, not a freeze |
| First signal to check | dmesg / kdump for iSCSI session drops and memory pressure | ps aux for D-state (D) processes on the affected node | Eviction event message quoting ephemeral-storage thresholds |
| Fix category | Firmware + Longhorn timeout tuning + swap | Network-level liveness probe sidecar, not a mount-option fix | Bind-mount the real disk at the conventional kubelet path |
Start here: what does the eviction message or lack of one tell you
If there’s a Kubernetes eviction event with a specific pod name and a resource threshold quoted (ephemeral-storage, memory), start with kubelet root-dir eviction — that’s a scheduler decision, not a hang, and it’s the only one of the three where the node itself stays fully responsive. If the node is unresponsive to SSH and SysRq with no eviction events at all, it’s one of the other two, and the next check is whether the workload touches a CIFS/SMB mount or a Longhorn-backed PVC.
If it’s CIFS/SMB: check for D-state first
ps aux | grep ' D ' on the affected node before doing anything else. Processes in D-state ignore SIGKILL — that’s the fingerprint of the CIFS mount hang, and it means the fix is not a liveness-probe timeout tweak, it’s replacing any probe that touches the mount with one that doesn’t. NFS-style timeo/retrans options don’t apply to the CIFS module; don’t spend time on them.
If it’s Longhorn: check kdump before you assume it’s the disk
The Longhorn/memory postmortem covers two distinct triggers that both end in the same iSCSI session drop — an NVMe fdatasync stall cascading through etcd into a Longhorn liveness probe timeout, and, separately, memory overcommit deadlocking kswapd on ext4 inode flush before a crash dump could even be written. journald alone can’t distinguish these; kdump output can. If the cluster doesn’t have crashkernel configured yet, that’s the first gap to close — the memory-overcommit variant of this failure leaves no other record.
Why these don’t share a fix
It’s tempting to treat “node freezes” as one problem with one hardening checklist. It isn’t, on this cluster: the Longhorn fix is firmware and replica-timeout tuning, the CIFS fix is an architectural change to how liveness is probed, and the kubelet fix is a bind-mount that has nothing to do with either. Applying one incident’s fix as a blanket policy — for example, assuming every freeze is memory pressure and just adding swap — will miss the other two entirely and burn an incident window confirming a hypothesis that was wrong from the start.
The one thing genuinely shared across all three: none of them were diagnosable from journald alone. Each required either kdump, a process-state check (ps aux for D-state), or reading the eviction message literally instead of assuming it pointed at the obvious culprit.
References
- k3s Node Freeze: Longhorn iSCSI Stalls and Memory Pressure — full postmortem, NVMe/memory-overcommit root causes
- SMB Mounts in Kubernetes Will Freeze Your Node — full postmortem, CIFS D-state hang and the sidecar fix
- Jellyfin Evictions Traced to kubelet’s root-dir on Wrong Disk — full postmortem, ephemeral-storage miscalculation
