Christopher Mosetick

Kubernetes: cAdvisor and ZFS Nodes

Did you move the world? …Or did the world move you?

Often you can feel like the world is moving you. A dependency breaks, you work around it, you wait for someone else to fix it, and you move on. This is a short story about the other direction: a fix that existed but wasn’t reaching anyone, and what it took to actually push it the last mile into the hands of every Kubernetes operator who needed it.

The Symptom

I run kind Kubernetes IN Docker, or in my case Kubernetes in podman on a machine whose underlying storage is ZFS. On a node whose container store uses the native ZFS graph driver, kubelet would refuse to start. The error pointed at the storage layer but explained nothing:

cannot find filesystem info for device "tank/..."

kubelet would then exit 1 and crash-loop. Nothing in that message points at the real cause, and it took a multi-hour trace to land on it.

The Real Cause

Kubernetes vendors cadvisor to gather node and container stats. cadvisor’s ZFS support works by shelling out to the zfs command line tool:

zfs list -Hp -o used,available,usedbydataset <dataset>

On my node, /dev/zfs was present but the zfs userspace binary was not. So that subprocess call failed with exit 127. Released cadvisor treated that failure as fatal for the filesystem: it produced no Fs record for the ZFS-backed rootfs, kubelet’s ContainerManager never got the filesystem info it required, and the whole thing fell apart.

The correct behavior is obvious in hindsight: if the zfs tool is missing, do not treat it as a hard error, fall back to a plain statfs on the mount point like every other filesystem. That is exactly what a small fix on cadvisor master did.

The Fix Existed. It Just Wasn’t Published Anywhere.

Here is the part that makes this a “trenches” story rather than a “file a bug” story. The fix was already written and merged. It landed on cadvisor master in July: “fs/zfs: fall back to VFS when zfs stats are unavailable.”

But it was in no released tag. It merged a few days after the last cadvisor release, and then it just sat there. Every shipping Kubernetes line still pinned a cadvisor that carried the bug. The fix was real, reviewed, merged, and completely unreachable by anyone actually running Kubernetes.

A merged commit that is in no release helps no one. This is the gap where most fixes quietly die, and where an operator either shrugs or starts pushing.

Pushing, Part One: Get a Release Cut

I opened the issue on cadvisor asking for a tagged release that contained the fix:

I was not the only one affected. Another developer had already asked, twice, for a backport to an older series so it could reach their Kubernetes version. Both of their requests had gone unanswered. The fix was popular and stuck.

The first thing I tried was the polite, private route. In mid-September I sent a direct message to one of the maintainers asking, essentially, “could you cut a tagged release, it should be fast.” A direct message felt reasonable and low-noise.

It produced nothing for a full week.

Slack DM on September 11 asking a cadvisor maintainer for a tagged release, linking cadvisor issue 3922

So I posted in the public Kubernetes #sig-node Slack channel, linking the issue and @-mentioning several people by name. That channel has over 5,000 members, so a plain post vanishes. Naming people is what makes it visible.

Public post in the Kubernetes sig-node Slack channel on September 18, @-mentioning several maintainers by name and linking cadvisor issue 3922

A new cadvisor release went out roughly 90 minutes later.

GitHub release page for cAdvisor v0.60.6, released by dims, listing the fs/zfs fall back to VFS fix (#3918) in What's Changed

The lesson is worth stating plainly, because it is reusable: the private DM did nothing for a week; the public, multi-person, named ping in the right channel got a release in under two hours. The release was not gated by a calendar. It was gated by someone needing to press publish, and by that someone actually seeing the request. Visibility was the whole game.

To be fair to the maintainer, it was a batch flush, not a single-fix release. It bundled four already-merged PRs that had been waiting. He did not cut a release just for me. But the PRs had been sitting for weeks, and the ping is what turned “eventually” into “that afternoon.”

Pushing, Part Two: Get It Into Kubernetes

A cadvisor release is necessary but not sufficient. Kubernetes vendors a pinned version, so the fix does nothing for kind or for any real cluster until Kubernetes bumps its pin.

That is not maintainer-only work. It is a vendored dependency bump, and an external contributor can do it. So I did.

The mechanics, for anyone who wants to do the same:

  1. Fork kubernetes/kubernetes, and clone the fork as a blobless partial clone (--filter=blob:none) so it stays small on disk but can still push, unlike a shallow clone.
  2. hack/pin-dependency.sh github.com/google/cadvisor/[email protected]
  3. hack/update-vendor.sh to regenerate go.mod, go.sum, vendor/modules.txt, and the vendored tree.
  4. hack/verify-vendor.sh to confirm it is consistent before you push.
  5. Commit signed off (-s, for the DCO) and, in my case, GPG-signed with a YubiKey.

The result is a tidy five-file change: the top-level module files plus the two vendored cadvisor source files that actually carry the fix. Prow labeled it, the CNCF CLA went green, and a sig-node reviewer triaged it and kicked off CI.

GitHub PR timeline showing the /sig node comment and the kubernetes-prow bot adding the sig/node label and removing do-not-merge/needs-sig
GitHub PR checks panel for kubernetes/kubernetes #142272 showing pending required checks before merge

It merged about two hours after I opened it. The fix is already in the Kubernetes 1.38 alphas and will ship with the 1.38 release. Today’s stable releases don’t have it yet.

git log on kubernetes master showing merge commit 7efbf84 for pull request 142272 from cmosetick/bump-cadvisor-v0.60.6 on September 21 2026

Closing the Loop with kind

The whole thing started because of kind, so I closed that loop too. On the kind issue I had originally proposed a kind-side workaround (ship the zfs tool in the node image). The maintainer, reasonably, did not want to add binaries to a carefully slimmed image for a relatively untested path.

Rather than argue that, I took the upstream path and reported back: cadvisor is released, the Kubernetes bump is merged, and kind node images for 1.38 will pick up the fix automatically with no kind-side change at all. If you can’t wait, build your own node image from Kubernetes master. No new binaries, no image bloat, no special support burden. Everyone’s constraints respected, and the actual users get the fix.

Did You Move the World, or Did the World Move You?

The fix was not mine. I did not write the one-line VFS fallback. What I did was refuse to let a merged-but-unreleased commit stay stuck: I opened the issues, found the escalation path that actually worked, and then did the downstream Kubernetes vendor bump myself instead of waiting for someone else to notice it needed doing.

None of it required being a project maintainer. It required being an operator who hit the bug in the field, understood the whole chain from zfs list up through kubelet, and was willing to push at each link until the fix reached the people who needed it.

That is the difference I care about. The world was not going to move on its own here. The fix would have kept sitting in master. Somebody had to move it. This time, that somebody was me: from a Slack ping to a cadvisor release and a merged Kubernetes PR in three days.

Reference