overlayfs rejects relative lowerdir paths, capping image depth under Docker 29

[kvanzuijlen]

Description

gVisor's overlayfs rejects a mount that uses relative lowerdir paths. On Linux, the kernel resolves those paths against the working directory of the calling process. The mount succeeds.

Expected: the mount succeeds. Observed: mount(2) returns EINVAL.

Docker 29 triggers the rejection on images with many layers. containerd switches to relative lowerdir paths after a chdir once the absolute option string would pass one page. Shallow images therefore work and deep ones fail:

failed to mount /tmp/containerd-mount3312412956: mount source: "overlay",
 target: "/tmp/containerd-mount3312412956", fstype: overlay, flags: 0,
 data: "workdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/60/work,
        upperdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/60/fs,
        lowerdir=59/fs:58/fs:57/fs:56/fs:51/fs:...:13/fs", err: invalid argument

A tmpfs is already mounted at /var/lib/docker, so this is a different failure from #12475.

With the default --data-root, images of about 38 layers or more fail. Those images neither run nor build. docker create paketobuildpacks/builder-jammy-java-tiny fails at 39 layers. That image is the default builder for Cloud Native Buildpacks. pack build and Spring Boot's bootBuildImage therefore do not work under gVisor.

Steps to reproduce

Run this inside a sandbox. Docker is not needed.

mkdir -p /t/rel && cd /t/rel
mkdir -p l0 l1 u w m

# Relative lowerdir paths. Fails.
mount -t overlay overlay -o lowerdir=l0:l1,upperdir=u,workdir=w m
# mount: mounting overlay on m failed: Invalid argument

# The same mount with absolute paths. Succeeds.
mount -t overlay overlay -o lowerdir=/t/rel/l0:/t/rel/l1,upperdir=/t/rel/u,workdir=/t/rel/w /t/rel/m

To separate layer count from path length, I called mount(2) directly on a tmpfs and varied each:

lower layers	paths	option string	result
80	absolute, short	749 B	mounts
40	absolute, long	3119 B	mounts
60	absolute, long	4599 B	ENAMETOOLONG
10	relative	about 100 B	EINVAL

The number of lower layers does not cause the failure. The one-page limit on the option string behaves as it does on Linux. Only relative lowerdir paths fail.

I verified two workarounds. Both are Docker-specific, and both avoid relative lowerdir paths rather than fix the mount. The chdir behaviour lives in containerd's mount helper, so other containerd consumers inside a sandbox are still affected.

• A shorter --data-root keeps the absolute string under a page for longer. The limit rises by a few layers.
• --feature containerd-snapshotter=false moves Docker off the containerd snapshotter entirely. The classic overlay2 driver keeps a symlink index at <root>/l/<id>, so its option string stays short. The same 39-layer image then creates successfully. Docker in gVisor: docker v29 fails to mount overlayfs #12475 recommends this flag for other Docker 29 problems. It is not a durable answer: Docker documents overlay2 as "a legacy storage driver that is superseded by the overlayfs containerd snapshotter".

runsc version

GKE Sandbox manages runsc. The node does not expose its version.

docker version (if using docker)

Docker Engine 29.7.2, in a Docker-in-Docker sidecar. It starts with --iptables=false --ip6tables=false and a tmpfs at /var/lib/docker.

uname

/proc/version reports Linux version 4.19.0-gvisor.

kubectl (if using Kubernetes)

GKE 1.36.3-gke.1537000. Node pool: imageType COS_CONTAINERD, sandboxConfig.type GVISOR.

[jdymitarai]

I investigated this issue and verified that on Linux, \overlayfs\ resolves relative \lowerdir, \upperdir, and \workdir\ paths against the caller's working directory (\AT_FDCWD\ / \pwd).

I have submitted PR #14728 which wires \CtxWorkingDir\ into VFS and allows \lowerdir, \upperdir, and \workdir\ to resolve relative to the task's current working directory, along with a regression test in \mount.cc.