When setup-gcp bootstrap runs against an existing cluster that doesn't have the PodCertificate beta APIs, its reconcile path turns them on with UpdateCluster (cluster.go#L225-L249). That repair works. On 1.36 and older it's only half the job, and neither the tool nor the README says so.
What I measured (throwaway GKE clusters, 2026-09-21). Each test ran the full loop: ClusterTrustBundle → kubelet-issued PodCertificateRequest → signed → pod Running.
|
1.36 |
1.37 |
UpdateCluster turning on both beta APIs on a running cluster |
accepted, about 8 min |
accepted, about 11 min |
v1beta1 served afterward |
yes |
yes |
| Pod certificate projection on nodes that existed before the update |
❌ MountVolume.SetUp failed: unimplemented |
✅ |
| Pod certificate projection on nodes created after the update |
✅ |
✅ |
PodCertificateProjection is a kubelet feature. It's gated through 1.36 and GA in 1.37, so on 1.36 a node started before the update never gets it. The control plane looks healthy and the new APIs are served, but any pod with a pod certificate volume that lands on an old node fails to mount. The fix is to replace those nodes, for example by recreating the node pool or letting a surge upgrade cycle them.
What goes wrong for users
- On a 1.36 cluster, bootstrap logs
Mismatch in EnableK8SBetaApis, the update succeeds, and bootstrap carries on. Workloads then fail on every node that was already running, with nothing in the output pointing at the cause.
- The warning at
tools/setup-gcp/README.md#L77-L83 makes two claims that didn't hold up in testing:
- "Enabling the APIs later on an existing cluster is not recoverable in place … the APIs do not become served — the cluster must be recreated." In testing, the APIs were served after the update on both 1.36 and 1.37. What's actually missing on 1.36 is the node side.
- "GKE 1.37 or higher, where the APIs are served by default and no beta enablement is needed." A plain 1.37 cluster serves PodCertificateRequest and ClusterTrustBundle only under
certificates.k8s.io/v1. Substrate's controllers use the v1beta1 types (e.g. cmd/podcertcontroller/internal/signercontroller/signercontroller.go, cmd/atelet/trustbundle.go), so a bring-your-own 1.37 cluster set up by following the README hangs at "Waiting for podcertificate ClusterTrustBundles to be ready". setup-gcp itself is fine here because it always requests the beta APIs.
Proposed fix
- In the reconcile path, after a successful beta-API update on a cluster whose control plane is below 1.37, log a clear warning. It should say that existing nodes won't get pod certificate projection until they're replaced, and give the command to do it. Automatically recreating node pools seems too destructive for a setup tool, but I'm open to it behind a flag.
- Rewrite the README warning: the APIs can be turned on in place on any supported release; on 1.36 existing nodes have to be replaced afterward; on 1.37 no node replacement is needed, but the beta APIs are still required because Substrate uses
v1beta1.
I'm happy to send the PR.
When
setup-gcp bootstrapruns against an existing cluster that doesn't have the PodCertificate beta APIs, its reconcile path turns them on withUpdateCluster(cluster.go#L225-L249). That repair works. On 1.36 and older it's only half the job, and neither the tool nor the README says so.What I measured (throwaway GKE clusters, 2026-09-21). Each test ran the full loop: ClusterTrustBundle → kubelet-issued PodCertificateRequest → signed → pod Running.
UpdateClusterturning on both beta APIs on a running clusterv1beta1served afterwardMountVolume.SetUp failed: unimplementedPodCertificateProjectionis a kubelet feature. It's gated through 1.36 and GA in 1.37, so on 1.36 a node started before the update never gets it. The control plane looks healthy and the new APIs are served, but any pod with a pod certificate volume that lands on an old node fails to mount. The fix is to replace those nodes, for example by recreating the node pool or letting a surge upgrade cycle them.What goes wrong for users
Mismatch in EnableK8SBetaApis, the update succeeds, and bootstrap carries on. Workloads then fail on every node that was already running, with nothing in the output pointing at the cause.tools/setup-gcp/README.md#L77-L83makes two claims that didn't hold up in testing:certificates.k8s.io/v1. Substrate's controllers use thev1beta1types (e.g.cmd/podcertcontroller/internal/signercontroller/signercontroller.go,cmd/atelet/trustbundle.go), so a bring-your-own 1.37 cluster set up by following the README hangs at "Waiting for podcertificate ClusterTrustBundles to be ready". setup-gcp itself is fine here because it always requests the beta APIs.Proposed fix
v1beta1.I'm happy to send the PR.