AutoscalingRunnerSet deletion wedges forever when githubConfigSecret is gone

[KR-Ravindra]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.2 (gha-runner-scale-set-controller); same code on master at 03328aa

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install a gha-runner-scale-set release with minRunners: 0 and wait until the AutoscalingRunnerSet (ARS) carries the runner-scale-set-id annotation, i.e. the scale set is registered with the Actions service.
2. Remove the configured secret: kubectl delete secret <githubConfigSecret> -n <ns> (drop its autoscalingrunnerset.actions.github.com/cleanup-protection finalizer first if the chart created it).
3. kubectl delete autoscalingrunnerset <name> -n <ns> (or helm uninstall, or delete the namespace).
4. The ARS stays in Terminating indefinitely. The controller logs, on every requeue:

ERROR AutoscalingRunnerSet Failed to initialize Actions service client for updating a existing runner scale set {"error": "failed to resolve app config: failed to get kubernetes secret: \"<ns>/<githubConfigSecret>\""}
ERROR AutoscalingRunnerSet Failed to delete runner scale set
ERROR AutoscalingRunnerSet Failed to clean up resources during deletion

The finalizer autoscalingrunnerset.actions.github.com/finalizer is never removed, so kubectl delete / helm uninstall hang and a namespace containing the ARS can never finish terminating. Only manual intervention (kubectl patch ... --type=merge -p '{"metadata":{"finalizers":null}}') or recreating a secret with the old name unblocks it.

The same happens whenever the secret is gone for good rather than temporarily: the namespace is being torn down and the secret went first, the secret was rotated to a new name and the old one deleted before the ARS, or Helm removed the secret before the ARS during uninstall.

Describe the bug

During deletion, Reconcile calls cleanUpResources, which ends with deleteRunnerScaleSet:

• deleteRunnerScaleSet builds an Actions client from githubConfigSecret via GetActionsService and returns the error if that fails:

  actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go

  Lines 768 to 772 in 03328aa

  	actionsClient, err := r.GetActionsService(ctx, autoscalingRunnerSet)
  	if err != nil {
  	logger.Error(err, "Failed to initialize Actions service client for updating a existing runner scale set")
  	return err
  	}

• cleanUpResources propagates it:

  actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go

  Lines 446 to 450 in 03328aa

  	err = r.deleteRunnerScaleSet(ctx, autoscalingRunnerSet, log)
  	if err != nil {
  	log.Error(err, "Failed to delete runner scale set")
  	return false, err
  	}

• Reconcile returns the error without touching the finalizer:

  actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go

  Lines 86 to 89 in 03328aa

  	done, err := r.cleanUpResources(ctx, &autoscalingRunnerSet, log)
  	if err != nil {
  	log.Error(err, "Failed to clean up resources during deletion")
  	return ctrl.Result{}, err

A missing secret is a permanent condition, so the object is requeued forever with controller-runtime's exponential backoff. With many such objects (e.g. a namespace holding thousands of ARS objects being torn down) the workqueue fills with long-delayed items and every other reconcile in the controller slows down.

The deletion path already treats the two neighbouring "nothing to deregister" cases as terminal:

• If the runner-scale-set-id annotation is absent, deleteRunnerScaleSet returns nil, documenting that "manual deletion of the scale set is required":

  actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go

  Lines 742 to 757 in 03328aa

  	scaleSetID, ok := autoscalingRunnerSet.Annotations[runnerScaleSetIDAnnotationKey]
  	if !ok {
  	// Annotation not being present can occur in 3 scenarios
  	// 1. Scale set is never created.
  	// In this case, we don't need to fetch the actions client to delete the scale set that does not exist
  	//
  	// 2. The scale set has been deleted by the controller.
  	// In that case, the controller will clean up annotation because the scale set does not exist anymore.
  	// Removal of the scale set id is also useful because permission cleanup will later lose permission
  	// assigned to it on a GitHub secret, causing actions client from secret to result in permission denied
  	//
  	// 3. Annotation is removed manually.
  	// In this case, the controller will treat this as if the scale set is being removed from the actions service
  	// Then, manual deletion of the scale set is required.
  	return nil
  	}

• If the Actions service reports the scale set as already gone (scaleset.NotFoundError), it logs and continues to release the finalizer (added in Re-register the runner scale set when it is gone from the Actions service #4571):

  actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go

  Lines 775 to 779 in 03328aa

  	case errors.Is(err, scaleset.NotFoundError):
  	// Already gone, so the desired state is met. Returning the error instead would leave the
  	// finalizer in place and the autoscaling runner set stuck in Terminating.
  	logger.Info("Runner scale set is already deleted from the Actions service", "runnerScaleSetId", runnerScaleSetID)
  	case err != nil:

• removeGitHubSecretFinalizer tolerates NotFound/Forbidden on the very same secret ("GitHub secret has already been deleted"):

  actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go

  Lines 1140 to 1142 in 03328aa

  	case kerrors.IsNotFound(err) || kerrors.IsForbidden(err):
  	c.logger.Info("GitHub secret has already been deleted", "name", githubSecretName)
  	return

Only the "secret is NotFound" case blocks forever.

One detail for the fix: the resolver currently discards the underlying Kubernetes error. k8sResolver.appConfig formats it with %q of the key rather than wrapping the error (

actions-runner-controller/controllers/actions.github.com/secretresolver/secret_resolver.go

Lines 221 to 231 in 03328aa

	}
	secret := new(corev1.Secret)
	if err := r.client.Get(
	ctx,
	nsName,
	secret,
	); err != nil {
	return nil, fmt.Errorf("failed to get kubernetes secret: %q", nsName.String())
	}
	return appconfig.FromSecret(secret)

), and GetActionsService wraps with %v (

actions-runner-controller/controllers/actions.github.com/secretresolver/secret_resolver.go

Lines 70 to 79 in 03328aa

	func (sr *SecretResolver) GetActionsService(ctx context.Context, obj object.ActionsGitHubObject) (multiclient.Client, error) {
	resolver, err := sr.resolverForObject(ctx, obj)
	if err != nil {
	return nil, fmt.Errorf("failed to get resolver for object: %v", err)
	}
	appConfig, err := resolver.appConfig(ctx, obj.GitHubConfigSecret())
	if err != nil {
	return nil, fmt.Errorf("failed to resolve app config: %v", err)
	}

). So kerrors.IsNotFound(err) cannot currently see through the chain; both sites need %w (or a typed sentinel) before the controller can distinguish NotFound from transient failures.

Related but distinct: #4093 covers a secret that never existed (the ARS never registered, so the annotation-absent guard above already releases the ARS finalizer; the remaining wedge there is on EphemeralRunner, addressed by #4619). This report is about an ARS that did register and whose secret was removed afterwards, which neither of those covers.

Describe the expected behavior

When githubConfigSecret is definitively NotFound during deletion, the controller should:

1. log at Error level that the runner scale set (with its id) could not be deregistered from the Actions service and must be removed manually, matching the contract already documented for the missing-annotation case;
2. remove the runner-scale-set-id annotation;
3. return nil so the finalizer is released and the object, Helm release and namespace can finish deleting.

Transient errors (API server unavailable, 5xx from the Actions service, Forbidden on the secret while RBAC is being torn down) should keep returning the error and requeueing as today. Optionally emit a Kubernetes Event so the leaked scale set is visible.

Sketch of the change in deleteRunnerScaleSet:

actionsClient, err := r.GetActionsService(ctx, autoscalingRunnerSet)
switch {
case kerrors.IsNotFound(err): // requires %w in secretresolver
    logger.Error(err, "GitHub config secret no longer exists; runner scale set cannot be deregistered and must be deleted manually from the Actions service", "runnerScaleSetId", runnerScaleSetID)
    // fall through to the annotation removal below and return nil
case err != nil:
    logger.Error(err, "Failed to initialize Actions service client for updating a existing runner scale set")
    return err
default:
    // existing DeleteRunnerScaleSet call
}

Test: in autoscalingrunnerset_controller_test.go, create an ARS with the runner-scale-set-id annotation and a valid secret, delete the secret, delete the ARS, and Eventually assert the ARS is gone and the fake client's DeleteRunnerScaleSet was not called; a second case with a non-NotFound secret error asserts the ARS remains Terminating.

I am happy to open a PR with the envtest case if maintainers agree that releasing the finalizer is the right contract here.

Additional Context

# gha-runner-scale-set values (nothing unusual)
githubConfigUrl: https://github.com/<org>
githubConfigSecret: <name of a pre-created secret>
minRunners: 0
maxRunners: 5

Controller Logs

# repeated on every requeue, with growing backoff, until the finalizer is patched away by hand:
INFO  AutoscalingRunnerSet Deleting resources
INFO  AutoscalingRunnerSet Deleting the listener
INFO  AutoscalingRunnerSet Listener is deleted
INFO  AutoscalingRunnerSet deleting ephemeral runner sets
INFO  AutoscalingRunnerSet Ephemeral runner set is deleted
INFO  AutoscalingRunnerSet deleting runner scale set
INFO  AutoscalingRunnerSet Deleting the runner scale set from Actions service
ERROR AutoscalingRunnerSet Failed to initialize Actions service client for updating a existing runner scale set {"error": "failed to resolve app config: failed to get kubernetes secret: \"<ns>/<githubConfigSecret>\""}
ERROR AutoscalingRunnerSet Failed to delete runner scale set
ERROR AutoscalingRunnerSet Failed to clean up resources during deletion
ERROR Reconciler error {"controller": "autoscalingrunnerset", ... , "error": "failed to resolve app config: failed to get kubernetes secret: \"<ns>/<githubConfigSecret>\""}

Runner Pod Logs

N/A - no runner pods exist at this point (minRunners: 0; EphemeralRunnerSet already deleted before the wedge).

Analysis prepared with an AI agent operated by KR-Ravindra, who verified the code paths.

[ankit373]

I picked this up since it was open and unassigned, PR is here: #4688. It takes the approach from the issue: wrap the secret resolver errors with %w, and on NotFound during deletion log, drop the scale set ID annotation and release the finalizer. Other errors still requeue.

[ankit373]

@KR-Ravindra I noticed you offered to open a PR with the envtest case, sorry for missing that. #4688 follows your approach and includes an envtest for it. If you'd rather land your own, let me know and I'll close mine, or I'm happy to fold in anything you'd change.

[KR-Ravindra]

@ankit373 please land yours — no need to close it. You have gone further than what I offered: tagging the Azure Key Vault driver separately so a 403 or a throttled read still requeues is the part I would have got wrong, and pulling removeRunnerScaleSetIDAnnotation out is the right shape.

One note rather than a change request. On the ErrNotFound path the scale set stays registered in the Actions service and the message says it has to be removed by hand, which is the tradeoff the issue accepted. The case that still worries me is a secret that is missing because it was renamed or moved rather than deleted: it is a genuine 404, so the finalizer releases and the registration is orphaned with only a log line behind it. If a maintainer wants that visible, an event on the AutoscalingRunnerSet alongside the log would make it survive log rotation, but I would not hold the PR for it.

I will not send a competing PR. Happy to review revisions if that is useful.