feat(ci): build the scratchpad image on a GitHub-hosted runner with OIDC (ENG-2000)

[lucas-koontz]

User story

As an anton maintainer who merges to staging and main
I want the scratchpad image to build on a GitHub-hosted runner with an AWS role for its own tier
So that pull requests, staging pushes and releases keep publishing the worker image without a self-hosted runner

Why this matters

When an engineer opens a pull request or merges to staging or main, anton builds the scratchpad image, the sandbox pod image that scratchpad-controller runs. Until now that build ran on a self-hosted runner, although GitHub recommends GitHub-hosted runners for public repositories like this one. Now it runs on ubuntu-latest and pushes with a short-lived AWS role that the job assumes through GitHub's OpenID Connect (OIDC) tokens. Pull request and staging images land in a new dev-tier repository, minds-anton-scratchpad-dev, and only a build from main pushes to minds-anton-scratchpad.

Acceptance criteria

• A pull request from a branch in this repo builds on ubuntu-latest with no GitHub environment, assumes gha-anton-ecr-dev, and pushes development-<sha>, development-head-<head sha>, development and latest to minds-anton-scratchpad-dev.
• A push to staging builds in the staging environment with gha-anton-ecr-dev and moves staging in minds-anton-scratchpad-dev. A push to main builds in the prod environment with gha-anton-ecr-prod and moves production in minds-anton-scratchpad.
• The build job grants itself id-token: write and assumes its role before build-push-ecr runs with builder: local. release.yml and publish-staging.yml grant id-token: write only on the job that calls the build.
• Negative: a fork pull request skips the build, and the run summary says Build skipped: fork pull request.
• Negative: a pull request run that asks for gha-anton-ecr-prod fails at Assume the ECR writer role, so nothing is pushed.
• Negative: a release pipeline that passes anything other than staging or production fails the gate's target step, and the build never starts.
• Negative: a job that can mint an OIDC token, in a workflow an outside account can start, fails tests/test_fork_build_gate.py unless it needs the gate and carries its condition.

How to test

Start as an anton maintainer with uv, actionlint and read access to ECR in account 168681354662, after the steps that Ships with lists before this PR.

1. Run uv run --group dev pytest tests/test_fork_build_gate.py tests/test_pod_image_version.py -v. Expect 31 passed.
2. Run actionlint .github/workflows/scratchpad-dev-build.yml .github/workflows/publish-staging.yml .github/workflows/release.yml. Expect no output and exit code 0.
3. Re-run this PR's Scratchpad image build run. Expect the build job on ubuntu-latest with no environment, and expect Assume the ECR writer role to assume gha-anton-ecr-dev. Then aws ecr describe-images --region us-east-1 --repository-name minds-anton-scratchpad-dev --image-ids imageTag=development-head-<head sha> finds the image.
4. Open a draft pull request from a fork of mindsdb/anton. Expect gate to pass, build to be skipped, and the run summary to say Build skipped: fork pull request. Close that pull request.
5. On a throwaway branch in this repo, open a draft pull request that sets the "" row's role in scratchpad-dev-build.yml to arn:aws:iam::168681354662:role/gha-anton-ecr-prod. Expect Assume the ECR writer role to fail with Not authorized to perform sts:AssumeRoleWithWebIdentity, so nothing is pushed. Close that pull request.
6. Merge this PR to staging. In the publish-staging.yml run, expect scratchpad-image / build to hold the staging environment and assume gha-anton-ecr-dev. Then aws ecr describe-images --region us-east-1 --repository-name minds-anton-scratchpad-dev --image-ids imageTag=staging shows an imagePushedAt after the merge.
7. After the next promotion from staging to main, expect release.yml's scratchpad-image / build to hold the prod environment and assume gha-anton-ecr-prod. Then aws ecr describe-images --region us-east-1 --repository-name minds-anton-scratchpad --image-ids imageTag=production shows an imagePushedAt after the release. The production path runs for the first time here, so this step is the one most likely to find a problem.

Notes for the reviewer

Merge order: this PR merges into staging after mindsdb/terraform#239 is applied and mindsdb/github-actions#71 is on main. The build assumes gha-anton-ecr-dev or gha-anton-ecr-prod and pushes to minds-anton-scratchpad-dev, and mindsdb/terraform#239 creates all three. It also passes builder: local, which build-push-ecr gains in mindsdb/github-actions#71. Until both land, this PR's own image build fails at Assume the ECR writer role, so the PR stays a draft. The unit tests run as usual.

The operator creates this repo's staging and prod environments before the merge. GitHub creates a missing environment the first time a job names it, and that environment has no branch rule. Each environment admits one branch, with no reviewers and no admin bypass. mindsdb/terraform#239 lists the same calls, so run them once.

body='{"can_admins_bypass":false,"deployment_branch_policy":{"protected_branches":false,"custom_branch_policies":true}}'
printf '%s' "$body" | gh api -X PUT repos/mindsdb/anton/environments/staging --input - && gh api -X POST repos/mindsdb/anton/environments/staging/deployment-branch-policies -f name=staging -f type=branch
printf '%s' "$body" | gh api -X PUT repos/mindsdb/anton/environments/prod --input - && gh api -X POST repos/mindsdb/anton/environments/prod/deployment-branch-policies -f name=main -f type=branch
for env in staging prod; do
  gh api "repos/mindsdb/anton/environments/$env" --jq '{name, can_admins_bypass, rules: [.protection_rules[].type]}'
  gh api "repos/mindsdb/anton/environments/$env/deployment-branch-policies" --jq '[.branch_policies[].name]'
done

The loop prints "can_admins_bypass": false and ["branch_policy"] for both environments, then ["staging"] for staging and ["main"] for prod. If can_admins_bypass reads true, the PUT ignored it, so stop before the merge.

Rollback: revert this commit on staging, and on main if it was promoted, until mindsdb/terraform#240 applies. The revert restores the previous build, which pushes every image to minds-anton-scratchpad. Revert the mindsdb/scratchpad-controller and mindsdb/argocd-envs#25 changes with it, or the staging tag they pull stops moving. Once mindsdb/terraform#240 lets only gha-anton-ecr-prod push to minds-anton-scratchpad, the reverted build cannot push there, so roll forward instead.

After the merge, the moving staging tag lives in the dev tier. Each staging push moves minds-anton-scratchpad-dev:staging, and minds-anton-scratchpad:staging keeps its last image. mindsdb/scratchpad-controller and mindsdb/argocd-envs#25 switch their staging references once that tag exists. latest splits the same way: in minds-anton-scratchpad-dev it is the last pull request or staging build, and in minds-anton-scratchpad it moves only on main builds.

One build job serves all three ways in, so a pull request passes an empty environment. environment: ${{ needs.gate.outputs.environment }} evaluates to '' on a pull request. A job whose environment expression comes out empty runs with no environment, as actions/runner#2610 shows, so its token keeps the pull_request subject. The empty value cannot select staging or prod, so if GitHub ever handled it differently, the role step would fail rather than grant more. The alternative was a second copy of the build job.

The gate's target step holds the whole mapping. One case maps each way in to its environment, role and repository, and fails on any other input. The tests execute that shell, so a wrong row fails test_each_way_in_builds_into_its_own_tier.

Deliberate omissions.

• mindsdb/github-actions/setup-env and build-push-ecr stay on @main, as in every anton workflow, so zizmor still reports those two unpinned-uses findings. The third-party actions in the build workflow are pinned to the commits their floating major tags point at today, so nothing changes at run time.
• .github/actionlint.yaml is deleted. It only declared the mdb-dev and mdb-prod labels, which no anton workflow uses now, so actionlint flags any new use of them. Restore the file if you would rather keep it.
• This PR's own run covers only the pull request path. The staging path first runs on the merge to staging, and the production path on the promotion to main.
• The first build of each tier runs with a cold layer cache. build-push-ecr keeps the cache in the GitHub Actions cache, keyed by the image repository name, so it warms after one build.
• The 14 zizmor findings in untouched jobs of release.yml and publish-staging.yml stay as they are.

Verified locally

Check	Result
uv run --group dev pytest tests/test_fork_build_gate.py tests/test_pod_image_version.py -v	31 passed: 19 gate tests and 12 version tests
The five workflow-shape test files (test_verifier_eval_gate, test_dependency_pr_scope, test_fork_build_gate, test_pod_image_version, test_image_smoke)	99 passed
uv run --group dev pytest tests/ -q --ignore=tests/e2e	4421 passed, 60 skipped
Mutation checks on throwaway copies of .github and the gate tests	Each of 28 deliberate breakages fails at least one test: 13 in the build wiring, 8 from code review, and 7 ungated jobs on events an outside account can start. Two control workflows, one on push and one on schedule plus workflow_dispatch, still pass
actionlint 1.7.12 on the three changed workflows	Exit 0, no findings
actionlint 1.7.12 on every workflow	The same 3 shellcheck notes as staging, in evals.yml:145 and verifier-eval.yml:163
zizmor 1.28.0, offline, regular persona, on the three changed workflows	21 findings on staging drop to 16. The build workflow goes from 7 to 2, the two @main refs above. The other 14 sit in untouched jobs and match staging
workflow_graph.py --default-permissions read from mindsdb/github-actions	No permission mismatch. The output matches staging line for line, including its 3 notes about events that start two workflows
gh api tag lookups, 2026-10-03	actions/checkout v4.4.0 and v4, astral-sh/setup-uv v5.4.2 and v5, and aws-actions/configure-aws-credentials v6.3.0 and v6 each resolve to the pinned commit
build-push-ecr at the mindsdb/github-actions#71 head	Tags <env>-<sha>, <env> and latest, plus <env>-head-<head sha> on pull requests, which matches the README
Pre-PR sweep against origin/staging	No whitespace errors, ticket ids in added comments or debug leftovers
Read-only preconditions, 2026-10-03	gha-anton-ecr-dev, gha-anton-ecr-prod and minds-anton-scratchpad-dev do not exist yet. mindsdb/github-actions main has no builder input. anton has no staging or prod environment

Not run: the docs site build (npm run build in docs/). The docs change adds one list item with no links.

Ships with

Part of ENG-2000. This PR carries no Deploys: lines and no deploy label.

Merge order

1. mindsdb/terraform#239 merges, and the operator applies it: the GitHub OIDC providers, the image build and installer upload roles, the dev-tier image repositories and the deployment environment settings. The operator also creates this repo's staging and prod environments.
2. mindsdb/github-actions#71 merges.
3. The operator creates mindsdb/deployer, a new private repository, and the deployer PR opens then. That PR merges once it re-pins argocd-pr-env-deploy to the github-actions merge commit from step 2. The operator also creates the deployer's GitHub App and sets its client ID and private key in the staging and prod environments of cowork and cowork-server.
4. This PR, mindsdb/cowork#1117 and mindsdb/cowork-server#621 merge into staging. mindsdb/minds_python_sdk#90, mindsdb/engine#7, mindsdb/data-vault#4 and mindsdb/hashnode-starter-kit#39 merge into main. These four depend on no other step.
5. Once each dev-tier image repository holds its staging tag, mindsdb/scratchpad-controller#80 and mindsdb/argocd-envs#25 merge.
6. anton, cowork and cowork-server each promote staging to main in their next release.
7. mindsdb/terraform#240 applies once the main builds push through the prod writer roles.
8. mindsdb/Kubernetes-Foundational-Services#166 merges, and the operator upgrades it on both clusters, once cowork-server's change is on main and staging.
9. The operator finishes with one step outside these repositories.

Sibling PRs, in merge order:

• mindsdb/terraform#239: creates the GitHub OIDC providers, the image build and installer upload roles and the dev-tier image repositories, and sets up the deployment environments.
• mindsdb/github-actions#71: build-push-ecr gains builder: local for GitHub-hosted builds, and argocd-pr-env-deploy takes the pull request as inputs and checks its image in the dev tier.
• mindsdb/deployer, a new private repository whose PR opens in step 3: rolls cowork and cowork-server out to staging and prod, and syncs their PR environments.
• mindsdb/cowork#1117: runs every cowork job on GitHub-hosted runners, and rolls staging and prod out through mindsdb/deployer.
• mindsdb/cowork-server#621: runs every job on GitHub-hosted runners, and deploys staging and prod through mindsdb/deployer.
• mindsdb/minds_python_sdk#90: runs the release tests and the PyPI publish on GitHub-hosted runners, and publishes only after the release tests pass.
• mindsdb/engine#7: deletes the jobs that ran on self-hosted runners, and keeps the pull request unit tests on GitHub-hosted runners.
• mindsdb/data-vault#4: deletes the jobs that ran on self-hosted runners, and keeps the pull request unit tests on GitHub-hosted runners.
• mindsdb/hashnode-starter-kit#39: deletes the two workflows that build and deploy the blog.
• mindsdb/scratchpad-controller#80: points the dev and staging scratchpad workers at the dev-tier image.
• mindsdb/argocd-envs#25: points PR environments at the dev-tier images of cowork, cowork-server and the scratchpad worker.
• mindsdb/terraform#240, the second terraform change: sets the prod-tier image repository policies.
• mindsdb/Kubernetes-Foundational-Services#166: updates the self-hosted runner chart on both clusters.