Skip to content

Pull stagings by the references staging-check resolves #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alexadereyko merged 1 commit into from
Oct 15, 2026
Merged

Pull stagings by the references staging-check resolves #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 45 additions & 83 deletions .github/workflows/staging-pull.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
name: Staging Pull

# Resolves requested staging packages in an OCI registry (ghcr) to digests, pulls them as
# tarballs, and uploads each set as a deps artifact (tarballs + manifest.json) for a build to
# consume. The mirror of staging-push: push writes children under a path; pull selects a child
# by its descriptor annotations and fetches it by digest.
# Pulls staging packages from an OCI registry (ghcr) and uploads each set as a deps artifact
# (the packages + manifest.json) for a build to consume.
#
# Selection is by annotation globs -- org.opencontainers.image.title (the file name, always
# there) and any com.opendaq.* (there when the staging carried staging-meta.json). A request
# with no match is "not found" and fails; every match is pulled, but two pulled artifacts may
# not share a file name -- they would clobber each other in the dir, so that fails loudly.
# Which packages is the staging-check action's answer: references, path@digest, to the packages
# whose annotations match the request, from the newest index of each package that has any. This
# workflow only downloads them. Two packages in one artifact may not share a file name -- they
# would clobber each other in the dir, so that fails loudly.
#
# One matrix job per entry uploads one artifact, so a build consumes its own deps by name.
on:
Expand All @@ -20,12 +18,12 @@ on:
[
{
"artifact": "staging-<job>", // the artifact name to upload
"packages": [ // one or more children to pull into it
"packages": [ // one or more packages to pull into it
{
"path": "ghcr.io/<org>/path/to/package", // the package to resolve
"tag": "latest", // which index version
"annotations": { // glob filters; every matching child is pulled
"org.opencontainers.image.title": "*-x86_64-linux-gcc-11-release.tar.gz"
"annotations": { // globs; every matching package is pulled
"org.opencontainers.image.revision": "<commit>",
"com.opendaq.custom.platform": "linux"
},
"optional": false // true -> resolving to nothing is a warning
}
Expand All @@ -45,89 +43,53 @@ jobs:
permissions:
packages: read
steps:
- name: Setup ORAS
uses: oras-project/setup-oras@v1
# staging-check of this workflow's own version: a local action is looked for in the
# workspace, so the repository this workflow comes from is checked out there.
- name: Get staging-check
uses: actions/checkout@v6
with:
repository: ${{ job.workflow_repository }}
ref: ${{ job.workflow_sha }}
path: .openDAQ-CI

- name: Resolve
id: resolve
uses: ./.openDAQ-CI/.github/actions/staging-check
with:
packages: ${{ format('[{0}]', toJSON(matrix.entry)) }}

- name: Resolve and pull
- name: Pull
id: pull
shell: python3 {0}
env:
ENTRY: ${{ toJSON(matrix.entry) }}
TOKEN: ${{ github.token }}
ACTOR: ${{ github.actor }}
RESOLVED: ${{ steps.resolve.outputs.resolved }}
run: |
import json, os, subprocess, sys, fnmatch

entry = json.loads(os.environ["ENTRY"])
token = os.environ["TOKEN"]
actor = os.environ["ACTOR"]
packages = entry["packages"]

def run(cmd, err, fatal=True, **kw):
print("+ " + " ".join(cmd), flush=True)
r = subprocess.run(cmd, text=True, capture_output=True, **kw)
if r.stderr: print(r.stderr, end="")
if r.returncode != 0:
if fatal:
sys.exit(f"::error::{err}")
print(f"::warning::{err}")
return r

# login is best-effort: public packages pull anonymously and a fork PR has only a
# read-only token, so warn and carry on rather than failing the pull
hosts = {p["path"].rstrip("/").split("/")[0].lower() for p in packages}
for host in sorted(hosts):
run(["oras", "login", host, "-u", actor, "--password-stdin"],
f"login failed for {host}; continuing (public packages pull anonymously)",
fatal=False, input=token)
import json, os, shutil, subprocess, sys, tempfile

out = os.path.abspath("out")
os.makedirs(out, exist_ok=True)
resolved = []
pulled = {} # file name -> source, to catch two artifacts clobbering each other
for p in packages:
path = p["path"].rstrip("/").lower()
tag = p["tag"]
want = p["annotations"]
optional = p.get("optional", False)

r = run(["oras", "manifest", "fetch", f"{path}:{tag}"],
f"failed to fetch the index for {path}:{tag}", fatal=not optional)
if r.returncode != 0:
print(f"::warning::{path}:{tag} is optional and was not found, skipping")
continue
children = json.loads(r.stdout).get("manifests", [])

matches = [c for c in children
if all(fnmatch.fnmatch(c.get("annotations", {}).get(k, ""), pat)
for k, pat in want.items())]
if not matches:
seen = [c.get("annotations", {}).get("org.opencontainers.image.title", c["digest"])
for c in children]
miss = f"no child of {path}:{tag} matched {want}; available: {seen}"
if optional:
print(f"::warning::{miss}, skipping")
continue
sys.exit(f"::error::{miss}")

for child in matches:
digest = child["digest"]
title = child.get("annotations", {}).get("org.opencontainers.image.title", digest)
if title in pulled:
sys.exit(f"::error::two artifacts share the file name {title!r}: "
f"{pulled[title]} and {path}@{digest}")
pulled[title] = f"{path}@{digest}"
run(["oras", "pull", f"{path}@{digest}", "-o", out],
f"failed to pull {path}@{digest}")
resolved.append({"path": path, "tag": tag, "digest": digest,
"title": title, "annotations": child.get("annotations", {})})
print(f"resolved {path}:{tag} -> {title} ({digest})")
pulled = []
for package in json.loads(os.environ["RESOLVED"])[0]["packages"]:
# each into a dir of its own first, so two packages of one name cannot clobber
with tempfile.TemporaryDirectory() as tmp:
run = subprocess.run(["oras", "pull", package["path"], "-o", tmp, "--format", "json"],
capture_output=True, text=True)
if run.returncode != 0:
sys.exit(f"::error::failed to pull {package['path']}\n{run.stderr}")
for file in json.loads(run.stdout)["files"]:
name = os.path.basename(file["path"])
if os.path.exists(os.path.join(out, name)):
sys.exit(f"::error::two packages share the file name {name!r}")
shutil.move(file["path"], os.path.join(out, name))
pulled.append({"path": package["path"], "title": name, "digest": file["digest"]})
print(f"pulled {package['path']} -> {name}")

json.dump(resolved, open(os.path.join(out, "manifest.json"), "w"), indent=2)
json.dump(pulled, open(os.path.join(out, "manifest.json"), "w"), indent=2)

# nothing resolved -> no artifact, so a consumer sees the dependency as absent
# nothing pulled -> no artifact, so a consumer sees the dependency as absent
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"pulled={len(resolved)}\n")
f.write(f"pulled={len(pulled)}\n")

- name: Upload deps
if: steps.pull.outputs.pulled != '0'
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/test-staging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,13 @@ jobs:
with:
staging-pull: >-
[
{ "artifact": "dep-core-ubuntu-22.04-x86_64-ninja-gcc-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "tag": "run-${{ github.run_id }}-${{ github.run_attempt }}", "annotations": { "com.opendaq.custom.platform": "linux", "com.opendaq.settings.arch": "x86_64" } }] },
{ "artifact": "dep-core-ubuntu-22.04-armv8-ninja-gcc-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "tag": "run-${{ github.run_id }}-${{ github.run_attempt }}", "annotations": { "com.opendaq.custom.platform": "linux", "com.opendaq.settings.arch": "armv8" } }] },
{ "artifact": "dep-core-manylinux_2_28-x86_64-ninja-gcc-14-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "tag": "run-${{ github.run_id }}-${{ github.run_attempt }}", "annotations": { "com.opendaq.custom.platform": "manylinux_2_28", "com.opendaq.settings.arch": "x86_64" } }] },
{ "artifact": "dep-core-windows-2022-x86_64-msvs-v143-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "tag": "run-${{ github.run_id }}-${{ github.run_attempt }}", "annotations": { "com.opendaq.custom.platform": "windows", "com.opendaq.settings.arch": "x86_64" } }] },
{ "artifact": "dep-core-windows-2022-x86-msvs-v143-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "tag": "run-${{ github.run_id }}-${{ github.run_attempt }}", "annotations": { "com.opendaq.custom.platform": "windows", "com.opendaq.settings.arch": "x86" } }] },
{ "artifact": "dep-core-macos-26-x86_64-ninja-appleclang-21-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "tag": "run-${{ github.run_id }}-${{ github.run_attempt }}", "annotations": { "com.opendaq.custom.platform": "macos", "com.opendaq.settings.arch": "x86_64" } }] },
{ "artifact": "dep-core-macos-26-armv8-ninja-appleclang-21-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "tag": "run-${{ github.run_id }}-${{ github.run_attempt }}", "annotations": { "com.opendaq.custom.platform": "macos", "com.opendaq.settings.arch": "armv8" } }] }
{ "artifact": "dep-core-ubuntu-22.04-x86_64-ninja-gcc-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "annotations": { "org.opencontainers.image.revision": "${{ github.sha }}", "com.opendaq.custom.platform": "linux", "com.opendaq.settings.arch": "x86_64" } }] },
{ "artifact": "dep-core-ubuntu-22.04-armv8-ninja-gcc-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "annotations": { "org.opencontainers.image.revision": "${{ github.sha }}", "com.opendaq.custom.platform": "linux", "com.opendaq.settings.arch": "armv8" } }] },
{ "artifact": "dep-core-manylinux_2_28-x86_64-ninja-gcc-14-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "annotations": { "org.opencontainers.image.revision": "${{ github.sha }}", "com.opendaq.custom.platform": "manylinux_2_28", "com.opendaq.settings.arch": "x86_64" } }] },
{ "artifact": "dep-core-windows-2022-x86_64-msvs-v143-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "annotations": { "org.opencontainers.image.revision": "${{ github.sha }}", "com.opendaq.custom.platform": "windows", "com.opendaq.settings.arch": "x86_64" } }] },
{ "artifact": "dep-core-windows-2022-x86-msvs-v143-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "annotations": { "org.opencontainers.image.revision": "${{ github.sha }}", "com.opendaq.custom.platform": "windows", "com.opendaq.settings.arch": "x86" } }] },
{ "artifact": "dep-core-macos-26-x86_64-ninja-appleclang-21-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "annotations": { "org.opencontainers.image.revision": "${{ github.sha }}", "com.opendaq.custom.platform": "macos", "com.opendaq.settings.arch": "x86_64" } }] },
{ "artifact": "dep-core-macos-26-armv8-ninja-appleclang-21-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "annotations": { "org.opencontainers.image.revision": "${{ github.sha }}", "com.opendaq.custom.platform": "macos", "com.opendaq.settings.arch": "armv8" } }] }
]
include-jobs: >-
[
Expand Down Expand Up @@ -142,7 +142,7 @@ jobs:
with:
staging-pull: >-
[
{ "artifact": "dep-optional-ubuntu-22.04-x86_64-ninja-gcc-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "tag": "no-such-staging", "annotations": {}, "optional": true }] }
{ "artifact": "dep-optional-ubuntu-22.04-x86_64-ninja-gcc-release", "packages": [{ "path": "ghcr.io/${{ github.repository_owner }}/test/staging/hello-core", "annotations": { "org.opencontainers.image.revision": "no-such-staging" }, "optional": true }] }
]
include-jobs: >-
[
Expand Down
Loading