Skip to content

Commit bfb4cea

Browse files
authored
perf(release): wait for npm registry visibility of all packages concurrently (#13495)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release system publishes ~33 public npm packages per release across the canary, nightly, beta, and stable channels > - `scripts/release.sh` publishes them strictly sequentially: publish one package, poll npm until that version is registry-visible, then start the next > - npm accepts a publish in seconds, but registry packument propagation can lag minutes per package (the CI budget was raised to 30 minutes per package after two aborted releases), so the total publish time is the sum of every package's lag — about two hours on a bad npm day, paid by every channel run including every canary on every master push > - This pull request keeps the publishes sequential but runs all the registry visibility polls concurrently once every publish is accepted > - The benefit is that the wall-clock cost of npm propagation drops from the sum of all packages' lag to the single slowest package's lag, with every existing safety property preserved ## Linked Issues or Issue Description No existing issue. Description follows the enhancement template: **What existing behavior does this improve?** The npm publish step of `scripts/release.sh` (Step 5), used by every release channel. **Subsystem affected** Release tooling (`scripts/release.sh`, `scripts/release-lib.sh`). **Current behavior** Packages publish one at a time, and after each publish the script polls npm until that package's version is visible in the registry packument before publishing the next. With per-package propagation lag of minutes (observed up to ~15 minutes; per-package CI budget is 30 minutes), the full 33-package set takes up to ~2 hours of mostly idle waiting. **Proposed behavior** Phase 1 publishes every package sequentially exactly as today (a rejected publish still aborts the batch immediately with exact attribution). Phase 2 then polls registry visibility for all packages concurrently. Each package keeps its own `NPM_PUBLISH_VERIFY_ATTEMPTS` × `NPM_PUBLISH_VERIFY_DELAY_SECONDS` budget, and any version that never becomes visible still hard-fails the release, now naming every straggler. **Reason and benefit** Total publish wait becomes the slowest single package's lag instead of the sum of all lags — typically minutes instead of hours. This shortens every canary, nightly, beta, and stable run and reduces exposure to job timeouts during npm slowdowns. **Breaking changes** None. Dry-run output is byte-identical in structure, dist-tags are still applied at publish time (`--tag`), the Sigstore TLOG duplicate-recovery path is untouched, and the later dist-tag integrity check (`wait_for_release_registry_state`) is unchanged. ## What Changed - `scripts/release-lib.sh`: replaced `publish_package_to_npm_and_wait` with `wait_for_npm_package_versions`, which takes the package tuple list and polls every package's visibility in background subshells, each reusing the existing `wait_for_npm_package_version` poll (same per-package budget), then reports per-package success or fails naming all stragglers - `scripts/release.sh` Step 5: the publish loop calls `publish_package_to_npm` only (sequential, fail-fast on a rejected publish), followed by one call to `wait_for_npm_package_versions` for the whole set; Step 6's recap line updated to match - `scripts/release-lib.test.mjs`: the registry-visibility and workflow-budget tests now drive the new function (same assertions on `npm view` counts, virtual sleeps, and the fail-closed message, which now names the straggler); a new cross-visibility test proves concurrency — two fake packages that each become visible only after the other has been polled can only converge when polled in parallel, so the test fails if the waits ever serialize again Safety analysis for the ordering change: nothing in the publish loop resolves sibling packages from the registry. `prepare-bundled-package.mjs` bundles and patches from the local workspace tree, and the TLOG duplicate-recovery path only queries the package it just published. The only consumer of the "visible before next publish" invariant was the release script's own final verification, which still runs against the full set. ## Verification - `node --test scripts/release-lib.test.mjs` — 15 tests pass, including the new concurrency proof and the existing 15-minute-20-second budget tolerance test against the new function - `npm run test:release-registry` — full lane, 140 tests pass - `bash -n` on both scripts; `shellcheck` reports no findings beyond the three pre-existing ones on master (verified by comparing counts against `origin/master` copies) - A real-release exercise happens on the next master push: every canary run executes this exact path ## Risks - Low. The failure mode most worth watching is a release where some packages become visible and others never do: previously the run stopped at the first invisible package with later packages unpublished; now all packages are accepted before visibility is enforced, and the run fails naming every straggler. Recovery is identical in both worlds (the next attempt derives a new version number), and the accepted-but-lagging packages carry the correct dist-tag either way. - Publish jobs run the source commit's copy of `release.sh`, so this change takes effect for a given channel only once its source commit includes this merge — promoted nightlies/betas cut from older commits keep the old sequential behavior until their trains catch up. ## Model Used Claude Fable 5 (`claude-fable-5`), extended thinking with tool use (Claude Code). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
1 parent 669bd0e commit bfb4cea

3 files changed

Lines changed: 148 additions & 29 deletions

File tree

scripts/release-lib.sh

Lines changed: 72 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -410,22 +410,80 @@ publish_package_to_npm() {
410410
return 1
411411
}
412412

413-
publish_package_to_npm_and_wait() {
414-
local dist_tag="$1"
415-
local package_name="$2"
416-
local package_version="$3"
417-
local publish_tool="${4:-pnpm}"
418-
local attempts="${5:-12}"
419-
local delay_seconds="${6:-5}"
420-
421-
publish_package_to_npm "$dist_tag" "$package_name" "$package_version" "$publish_tool" || return 1
413+
# Wait for every already-published package to become registry-visible,
414+
# polling all of them concurrently. npm accepts a publish in seconds, but
415+
# packument propagation through the registry CDN can lag minutes per package;
416+
# waiting on each package before publishing the next made the total wait the
417+
# SUM of every package's lag (~2 hours on a bad day for the full set). Every
418+
# publish has already been accepted by the time this runs, so the polls can
419+
# race: the wall-clock cost becomes the single slowest package's lag. Each
420+
# package keeps its own attempts x delay budget, and a package that never
421+
# becomes visible still fails the release, naming every straggler.
422+
#
423+
# $3 is the list_public_package_info tuple list (pkg_dir<TAB>name<TAB>version
424+
# lines); the directory field is ignored.
425+
wait_for_npm_package_versions() {
426+
local attempts="${1:-12}"
427+
local delay_seconds="${2:-5}"
428+
local package_info="$3"
429+
430+
# The polling phase runs in a subshell that owns its own EXIT trap: a
431+
# cancelled or signalled release reaps every in-flight poller and the
432+
# scratch directory instead of leaking one npm poll per package for the
433+
# rest of its budget. The subshell also keeps this trap from clobbering
434+
# the caller's cleanup trap.
435+
(
436+
local status_dir
437+
local pids=()
438+
local specs=()
439+
local failures=()
440+
local index=0
441+
local pkg_name
442+
local pkg_version
443+
local i
444+
445+
status_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-release-visibility.XXXXXX")"
446+
447+
# shellcheck disable=SC2329 # invoked via the trap below
448+
reap_visibility_pollers() {
449+
local pid
450+
for pid in ${pids[@]+"${pids[@]}"}; do
451+
kill "$pid" 2>/dev/null || true
452+
done
453+
rm -rf "$status_dir"
454+
}
455+
trap reap_visibility_pollers EXIT INT TERM
456+
457+
while IFS=$'\t' read -r _pkg_dir pkg_name pkg_version; do
458+
[ -z "$pkg_name" ] && continue
459+
(
460+
if wait_for_npm_package_version "$pkg_name" "$pkg_version" "$attempts" "$delay_seconds"; then
461+
: > "$status_dir/$index.ok"
462+
fi
463+
) &
464+
pids+=("$!")
465+
specs+=("${pkg_name}@${pkg_version}")
466+
index=$((index + 1))
467+
done <<< "$package_info"
468+
469+
if [ "${#pids[@]}" -gt 0 ]; then
470+
for i in "${!pids[@]}"; do
471+
wait "${pids[$i]}" || true
472+
if [ -e "$status_dir/$i.ok" ]; then
473+
release_info "${specs[$i]} is registry-visible"
474+
else
475+
failures+=("${specs[$i]}")
476+
fi
477+
done
478+
fi
422479

423-
if wait_for_npm_package_version "$package_name" "$package_version" "$attempts" "$delay_seconds"; then
424-
return 0
425-
fi
480+
if [ "${#failures[@]}" -gt 0 ]; then
481+
release_warn "npm accepted every publish, but these versions did not become registry-visible: ${failures[*]}"
482+
exit 1
483+
fi
426484

427-
release_warn "npm accepted ${package_name}@${package_version}, but the version did not become registry-visible."
428-
return 1
485+
exit 0
486+
)
429487
}
430488

431489
verify_npm_installable() {

scripts/release-lib.test.mjs

Lines changed: 63 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ function runPublishHelper({
2929
npmVersionExistsAfterChecks = 0,
3030
verifyAttempts = 1,
3131
verifyDelaySeconds = 0,
32+
visibilityPackages = null,
3233
}) {
3334
const fixtureDir = mkdtempSync(join(tmpdir(), "paperclip-release-lib-"));
3435
const binDir = join(fixtureDir, "bin");
@@ -86,6 +87,26 @@ exit 1
8687
set -euo pipefail
8788
printf 'npm %s\\n' "$*" >> "$FAKE_CALL_LOG"
8889
if [ "$1" = "view" ]; then
90+
if [ "\${NPM_VIEW_CROSS:-}" = "true" ]; then
91+
# Cross-visibility mode: a package resolves only after every OTHER
92+
# package has been polled at least once. Mutually dependent visibility
93+
# can only converge when the polls run concurrently.
94+
spec="$2"
95+
name="\${spec%@*}"
96+
safe="$(printf '%s' "$name" | tr '/@' '__')"
97+
touch "$FAKE_STATE_DIR/seen-$safe"
98+
all_seen=true
99+
for other in $CROSS_PACKAGES; do
100+
osafe="$(printf '%s' "$other" | tr '/@' '__')"
101+
[ "$osafe" = "$safe" ] && continue
102+
[ -f "$FAKE_STATE_DIR/seen-$osafe" ] || all_seen=false
103+
done
104+
if [ "$all_seen" = "true" ]; then
105+
echo "1.2.3"
106+
exit 0
107+
fi
108+
exit 1
109+
fi
89110
checks=0
90111
if [ -f "$FAKE_STATE_DIR/view-checks" ]; then
91112
read -r checks < "$FAKE_STATE_DIR/view-checks"
@@ -147,15 +168,30 @@ exec npm "$@"
147168
);
148169

149170
const shellOptions = callerPipefail ? "set -euo pipefail" : "set -eu";
171+
// The cross-visibility concurrency test needs real (fractional-second)
172+
// sleeps so slower-starting sibling polls get a chance to run; every other
173+
// test records virtual waiting so registry-delay tests stay fast/offline.
174+
const sleepStub = visibilityPackages
175+
? ""
176+
: `sleep() { printf 'sleep %s\\n' "$*" >> "$FAKE_CALL_LOG"; }`;
177+
const packageInfo = (
178+
visibilityPackages ?? ["@paperclipai/example"]
179+
)
180+
.map((name) => `packages/example\\t${name}\\t1.2.3`)
181+
.join("\\n");
150182
const script = `
151183
${shellOptions}
152184
source "${repoRoot}/scripts/release-lib.sh"
153-
# Record virtual waiting so registry-delay tests remain fast and offline.
154-
sleep() { printf 'sleep %s\\n' "$*" >> "$FAKE_CALL_LOG"; }
185+
${sleepStub}
155186
${
156-
waitForRegistry
157-
? `publish_package_to_npm_and_wait ${distTag} @paperclipai/example 1.2.3 ${publishTool} "$VERIFY_ATTEMPTS" "$VERIFY_DELAY_SECONDS"`
158-
: `publish_package_to_npm ${distTag} @paperclipai/example 1.2.3 ${publishTool}`
187+
visibilityPackages
188+
? `PACKAGE_INFO="$(printf '${packageInfo}')"
189+
wait_for_npm_package_versions "$VERIFY_ATTEMPTS" "$VERIFY_DELAY_SECONDS" "$PACKAGE_INFO"`
190+
: waitForRegistry
191+
? `publish_package_to_npm ${distTag} @paperclipai/example 1.2.3 ${publishTool}
192+
PACKAGE_INFO="$(printf '${packageInfo}')"
193+
wait_for_npm_package_versions "$VERIFY_ATTEMPTS" "$VERIFY_DELAY_SECONDS" "$PACKAGE_INFO"`
194+
: `publish_package_to_npm ${distTag} @paperclipai/example 1.2.3 ${publishTool}`
159195
}
160196
`;
161197

@@ -174,6 +210,8 @@ ${
174210
NPM_VERSION_EXISTS_AFTER_CHECKS: String(npmVersionExistsAfterChecks),
175211
VERIFY_ATTEMPTS: String(verifyAttempts),
176212
VERIFY_DELAY_SECONDS: String(verifyDelaySeconds),
213+
NPM_VIEW_CROSS: visibilityPackages ? "true" : "false",
214+
CROSS_PACKAGES: (visibilityPackages ?? []).join(" "),
177215
PNPM_MODE: pnpmMode,
178216
REPO_ROOT: fixtureDir,
179217
},
@@ -270,7 +308,7 @@ test("publish_package_to_npm does not retry stable publishes without provenance"
270308
assert.doesNotMatch(result.calls, /--provenance=false/);
271309
});
272310

273-
test("publish_package_to_npm_and_wait confirms registry visibility before returning", () => {
311+
test("wait_for_npm_package_versions confirms registry visibility after a publish", () => {
274312
const result = runPublishHelper({
275313
pnpmMode: "success",
276314
npmVersionExists: true,
@@ -282,12 +320,29 @@ test("publish_package_to_npm_and_wait confirms registry visibility before return
282320
assert.match(result.calls, /^npm view @paperclipai\/example@1\.2\.3 version$/m);
283321
});
284322

285-
test("publish_package_to_npm_and_wait blocks the release when registry visibility lags", () => {
323+
test("wait_for_npm_package_versions blocks the release and names the straggler", () => {
286324
const result = runPublishHelper({ pnpmMode: "success", waitForRegistry: true });
287325

288326
assert.notEqual(result.status, 0);
289327
assert.match(result.calls, /^npm view @paperclipai\/example@1\.2\.3 version$/m);
290-
assert.match(result.output, /did not become registry-visible/);
328+
assert.match(result.output, /did not become registry-visible: @paperclipai\/example@1\.2\.3/);
329+
});
330+
331+
test("wait_for_npm_package_versions polls every package concurrently", () => {
332+
// In cross-visibility mode each fake package resolves only after the OTHER
333+
// package has been polled at least once. Waiting out one package's full
334+
// budget before polling the next can never satisfy the first package;
335+
// only concurrent polling converges.
336+
const result = runPublishHelper({
337+
pnpmMode: "success",
338+
visibilityPackages: ["@paperclipai/alpha", "@paperclipai/beta"],
339+
verifyAttempts: 50,
340+
verifyDelaySeconds: 0.2,
341+
});
342+
343+
assert.equal(result.status, 0, result.output);
344+
assert.match(result.output, /@paperclipai\/alpha@1\.2\.3 is registry-visible/);
345+
assert.match(result.output, /@paperclipai\/beta@1\.2\.3 is registry-visible/);
291346
});
292347

293348
test("the workflow budget tolerates the observed 15-minute 20-second registry delay", () => {

scripts/release.sh

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -342,23 +342,29 @@ else
342342
node "$REPO_ROOT/scripts/prepare-bundled-package.mjs" "$REPO_ROOT/$pkg_dir" "$publish_dir"
343343
cd "$publish_dir"
344344
fi
345-
if ! publish_package_to_npm_and_wait \
345+
if ! publish_package_to_npm \
346346
"$DIST_TAG" \
347347
"$pkg_name" \
348348
"$pkg_version" \
349-
"$publish_tool" \
350-
"$VERIFY_ATTEMPTS" \
351-
"$VERIFY_DELAY_SECONDS"; then
349+
"$publish_tool"; then
352350
if [ "$publish_tool" = "npm" ]; then
353351
rm -rf "$publish_dir"
354352
fi
355-
release_fail "stopping release: npm did not publish and expose ${pkg_name}@${pkg_version}"
353+
release_fail "stopping release: npm did not accept the publish of ${pkg_name}@${pkg_version}"
356354
fi
357355
if [ "$publish_tool" = "npm" ]; then
358356
rm -rf "$publish_dir"
359357
fi
360-
release_info "Published version is registry-visible"
358+
release_info "npm accepted the publish"
361359
done <<< "$VERSIONED_PACKAGE_INFO"
360+
release_info " ✓ npm accepted every package publish under dist-tag $DIST_TAG"
361+
release_info " Waiting for registry visibility of all packages concurrently..."
362+
if ! wait_for_npm_package_versions \
363+
"$VERIFY_ATTEMPTS" \
364+
"$VERIFY_DELAY_SECONDS" \
365+
"$VERSIONED_PACKAGE_INFO"; then
366+
release_fail "stopping release: npm did not expose every published version under dist-tag $DIST_TAG"
367+
fi
362368
release_info " ✓ Published the full package set under dist-tag $DIST_TAG"
363369
fi
364370

@@ -369,7 +375,7 @@ else
369375
release_info "==> Step 6/7: Confirming npm package availability and dist-tag integrity..."
370376
REGISTRY_STATE_VERIFY_ATTEMPTS="${NPM_REGISTRY_STATE_VERIFY_ATTEMPTS:-12}"
371377
REGISTRY_STATE_VERIFY_DELAY_SECONDS="${NPM_REGISTRY_STATE_VERIFY_DELAY_SECONDS:-5}"
372-
release_info " ✓ Every version was registry-visible before the next package publish"
378+
release_info " ✓ Every version was registry-visible before tagging began"
373379

374380
verify_args=(
375381
--channel "$channel"

0 commit comments

Comments
 (0)