Skip to content

fix(build): resolve content anchors concurrently in parallel builds - #369

Merged
alexey-igrychev merged 1 commit into
mainfrom
fix/build/parallel-anchor-prepass-dk
Sep 27, 2026
Merged

alexey-igrychev merged 1 commit into
mainfrom
fix/build/parallel-anchor-prepass-dk

Conversation

@alexey-igrychev

Copy link
Copy Markdown
Collaborator

Summary

Everything werf does before the first build worker starts — a content-based digest per image, then a storage lookup per digest — walked the image graph one image at a time. On a Deckhouse configuration with 1122 content anchors that phase took 327 s of the cold run, dominated by two repository worktree preparations that waited for each other, and 61 s of a warm one. Both passes now run with the same worker limit the build itself uses.

What

  • With --parallel, anchor digests are computed over the dependency graph with at most --parallel-tasks-limit images at a time; an image is handed out only after every image it depends on has a digest, so the inputs of a digest are unchanged. VERIFIED: the digests of a parallel run equal those of a sequential run on the same graph.
  • With --parallel, the storage lookup for the resolved anchors runs with the same limit, each task with its own stages iterator and its own log buffer; per-image logs are still replayed with their image.
  • Without --parallel, or with a single image, both passes walk the graph in order exactly as before.
  • Log blocks of the prepass are emitted per image by the existing parallel printer, so they no longer appear in graph order when the build is parallel; they are not interleaved.
  • Nothing about what is built changes: same digests, same reuse decisions, same set of skipped images.

Why

The two passes are pure preparation: they read git worktrees and ask the storage what already exists, both dominated by waiting, and the build that follows already runs images concurrently under the same limit. Keeping them sequential meant the slowest repository in the graph blocked every unrelated lookup behind it. The dependency graph, not a flat list, drives the digest pass because a digest includes the digests of the image's dependencies; the scheduler that the parallel build already uses provides exactly that ordering.

Before any image is built, werf computes a content-based digest for every
image and looks each one up in the storage. Both passes walked the graph
one image at a time, so a repository worktree prepared for one image's
digest and a registry lookup for another never overlapped, even though the
build that follows runs images concurrently. Digests are now computed over
the dependency graph with the build's own worker limit, an image starting
only once its dependencies have theirs, and the lookups run with the same
limit, each with its own stages iterator. A build without --parallel keeps
walking both passes in graph order.

Signed-off-by: Aleksei Igrychev <aleksei.igrychev@palark.com>
@alexey-igrychev

Copy link
Copy Markdown
Collaborator Author

Verification

  • Mutation: replacing the graph scheduler with a flat parallel walk -> "calculates the same digests whether or not the build is parallel" failed, because the dependent image was handed out while its dependency was still being computed and ended up without a digest; ignoring --parallel in the worker count -> "bounds concurrency by the build parallelism" failed; sharing one prepass copy across the lookup tasks instead of a copy per task -> the existing "Content anchor cache prepass" specs failed.
  • task test:unit paths="./pkg/build/..." -- -race is clean. It was not at first: the test storage manager stub counted its lookups without a lock, which the concurrent lookup pass exposed; the stub now locks.

Review focus

  • resolveContentAnchor can copy an image from a secondary stages storage on a miss; this makes those copies concurrent. The storage manager is already used concurrently by build workers, but this is the path to look at.
  • Pre-existing, not touched here: with --loglevel=debug, parallel.DoTasksDynamic logs to the shared logger while the printer goroutine renders task output through the same logger, which the race detector flags. It affects the parallel build path equally.

@alexey-igrychev

Copy link
Copy Markdown
Collaborator Author

Upstream twin werf#7931 is green. Measured together with #367 and #368 on the Deckhouse configuration: cold run to the first build worker 13:09 -> 8:22, stage determination 7:24 -> 2:56, all 1122 anchor digests identical to the unmodified binary.

@alexey-igrychev
alexey-igrychev marked this pull request as ready for review September 27, 2026 08:13
@alexey-igrychev
alexey-igrychev merged commit 26d9ae6 into main Sep 27, 2026
12 of 14 checks passed
@alexey-igrychev
alexey-igrychev deleted the fix/build/parallel-anchor-prepass-dk branch September 27, 2026 08:13
alexey-igrychev added a commit that referenced this pull request Sep 28, 2026
🤖 I have created a release *beep* *boop*
---


##
[3.6.1-dk.1](v3.6.0-dk.1...v3.6.1-dk.1)
(2026-09-28)


### Features

* **sbom:** expose the ISPRAS checker options in sbom validate
([#375](#375))
([8fe11f0](8fe11f0))


### Bug Fixes

* **build, git:** reuse one ssh connection for git requests
([#368](#368))
([708a660](708a660))
* **build, git:** reuse one ssh connection for git requests
([werf#7930](https://github.com/deckhouse/delivery-kit/issues/7930))
([6bf8313](6bf8313))
* **build:** resolve content anchors concurrently in parallel builds
([#369](#369))
([26d9ae6](26d9ae6))
* **build:** resolve content anchors concurrently in parallel builds
([werf#7931](https://github.com/deckhouse/delivery-kit/issues/7931))
([39f664f](39f664f))
* **build:** speed up build preparation before workers start
([#367](#367))
([4e39f88](4e39f88))
* **build:** speed up build preparation before workers start
([werf#7928](https://github.com/deckhouse/delivery-kit/issues/7928))
([7a188c2](7a188c2))
* **build:** stop per-digest local image scans before image workers
start ([werf#7929](https://github.com/deckhouse/delivery-kit/issues/7929))
([20d74b6](20d74b6))
* **git:** keep a healthy cached worktree on canceled builds
([#374](#374))
([ed3e5fa](ed3e5fa))
* **git:** keep a healthy cached worktree on canceled builds
([werf#7935](https://github.com/deckhouse/delivery-kit/issues/7935))
([ad187a4](ad187a4))
* **git:** keep local submodule reuse for nested submodule names
([#371](#371))
([0e82710](0e82710))
* **git:** keep local submodule reuse for nested submodule names
([werf#7933](https://github.com/deckhouse/delivery-kit/issues/7933))
([196df14](196df14))
* **git:** rebuild a broken cached worktree instead of failing
([#370](#370))
([967c83d](967c83d))
* **git:** rebuild a broken cached worktree instead of failing
([werf#7932](https://github.com/deckhouse/delivery-kit/issues/7932))
([c89fdb6](c89fdb6))
* **sbom:** keep long sbom validate checker messages on one line
([#376](#376))
([5ec87d4](5ec87d4))
* **storage:** avoid repeated local image scans before builds
([#366](#366))
([88bf86f](88bf86f))
* **storage:** reuse recent tags listings for stage lookups on cache
misses ([#372](#372))
([fad4fea](fad4fea))
* **storage:** reuse recent tags listings for stage lookups on cache
misses ([werf#7934](https://github.com/deckhouse/delivery-kit/issues/7934))
([d3d0c41](d3d0c41))


### Miscellaneous Chores

* force release 3.6.1-dk.1
([30bdf77](30bdf77))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant