fix(build): resolve content anchors concurrently in parallel builds - #369
Merged
Merged
Conversation
Before any image is built, werf computes a content-based digest for every image and looks each one up in the storage. Both passes walked the graph one image at a time, so a repository worktree prepared for one image's digest and a registry lookup for another never overlapped, even though the build that follows runs images concurrently. Digests are now computed over the dependency graph with the build's own worker limit, an image starting only once its dependencies have theirs, and the lookups run with the same limit, each with its own stages iterator. A build without --parallel keeps walking both passes in graph order. Signed-off-by: Aleksei Igrychev <aleksei.igrychev@palark.com>
Collaborator
Author
Verification
Review focus
|
Collaborator
Author
alexey-igrychev
marked this pull request as ready for review
September 27, 2026 08:13
alexey-igrychev
added a commit
that referenced
this pull request
Sep 28, 2026
🤖 I have created a release *beep* *boop* --- ## [3.6.1-dk.1](v3.6.0-dk.1...v3.6.1-dk.1) (2026-09-28) ### Features * **sbom:** expose the ISPRAS checker options in sbom validate ([#375](#375)) ([8fe11f0](8fe11f0)) ### Bug Fixes * **build, git:** reuse one ssh connection for git requests ([#368](#368)) ([708a660](708a660)) * **build, git:** reuse one ssh connection for git requests ([werf#7930](https://github.com/deckhouse/delivery-kit/issues/7930)) ([6bf8313](6bf8313)) * **build:** resolve content anchors concurrently in parallel builds ([#369](#369)) ([26d9ae6](26d9ae6)) * **build:** resolve content anchors concurrently in parallel builds ([werf#7931](https://github.com/deckhouse/delivery-kit/issues/7931)) ([39f664f](39f664f)) * **build:** speed up build preparation before workers start ([#367](#367)) ([4e39f88](4e39f88)) * **build:** speed up build preparation before workers start ([werf#7928](https://github.com/deckhouse/delivery-kit/issues/7928)) ([7a188c2](7a188c2)) * **build:** stop per-digest local image scans before image workers start ([werf#7929](https://github.com/deckhouse/delivery-kit/issues/7929)) ([20d74b6](20d74b6)) * **git:** keep a healthy cached worktree on canceled builds ([#374](#374)) ([ed3e5fa](ed3e5fa)) * **git:** keep a healthy cached worktree on canceled builds ([werf#7935](https://github.com/deckhouse/delivery-kit/issues/7935)) ([ad187a4](ad187a4)) * **git:** keep local submodule reuse for nested submodule names ([#371](#371)) ([0e82710](0e82710)) * **git:** keep local submodule reuse for nested submodule names ([werf#7933](https://github.com/deckhouse/delivery-kit/issues/7933)) ([196df14](196df14)) * **git:** rebuild a broken cached worktree instead of failing ([#370](#370)) ([967c83d](967c83d)) * **git:** rebuild a broken cached worktree instead of failing ([werf#7932](https://github.com/deckhouse/delivery-kit/issues/7932)) ([c89fdb6](c89fdb6)) * **sbom:** keep long sbom validate checker messages on one line ([#376](#376)) ([5ec87d4](5ec87d4)) * **storage:** avoid repeated local image scans before builds ([#366](#366)) ([88bf86f](88bf86f)) * **storage:** reuse recent tags listings for stage lookups on cache misses ([#372](#372)) ([fad4fea](fad4fea)) * **storage:** reuse recent tags listings for stage lookups on cache misses ([werf#7934](https://github.com/deckhouse/delivery-kit/issues/7934)) ([d3d0c41](d3d0c41)) ### Miscellaneous Chores * force release 3.6.1-dk.1 ([30bdf77](30bdf77)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Everything werf does before the first build worker starts — a content-based digest per image, then a storage lookup per digest — walked the image graph one image at a time. On a Deckhouse configuration with 1122 content anchors that phase took 327 s of the cold run, dominated by two repository worktree preparations that waited for each other, and 61 s of a warm one. Both passes now run with the same worker limit the build itself uses.
What
--parallel, anchor digests are computed over the dependency graph with at most--parallel-tasks-limitimages at a time; an image is handed out only after every image it depends on has a digest, so the inputs of a digest are unchanged. VERIFIED: the digests of a parallel run equal those of a sequential run on the same graph.--parallel, the storage lookup for the resolved anchors runs with the same limit, each task with its own stages iterator and its own log buffer; per-image logs are still replayed with their image.--parallel, or with a single image, both passes walk the graph in order exactly as before.Why
The two passes are pure preparation: they read git worktrees and ask the storage what already exists, both dominated by waiting, and the build that follows already runs images concurrently under the same limit. Keeping them sequential meant the slowest repository in the graph blocked every unrelated lookup behind it. The dependency graph, not a flat list, drives the digest pass because a digest includes the digests of the image's dependencies; the scheduler that the parallel build already uses provides exactly that ordering.