Repository navigation
Move query graph to streaming encoding - #3659
Closed
peterebden wants to merge 2 commits into
Closed
peterebden wants to merge 2 commits into
peterebden wants to merge 2 commits into
Conversation
9013544vtm
reviewed
Nov 11, 2026
| pkgs := slices.SortedFunc(state.Graph.AllPackages(), func(a, b *core.Package) int { | ||
| return cmp.Or(cmp.Compare(a.SubrepoName, b.SubrepoName), cmp.Compare(a.Name, b.Name)) | ||
| }) | ||
| w := tokenWriter{enc: enc} |
There was a problem hiding this comment.
Can this fail? I wonder if it's worth looking at w.err and returning it if not nil, instead of proceeding to range orderedParallel<..>
9013544vtm
approved these changes
Nov 11, 2026
Collaborator
Author
|
I think this is too complicated, and the forthcoming |
peterebden
added a commit
that referenced
this pull request
Nov 12, 2026
Baby version of #3659 (I think #3654 will obviate a lot of the use for this anyway so I'd rather do less here). This is a big win because when emitting formatted output v1 outputs the whole thing into a buffer then reformats that; v2 does it all as it goes. These graphs are usually pretty big so it saves us a lot of memory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've been using this to test changes internally, and quite frequently it gets oom killed on our repo before finishing. This is partly because v1's
MarshalIndentdoes everything in memory then reformats it, so moving toencoding/json/v2helps quite a bit.I've also moved this to stream packages when parsing the whole graph, so we don't have to build every single thing all at once. That is more complex, we could drop that and stick with just the library change if we want something simple.
Outcome:
(user CPU doesn't go down because we GC more often because of not having an enormous graph in memory. we could tune that, but not today).