Skip to content

Move query graph to streaming encoding - #3659

Closed
peterebden wants to merge 2 commits into
thought-machine:masterfrom
peterebden:encoding-json-v2
Closed

peterebden wants to merge 2 commits into
thought-machine:masterfrom
peterebden:encoding-json-v2

Conversation

@peterebden

Copy link
Copy Markdown
Collaborator

I've been using this to test changes internally, and quite frequently it gets oom killed on our repo before finishing. This is partly because v1's MarshalIndent does everything in memory then reformats it, so moving to encoding/json/v2 helps quite a bit.

I've also moved this to stream packages when parsing the whole graph, so we don't have to build every single thing all at once. That is more complex, we could drop that and stick with just the library change if we want something simple.

Outcome:

  │                                          │    wall    │ user CPU │   peak RSS   │
  ├──────────────────────────────────────────┼────────────┼──────────┼──────────────┤
  │ before (v1, whole graph built in memory) │ 27.3–27.9s │ 322–336s │ 15.4–15.9 GB │
  ├──────────────────────────────────────────┼────────────┼──────────┼──────────────┤
  │ after                                    │ 20.9s      │ 340s     │ 2.35 GB      │
  └──────────────────────────────────────────┴────────────┴──────────┴──────────────┘

(user CPU doesn't go down because we GC more often because of not having an enormous graph in memory. we could tune that, but not today).

Comment thread src/query/graph.go
pkgs := slices.SortedFunc(state.Graph.AllPackages(), func(a, b *core.Package) int {
return cmp.Or(cmp.Compare(a.SubrepoName, b.SubrepoName), cmp.Compare(a.Name, b.Name))
})
w := tokenWriter{enc: enc}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this fail? I wonder if it's worth looking at w.err and returning it if not nil, instead of proceeding to range orderedParallel<..>

@peterebden

Copy link
Copy Markdown
Collaborator Author

I think this is too complicated, and the forthcoming query sandbox may obviate even more need for it, so I'm gonna do this more simply.

@peterebden peterebden closed this Nov 12, 2026
peterebden added a commit that referenced this pull request Nov 12, 2026
Baby version of #3659 (I think #3654 will obviate a lot of the use for
this anyway so I'd rather do less here).

This is a big win because when emitting formatted output v1 outputs the
whole thing into a buffer then reformats that; v2 does it all as it
goes. These graphs are usually pretty big so it saves us a lot of
memory.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants