You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Long-running loops grew the worker heap linearly with block executions until the process was OOM-killed. Each iteration's output was retained as separate structural copies for the whole run: blockLog.output (a deep copy made only to drop the rarely-present childTraceSpans key) and the loop's allIterationOutputs (a second rebuild by compactSubflowResults of an output that compactBlockOutput had already compacted)
filterHiddenOutputKeys is now copy-on-write: an unchanged plain object/array is returned as-is and a copy starts only at the first changed child. Non-plain prototypes (Date, class instances, null-prototype objects) are still rebuilt, so the output is unchanged
compactEntries reuses unchanged plain subtrees only for compactSubflowResults, whose inputs are already-compacted state outputs. Reusing everywhere was measured and rejected: retaining raw handler (JSON.parse) objects instead of the rebuilt ones costs ~4x heap on wide rows
getJsonByteSize (moved to lib/logs/execution/json-byte-size.ts) tracks ancestors only. Its never-cleared seen-set counted a shared subtree once while JSON.stringify writes it per occurrence, so with shared outputs the 3 MB inline execution-data check would have undercounted
displayOutput for block-complete callbacks reuses blockLog.output via a shallow copy instead of re-walking the output (top level stays separate because streaming later writes token/cost keys onto the log)
Type of Change
Bug fix
Testing
Executor-level benchmark (real DAGExecutor, for loop, mocked handler returning fresh table-shaped output; GC-forced live heap at the last iteration, repeated runs identical):
Equivalence checks against the previous implementations (27 cases: Date, Map/Set, Buffer/typed arrays, class instances with getters, null-prototype, own __proto__ key, sparse arrays, symbol/non-enumerable keys, hidden keys at any depth, UserFile base64 stripping, nested finalBlockLogs): identical JSON and structure
New json-byte-size.test.ts: shared-subtree cases fail on the pre-fix seen-set and pass after
Root bun run test (all workspaces), bun run lint, bun run type-check, bun run check:audits, check-block-registry.ts origin/staging, docs-manifest:check
Checklist
Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
[Medium risk] Optimizes memory use in execution logging by sharing object references.
The PR appears safe to merge; no actionable new issue or outstanding previous finding was identified.
Summary
This PR reduces duplicate retention of loop block outputs by reusing unchanged, already-compacted subtrees. It also updates execution-log byte counting to count shared subtrees at each occurrence and adds regression tests for sharing and size measurement.
The previously requested identity tests are present, and that thread is resolved.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[Block output] --> B[Compacted state output]
B --> C[Copy-on-write log filtering]
B --> D[Loop result aggregation]
C --> E[Block log and callback]
D --> F[Shared subtrees]
E --> G[Execution-log size check]
F --> G
Closing as superseded: every change in this PR landed on staging via #8620 (squash commit b2c5b76), which was built on top of this branch and merged first. All files touched here match staging exactly, apart from #8620's later refinements to json-byte-size.ts and the PII step in logger.ts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Summary
blockLog.output(a deep copy made only to drop the rarely-presentchildTraceSpanskey) and the loop'sallIterationOutputs(a second rebuild bycompactSubflowResultsof an output thatcompactBlockOutputhad already compacted)filterHiddenOutputKeysis now copy-on-write: an unchanged plain object/array is returned as-is and a copy starts only at the first changed child. Non-plain prototypes (Date, class instances, null-prototype objects) are still rebuilt, so the output is unchangedcompactEntriesreuses unchanged plain subtrees only forcompactSubflowResults, whose inputs are already-compacted state outputs. Reusing everywhere was measured and rejected: retaining raw handler (JSON.parse) objects instead of the rebuilt ones costs ~4x heap on wide rowsgetJsonByteSize(moved tolib/logs/execution/json-byte-size.ts) tracks ancestors only. Its never-cleared seen-set counted a shared subtree once whileJSON.stringifywrites it per occurrence, so with shared outputs the 3 MB inline execution-data check would have undercounteddisplayOutputfor block-complete callbacks reusesblockLog.outputvia a shallow copy instead of re-walking the output (top level stays separate because streaming later writes token/cost keys onto the log)Type of Change
Testing
DAGExecutor,forloop, mocked handler returning fresh table-shaped output; GC-forced live heap at the last iteration, repeated runs identical):__proto__key, sparse arrays, symbol/non-enumerable keys, hidden keys at any depth, UserFile base64 stripping, nestedfinalBlockLogs): identical JSON and structurejson-byte-size.test.ts: shared-subtree cases fail on the pre-fix seen-set and pass afterbun run test(all workspaces),bun run lint,bun run type-check,bun run check:audits,check-block-registry.ts origin/staging,docs-manifest:checkChecklist
test-auditauthoring gate)