Repository navigation
Break the ReturnDataView <-> XArrayFactory reference cycle - #3525
Open
FFroehlich wants to merge 1 commit into
Open
FFroehlich wants to merge 1 commit into
FFroehlich wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## claude/issue-3510-fix-43b6f4 #3525 +/- ##
================================================================
+ Coverage 80.24% 80.30% +0.06%
================================================================
Files 331 331
Lines 23434 23438 +4
Branches 1631 1631
================================================================
+ Hits 18804 18823 +19
+ Misses 4624 4609 -15
Partials 6 6
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
`ReturnDataView.__init__` stored `XArrayFactory(self)` as `self.xr`, and the factory keeps a reference to the view. Because of this cycle, no `ReturnDataView`, nor the C++ `ReturnData` it owns, was freed by reference counting; dropped simulation results stayed alive until the next cyclic garbage collection that reached them. Results from `run_simulations` survive into the oldest generation while they are in use, so with a large heap they accumulated over many calls until the next full collection. `xr` is now a property that creates the factory on access. `__dir__` keeps listing `xr`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FFroehlich
force-pushed
the
claude/quirky-margulis-928f12
branch
from
November 7, 2026 22:54
24f9bf2 to
32e3f4e
Compare
FFroehlich
marked this pull request as ready for review
November 7, 2026 22:58
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3524.
ReturnDataView.__init__storedXArrayFactory(self)asself.xr, and the factory holds a reference back to the view. Because of this cycle, aReturnDataViewand the C++ReturnDatait owns were never freed by reference counting, only by the cyclic garbage collector.This PR makes
xra property that creates the factory on access.ReturnDataView.__dir__still listsxr. A new test checks that a view and itsReturnDataare freed with the GC disabled; it fails without this change.Measurements
Investigated after the profile in #3510 (comment), where 18,338
ReturnDataViews took about 11 s, all charged toXArrayFactory.__init__. Setup: Python 3.13, a 2-state model,run_simulationswith 9169ExpDatas and forward sensitivities, and 4M dicts (12M GC-tracked objects) kept alive to stand in for a large PEtab problem.Memory (the cycle matters). Results that a caller drops after reading
llhstay alive. Whilerun_simulations' result list is in use, they move into the oldest GC generation, so they are freed only at the next full collection. Over 20 calls (200 time points):ReturnDatas alive after a call (max)main)The larger the heap, the rarer full collections are (CPython runs one only when the objects promoted since the last one exceed 25% of the old generation), so more results pile up.
Time (the cycle is not the cause of the 11 s).
ReturnDataViewmakes 40 GC-tracked allocations, mostly the_field_dimensionsdict of lists. Those allocations trigger the collections. Since Python 3.12, a collection runs at the next eval-breaker check, which is the entry ofXArrayFactory.__init__. So cProfile charges the collection to that function, including full collections of the whole heap. With this PR the same collections are charged toSwigPtrView.__init__instead.Not in this PR
Most of the remaining construction cost is the eager
_field_dimensions: about 35 SWIG attribute reads (about 30 µs per view) and 40 tracked allocations. In a quick emulation, computing dimensions lazily on first field access cut construction from 2.44 s to 0.02 s for 55k views, and gen0 collections from 1054 to 76. Done in #3526.🤖 Generated with Claude Code