Skip to content

Read ReturnDataView/ExpDataView field shapes on first access - #3526

Open
FFroehlich wants to merge 1 commit into
claude/quirky-margulis-928f12from
claude/rdataview-lazy-dims
Open

FFroehlich wants to merge 1 commit into
claude/quirky-margulis-928f12from
claude/rdataview-lazy-dims

Conversation

@FFroehlich

Copy link
Copy Markdown
Member

Stacked on #3525.

ReturnDataView.__init__ built a dict with the shapes of all array fields. For every view, that read about 35 attributes of the SWIG-wrapped ReturnData and allocated about 40 GC-tracked objects, whether or not any field was read. ExpDataView did the same for its fields.

_field_dimensions is now a class attribute that maps each field to a function of the wrapped object returning the field's shape. It is evaluated when the field is first accessed, and the resulting array is cached as before. Constructing a view is now just a type check plus setting the pointer and an empty cache.

Shapes are now read at the first access of a field rather than at view creation. So an ExpDataView sees changes to the ExpData made before a field is first read; before, this failed with a reshape error. A new test covers this.

Measurements

Setup: Python 3.13, a 2-state model, run_simulations with 9169 ExpDatas and forward sensitivities, 6 calls (55k views). Afterwards llh, sllh and FIM are read from every result.

view construction GC time gen0 collections reading llh, sllh, FIM
#3525 1.36 s 0.44 s 1049 0.32 s
this PR 0.03 s 0.01 s 77 0.31 s

With 12M long-lived GC-tracked objects in the process (standing in for a large PEtab problem), view construction went from 2.29 s to 0.03 s, and GC time from 1.32 s to 0.82 s. One full collection remains, triggered by the earlier allocations of the large heap.

This is the cost that the profile in #3510 (comment) charged to XArrayFactory.__init__ (about 11 s for 18,338 views). See #3525 for why cProfile charged it there.

🤖 Generated with Claude Code

`ReturnDataView.__init__` built a dict with the shapes of all array fields,
reading about 35 attributes of the SWIG-wrapped `ReturnData` and allocating
about 40 GC-tracked objects per view, whether or not a field was read.
For `run_simulations` with thousands of conditions, this dominated the time to
construct the views and triggered frequent garbage collections.
`ExpDataView` did the same for its fields.

`_field_dimensions` is now a class attribute that maps each field to a
function of the wrapped object returning the field's shape. It is evaluated
when the field is first accessed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FFroehlich
FFroehlich marked this pull request as ready for review November 7, 2026 23:12
@FFroehlich
FFroehlich requested a review from a team as a code owner November 7, 2026 23:12
@FFroehlich
FFroehlich added this pull request to stack #3523 November 7, 2026 23:13
@codecov

codecov Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.31%. Comparing base (32e3f4e) to head (c28e07b).

Additional details and impacted files

Impacted file tree graph

@@                      Coverage Diff                       @@
##           claude/quirky-margulis-928f12    #3526   +/-   ##
==============================================================
  Coverage                          80.30%   80.31%           
==============================================================
  Files                                331      331           
  Lines                              23438    23440    +2     
  Branches                            1631     1631           
==============================================================
+ Hits                               18823    18825    +2     
  Misses                              4609     4609           
  Partials                               6        6           
Flag Coverage Δ
cpp 74.81% <100.00%> (+<0.01%) ⬆️
cpp_python 34.64% <100.00%> (+0.01%) ⬆️
petab 46.93% <100.00%> (+0.01%) ⬆️
petab_sciml 16.62% <42.85%> (+<0.01%) ⬆️
petab_sciml_benchmarks 15.21% <42.85%> (+<0.01%) ⬆️
python 72.60% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
python/sdist/amici/sim/sundials/_numpy.py 88.12% <100.00%> (+0.15%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant