[FFroehlich]

What does this pull request do, and why?

Makes ExperimentManager.create_edatas (and PetabSimulator's sensitivity aggregation) for PEtab v2 problems scale linearly instead of quadratically in the number of experiments and parameters.

ExperimentManager looked up experiments, conditions, measurements, and estimated parameters through petab.v2.Problem, which scans its tables on every lookup (Problem.__getitem__ walks the condition table before the experiment table, get_measurements_for_experiment walks the whole measurement table) and rebuilds x_ids/x_free_ids on every access. With apply_parameters checking each free model parameter for membership in the x_free_ids list, create_edatas was quadratic both in the number of experiments and in the number of parameters.

Since the problem must not be modified after the ExperimentManager is created, lookup tables are now built once in __init__:

• experiment ID → experiment, condition ID → condition, experiment ID → measurements;
• the set of problem parameter IDs, and estimated problem parameter ID → index in x_free_ids;
• model parameter index → x_free_ids index for estimated parameters (this is also the experiment-independent part of plist), and the (problem parameter, model parameter index) pairs for free and fixed model parameters;
• model observable ID → index.

They are used in create_edata, _set_constants, _set_timepoints_and_measurements, apply_parameters, _get_placeholder_mapping, and PetabSimulator._aggregate_sllh/_get_plist_to_problem_par_ix. The latter also called plist.index() per estimated parameter, i.e. was quadratic in the number of parameters for each experiment; it now uses a dict. Smaller things in the same spirit:

• _get_measurements_and_sigmas uses bisect for the timepoint index instead of list.index (quadratic in the number of timepoints of an experiment);
• the replicate counting no longer loops over all observables for each timepoint;
• create_edatas computes the scaled nominal parameters once, not per experiment;
• _aggregate_s2llh finds non-unique indices with a mask instead of in on a NumPy array.

Testing:

• New test_simulate_does_not_scan_petab_problem: makes Problem.__getitem__, get_changes_for_period, and get_measurements_for_experiment raise, then simulates with forward sensitivities (so sllh, s2llh, and sres are aggregated). Fails on main, passes here.
• python/tests/petab_/test_petab_v2.py: all 24 tests pass locally (libpetab main, including the JAX tests).
• tests/petab_test_suite/test_petab_v2_suite.py, SUNDIALS cases (-k False): 41 passed, 6 skipped (the existing PySB "parameters as condition targets" / "compartment size changes" limitations).
• A scratch script compared the implementation on main with this one on a synthetic problem (preequilibration, replicates, observable/noise placeholders with numeric and parameter overrides, experiments without measurements, log/log10 scales, estimated and non-estimated parameters). All ExpData fields (timepoints, measurements, sigmas, fixed parameters incl. preequilibration, t_start, plist, parameters, pscale) after create_edatas and after apply_parameters with random parameters, and the _get_plist_to_problem_par_ix maps, are identical. Timings for 1000 experiments, 1504 free and 1501 fixed model parameters, 1508 estimated problem parameters, 4545 measurements:

	main	this PR
create_edatas()	10.3 s	0.94 s
apply_parameters for all experiments	10.4 s	0.58 s
_get_plist_to_problem_par_ix for all experiments	8.1 s	0.21 s

What remains is linear per experiment (filling each ExpData's parameter and scale vectors). Not run on the Froehlich_CellSystems2018 problem from the issue itself.

Related issues

Fixes #3519

Use of AI tools

Claude Code (Claude Opus 5.5) wrote the implementation, the test, the changelog entry, and this description, and ran the equivalence check and benchmark described above.

Checklist

• I have run the relevant tests locally
• I have added a changelog entry in changelog.d/ (or it is not needed)

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.30%. Comparing base (adfd4b2) to head (19cea70).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3520      +/-   ##
==========================================
+ Coverage   80.26%   80.30%   +0.04%     
==========================================
  Files         331      331              
  Lines       23420    23431      +11     
  Branches     1631     1631              
==========================================
+ Hits        18797    18816      +19     
+ Misses       4617     4609       -8     
  Partials        6        6              

Flag	Coverage Δ
cpp	74.81% <1.96%> (-0.04%)	⬇️
cpp_python	34.64% <1.96%> (-0.02%)	⬇️
petab	46.71% <100.00%> (+0.07%)	⬆️
petab_sciml	16.60% <1.96%> (-0.01%)	⬇️
petab_sciml_benchmarks	15.19% <1.96%> (-0.01%)	⬇️
python	72.59% <94.11%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
python/sdist/amici/sim/sundials/petab/_v2.py	99.05% <100.00%> (+2.65%)	⬆️
python/tests/petab_/test_petab_v2.py	100.00% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.