Skip to content

perf(migrations): speed up state replay and snapshots - #2274

Open
tsifrer wants to merge 11 commits into
tortoise:developfrom
tsifrer:faster-migrations
Open

tsifrer wants to merge 11 commits into
tortoise:developfrom
tsifrer:faster-migrations

Conversation

@tsifrer

@tsifrer tsifrer commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Description

This change improves migration performance by avoiding repeated reconstruction of the complete historical model state.

It introduces lightweight migration state snapshots that:

  • Copy mutable ModelState data, including field dictionaries.
  • Reuse already rendered historical model classes until a model changes.
  • Replace changed and relation dependent models non-destructively, preserving the models available through old_state.
  • Preserve the existing old_state and new_state operation API for forwards migrations, rollbacks, custom operations, and SQL collection.

It also changes model reloads so initial querysets are rebuilt only for models that were actually re-rendered. Previously, every operation rebuilt querysets for every model accumulated in the migration state.

Motivation and Context

Migration execution scaled approximately with the number of operations multiplied by the size of the accumulated schema.

Two repeated operations caused most of the overhead:

  1. Real migration and SQL generation operations recreated every historical ORM model when taking an old-state snapshot.
  2. Replaying migration state rebuilt initial querysets for every accumulated model after every operation, even when only one model changed.

This was particularly visible in a large project containing over 2,500 migration operations and approximately 900 models.

Running SQL generation for a late migration previously took approximately three minutes:

tortoise sqlmigrate core 0168_auto_20260908_1042

real 188.59 seconds
user 186.23 seconds
sys    1.40 seconds

With lightweight snapshots alone, the command still took approximately three minutes because state replay, not snapshot creation was the dominant cost for this migration.
After limiting queryset rebuilding to changed models, the same command completed in approximately five seconds:

real 5.05 seconds
user 4.78 seconds
sys  0.16 seconds

How Has This Been Tested?

Testing was performed on macOS arm64 with Python 3.13.14.
The complete migration test directory was run:

pytest tests/migrations -q

results 304 passed, 6 skipped

Regression tests were added to verify that:

  • Snapshots do not re-render unchanged model classes.
  • Mutable ModelState and field dictionaries remain isolated.
  • Removing a field affects only the new state; the field remains available in old_state.
  • Altering models preserves independent old and new rendered models.
  • Foreign key relations point to the appropriate historical model version.
  • Deleted models remain accessible through old_state.
  • Unchanged models do not have their initial querysets rebuilt.
  • Dry-run state replay does not create unnecessary snapshots.

The sqlmigrate command above and the same migrate command was run using an editable installation of this branch to measure the complete CLI path.

Ruff formatting and targeted lint checks also passed.

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added the changelog accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@codspeed

codspeed Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ 21 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 24 untouched benchmarks


Comparing tsifrer:faster-migrations (a2e1a21) with develop (eda0850)

Open in CodSpeed

@waketzheng

Copy link
Copy Markdown
Contributor

A few small things worth polishing before merge — none of them affect the core fix, just documentation and test coverage:

1. snapshot() docstrings should state the immutability invariant.
The correctness of this PR rests on rendered model classes being treated as immutable — state changes replace them via _reload rather than mutating them in place. That assumption isn't written down anywhere, and if someone later mutates a rendered model's _meta directly, snapshots will silently share corrupted state and no existing test will catch it. A sentence in both State.snapshot() and StateApps.snapshot() making the invariant explicit would go a long way.

2. test_state_snapshot_survives_model_deletion is a bit misleading.
The test verifies that the old snapshot can still access a deleted model, but "survives" reads as "the snapshot survives model deletion." Consider renaming to something like test_state_snapshot_keeps_deleted_model_accessible.

3. Consider an explicit no-op snapshot test.
You cover field add/remove/alter and model deletion, but not the simplest case: taking a snapshot of an unchanged state and asserting (a) no ModelState.render calls happen, (b) the snapshot shares rendered classes with the original (is), and (c) the ModelState dicts are independent copies (is not). It's a cheap regression guard against someone accidentally reintroducing re-rendering in snapshot().

4. _build_initial_querysets(reloaded_models) — worth a type hint and docstring.
If the parameter is a set/list, annotating it (e.g. reloaded_models: set[type[Model]]) and documenting "rebuilds querysets only for the given models; rebuilds all when omitted" would make the contract explicit.

5. CHANGELOG could mention the internal change, not just the speedup.
The entry focuses on 188s → 5s, but the PR also changes an internal invariant around state snapshots. A sentence like "Migration state snapshots now share rendered model classes instead of re-rendering them, relying on rendered models being immutable" would help anyone debugging snapshot-related issues later.

Everything else looks good — happy to approve once these are addressed (or if you'd rather handle them in a follow-up, that's fine too).

@tsifrer

tsifrer commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

@waketzheng sorry for the late reply and fixes. All 5 points are addressed in separate commits.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants