Repository navigation
fix: Fix default RNG synchronization for ditribution - #15141
JingyaHuang wants to merge 16 commits into
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…sers into sync-parallel-rng-seed
|
|
||
| @is_tensor_parallel | ||
| @is_context_parallel | ||
| @pytest.mark.skipif(_parallel_device_type() is None, reason="test requires two accelerators") |
There was a problem hiding this comment.
Can we use require_torch_multi_accelerator?
diffusers/tests/testing_utils.py
Line 600 in 84532a0
There was a problem hiding this comment.
There are tests using the decorator elsewhere eg. ContextParallelAttentionBackendsTesterMixin, simply adding neuron/tpu to the decorator enables ~240 existing multi-accelerator tests that don't necessarily pass yet.
But I can add neuron/tpu options to the decorator and enable them only for the rng test rn, maybe later we can include neuron/tpu for all related tests.
afe646e to
a0330c8
Compare
sayakpaul
left a comment
There was a problem hiding this comment.
I opened JingyaHuang#3 to make testing a bit more generalised. LMK your thoughts.
…ests generalize the test case for sync rng for all supported models.
What does this PR do?
Synchronize default RNGs across each parallel group so inference without an explicit generator uses matching random draws on all ranks.
Add tests covering different seeds and matching seeds with different RNG offsets.
Before submitting
self-reviewskill on the diff?documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.