Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/sphinx/source/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,22 @@ UniLab 遵循[语义化版本](https://semver.org/)。本共享页面以中英
PyPI 版本变更与未发布变更;发布日期采用 PyPI 上传日期。完整提交历史请参阅
[UniLab 仓库](https://github.com/unilabsim/UniLab)。

## Unreleased

### Added / 新增

- Opt-in sync-free masked autoreset (`masked_autoreset`), persistent step
buffers (`persistent_step_buffers`) and CUDA graph capture of the reset segment
(`autoreset_cuda_graph`) for `TorchEnv` / `ManagerBasedRlEnv`, with the
fixed-shape `ResetSelection` reset selector. All flags default to `False`. The
default and ROCm profiles now pin `unisim-core==2.7.2`, which provides the
MJWarp external graph capture scope.
`TorchEnv` / `ManagerBasedRlEnv` 新增可选的无同步 masked autoreset
(`masked_autoreset`)、持久 step buffer(`persistent_step_buffers`)与 reset
段 CUDA graph 捕获(`autoreset_cuda_graph`),以及固定形状的 `ResetSelection`
reset 选择器。所有开关默认 `False`。默认与 ROCm profile 现在均钉定
`unisim-core==2.7.2`,该版本提供 MJWarp 外部 graph 捕获 scope。

## 1.5.10 (2026-11-11)

### Changed / 变更
Expand Down
7 changes: 7 additions & 0 deletions docs/sphinx/source/en/2-user_guide/1-training/0-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ Run UniLab inside the checked-in Linux NVIDIA image workflow.
Reproduce and operate the scoped single-GPU G1 FlashSAC/MJWarp long soak.
:::

:::{grid-item-card} Sync-free autoreset
:link: 8-sync_free_autoreset
:link-type: doc
Opt-in fixed-shape autoreset, reset CUDA graph, and `ResetSelection` terms.
:::

::::

## When to Drop to `scripts/train_*.py`
Expand Down Expand Up @@ -78,4 +84,5 @@ Day-to-day runs should use the unified CLI. Reach for the low-level
5-resume_and_checkpoints
6-docker
7-tensor_runtime_production
8-sync_free_autoreset
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# Sync-Free Autoreset and Reset CUDA Graph

By default, `TorchEnv` resets done rows through row indices: it reads the done
mask on the host (`any()`), builds `env_ids` with `nonzero`, and launches reset
kernels whose shape depends on the number of reset rows. Each of these steps
synchronizes the host with the GPU.

An opt-in fixed-shape path removes these synchronizations, and on CUDA it can
replay the whole reset segment as one CUDA graph. Every switch defaults to
`False`, so tasks that do not opt in keep the selected-row behavior unchanged.

Implementation: `src/unilab/base/torch_env.py`,
`src/unilab/base/reset_selection.py`, `src/unilab/base/autoreset_graph.py`,
`src/unilab/base/host_transitions.py`. Tests:
`tests/envs/test_fixed_shape_autoreset.py`,
`tests/tasks/test_g1_motion_fixed_shape.py`.

## Requirements

- A Manager-based task whose every reset participant accepts a
`ResetSelection` (see "Opting terms into `ResetSelection`" below).
The G1 motion-tracking terms (`MotionCommand`, `JointPositionAction`, the
motion body rewards and the anchor/body height terminations) already opt in.
- `training.defer_finite_validation=true`, the default in every shipped
algorithm config.
- For the CUDA graph: a CUDA env device, and a backend that declares the
`external_graph_capture` capability and implements
`torch_graph_capture_scope`. MJWarp does both from `unisim-core` 2.7.2, the
pinned version.

## Configuration flags

All flags are fields of `ManagerBasedRlEnvCfg` (or of the base `EnvCfg` for the
backend flag). They live in the `env` section of the owner YAML. They are not
present in the shipped owner YAMLs, so set them with `++` from the command line:

```bash
uv run train --algo flashsac --task g1_motion_tracking --sim mjwarp \
++env.masked_autoreset=true \
++env.persistent_step_buffers=true \
++env.autoreset_cuda_graph=true \
++env.mjwarp_defer_tensor_validation=true
```

| Flag | Default | Effect |
| --- | --- | --- |
| `masked_autoreset` | `False` | Runs the autoreset on every step as a full-width `reset(ResetSelection)` under the device done mask, with no `any()` gate and no `nonzero`. Requires `defer_finite_validation`. |
| `persistent_step_buffers` | `False` | Allocates the observation, reward, terminated and truncated carriers once and updates them with `copy_`. Consumers receive one of `published_buffer_depth` persistent publication slots. |
| `published_buffer_depth` | `2` | Size of the publication ring. A published state stays valid for `published_buffer_depth - 1` further steps. `0` publishes the working carriers themselves, which the next step overwrites. |
| `autoreset_cuda_graph` | `False` | Captures the masked autoreset segment, including the backend reset, into one `torch.cuda.graph` and replays it afterwards. Requires `masked_autoreset`, `persistent_step_buffers`, a CUDA device and a backend with external graph capture. |
| `autoreset_graph_warmup` | `2` | Eager executions before capture, so that lazily allocated buffers exist outside the graph. |
| `autoreset_graph_capture_mode` | `"thread_local"` | `torch.cuda.graph` capture error mode: `"global"`, `"thread_local"` or `"relaxed"`. |
| `mjwarp_defer_tensor_validation` | `False` | Latches MJWarp's fail-closed operand and reset-mask checks into device verdicts instead of reading each one on the host. The env publishes them at its deferred-validation poll. Inside a graph capture MJWarp always defers. The flag also removes the host reads from the eager steps outside the captured segment. UniLab forwards it only when it is `True`. |

Invalid combinations fail when the config is validated: for example,
`autoreset_cuda_graph` without `masked_autoreset` and `persistent_step_buffers`,
or `masked_autoreset` without `defer_finite_validation`.

## Behavior notes

**RNG draw order.** With `masked_autoreset` the reset lifecycle runs on every
step; an all-false mask leaves the state unchanged. Full-width samplers, such as
the `MotionCommand` resample or the reset-event interval times, therefore draw
from the Manager RNG on steps where no row is done. With the same seed,
trajectories differ from the selected-row path. Compare the two modes
statistically, not value by value; MJWarp is not bitwise deterministic in any
case.

**`final_observation`.** It is published on every step and is meaningful only
on done rows.

**Deferred validation polling.** Every fail-closed check in the reset segment is
latched on the device. This covers finite checks, `TorchEnv.require` verdicts,
reset-mask containment and MJWarp/packed-reset verdicts.
`poll_finite_validation` reads all of them together every
`training.finite_validation_interval` steps (32 in the shipped configs). A
failure is therefore raised at the next poll, up to `interval - 1` steps after
the faulty step. Packed-reset failures name the poll step
(`first detected at poll boundary, step N`). To localize a fault, rerun with
`training.finite_validation_interval=1`; the masked path still requires
`training.defer_finite_validation=true`.

**Stale diagnostic counters.** In masked mode the env has no host-side reset row
count or per-phase split. Consequences:

- The per-reset detail keys in `info["timing"]`
(`RESET_DONE_DETAIL_TIMING_KEYS` in `src/unilab/base/backend_timing.py`, for
example `reset_done_count`) are written as `0` on every step. Only
`reset_done_reset_call_ms` is filled. A `0` there does not mean that no row
reset.
- Episode logs (`Episode_Termination/*`, `Episode_Metrics/*` and command reset
metrics) accumulate on the device. They are published into `info["log"]`
only at the deferred-validation poll, and each value covers every episode
that completed since the previous poll. Read them as interval aggregates,
not per-step values.

**Host state under graph replay.** A CUDA graph replays device work only. Python
state that the reset segment changes, such as a pending-metrics flag or a host
lower bound, must be declared with
`unilab.base.host_transitions.record_host_transition`. The graph owner
re-applies these transitions after every replay. Undeclared host state keeps the
value it had at capture time.

## Opting terms into `ResetSelection`

A `ResetSelection` (`src/unilab/base/reset_selection.py`) is a persistent
full-width bool `mask` plus `row_ids = arange(num_envs)`. It is passed through
the existing `env_ids` reset arguments. A participant opts in by declaring
support and handling the selection as a masked full-width write.

- **Class-based terms** (commands, actions, class reward/termination/metric
terms, class observation terms, stateful event terms) set
`supports_reset_selection = True` and accept a `ResetSelection` in `reset`.
- **Function-style terms** (for example reset-mode event functions) use the
`@reset_selection_capable` decorator, and their body must handle a
`ResetSelection` argument.
- Write reset state with the helpers instead of indexing:
- `fill_rows_(tensor, env_ids, value)` writes a constant;
- `assign_rows_(tensor, env_ids, values)` writes a full-width tensor, or
calls a factory `values(n)` that returns `n` rows (`num_envs` rows under a
selection).

Both accept row ids, `None`, a slice or a selection.
- Do not read the mask on the host (`bool(mask.any())`, `.item()`,
`nonzero`) inside `reset`; that reintroduces the synchronization.

```python
from typing import ClassVar

import torch

from unilab.base.reset_selection import ResetSelection, fill_rows_
from unilab.envs.mdp.actions.actions import BaseAction


class MyAction(BaseAction):
# Re-declare on every subclass that overrides reset().
supports_reset_selection: ClassVar[bool] = True

def reset(self, env_ids: torch.Tensor | ResetSelection | None = None) -> None:
super().reset(env_ids)
fill_rows_(self._my_state, env_ids, 0.0)
```

A declaration counts only at or below the class that defines the effective
`reset`. A subclass that overrides `reset` does not inherit its parent's opt-in
and is reported as a blocker until it declares support itself. A
`ResetSelection` is not a tensor, so unconverted code that receives one by
mistake fails at the first indexing operation instead of resetting wrong rows.

Some managers impose extra rules:

- `CurriculumManager` terms cannot run in a sync-free reset, because they
publish host curriculum state.
- The reward, termination, observation and metrics managers check only their
class-based terms. Stateless function terms need no declaration.

### Debugging `reset_selection_blockers`

With `masked_autoreset`, `init_state` calls
`ManagerBasedRlEnv.reset_selection_blockers()`. It collects one entry per
unsupported participant from every manager and raises:

```text
NotImplementedError: masked_autoreset needs every reset participant to accept a
ResetSelection; unsupported: EventManager: <term name> (<function name>); ...
```

Each entry has the form `<Manager>: <term name> (<function or class name>)`. A
manager that cannot take a selection for a structural reason reports a sentence
instead. One example is `TerminationManager: episode counts are read on the
host unless defer_finite_validation is enabled`.

To list blockers without starting a training run, construct the env with the
flag off and call the method:

```python
env.reset_selection_blockers() # [] when the task can enable masked_autoreset
```

Then convert, replace or remove each named term. Reset-randomization events are
common blockers; the G1 motion-tracking FlashSAC owner
(`src/unilab/conf/flashsac/task/g1_motion_tracking/base.yaml`) is a DR-free
baseline for this reason.
4 changes: 2 additions & 2 deletions pyproject.rocm.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ classifiers = [
requires-python = ">=3.10,<3.14"
dependencies = [
"numpy",
"unisim-core==2.7.1",
"unisim-core==2.7.2",
"prettytable>=3.10",
"torch==2.14.0",
"triton-rocm==3.8.0 ; sys_platform == 'linux' and platform_machine == 'x86_64'",
Expand Down Expand Up @@ -68,7 +68,7 @@ mujoco = [
"mujoco~=3.11.0",
"mjbatch-uni~=0.2.5",
]
motrix = ["unisim-core[motrix]==2.7.1"]
motrix = ["unisim-core[motrix]==2.7.2"]
genesis = ["genesis-world==1.3.3"]
# Keep the training-stack extra aligned with pyproject.toml.
unirl = [
Expand Down
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ classifiers = [
requires-python = ">=3.10,<3.14"
dependencies = [
"numpy",
"unisim-core==2.7.1",
"unisim-core==2.7.2",
"prettytable>=3.10",
# A range (not an exact pin) lets ROCm users substitute a ROCm torch
# build; uv.lock pins the tested CUDA builds via tool.uv.sources.
Expand Down Expand Up @@ -102,12 +102,12 @@ newton = [
"pyglet>=2.1.6,<3",
"imgui-bundle>=1.92.0",
]
motrix = ["unisim-core[motrix]==2.7.1"]
motrix = ["unisim-core[motrix]==2.7.2"]
genesis = ["genesis-world==1.3.3"]
# Wheels are CPython 3.12/3.13 Linux x86_64 only; the marker keeps uv lock
# resolvable for the other required-environments.
superdex = [
"unisim-core[superdex]==2.7.1 ; python_version >= '3.12' and sys_platform == 'linux' and platform_machine == 'x86_64'",
"unisim-core[superdex]==2.7.2 ; python_version >= '3.12' and sys_platform == 'linux' and platform_machine == 'x86_64'",
]
# Tensor ray perception is optional and is discovered through UniSim's public
# caster factory. The PyPI distribution is `uniray`; the import namespace is
Expand Down
Loading
Loading