enable upstream MONAI to build and run on AMD Instinct GPUs (MI300X/MI325X/MI355X). by nilapate · Pull Request #1 · AMD-Ecosystem/MONAI

nilapate · 2026-09-30T08:11:32Z

Description

Stock upstream MONAI runs on ROCm with three small fixes and a new container:

setup.py: detect ROCM_HOME when CUDA_HOME is absent so HIP extensions build correctly.
loader.py: key the JIT cache on torch.version.hip to prevent all ROCm builds colliding on one slot.
Dockerfile.rocm: new AMD Instinct container. Filters CUDA-only deps to prevent ~1.2 GB of nvidia-* wheels clobbering the ROCm torch.
test_densenet.py: skip bit-exactness check on ROCm — conv algorithm selection differs per graph (measured max abs diff 6.68e-06 on MI300X).

PR Checklist

Non-breaking change.
Commit message clearly explains why this commit is needed.
Commit message includes how this PR/commit was tested.
Commit title includes ticket number.

Commit Message Details

What is this commit about?

Two ROCm bug fixes (broken HIP builds, JIT cache collisions), a new Dockerfile.rocm for AMD Instinct, and a ROCm-specific test skip.

Why is this commit needed?

Users currently maintain out-of-tree forks to work around these issues on AMD Instinct hardware.

How was this PR tested?

MI300X (gfx942), PyTorch 2.13.0+rocm10.0.0:

soumitra-chatterjee-tech · 2026-09-30T09:23:58Z

@@ -0,0 +1,133 @@
+# Copyright (c) MONAI Consortium


If we are adding this file, then copyright should be AMD, not MONAI.

Yes, we need add AMD copy right banner

General Convention is to have of MONAI

soumitra-chatterjee-tech · 2026-09-30T09:25:13Z

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# MONAI on AMD ROCm (AMD Instinct MI300X / MI325X / MI355X).


You can drop the exact enumeration - just mention AMD Instinct.

Suggested change

# MONAI on AMD ROCm (AMD Instinct MI300X / MI325X / MI355X).

# MONAI on AMD ROCm (AMD Instinct GPUs).

soumitra-chatterjee-tech · 2026-09-30T09:26:04Z

+# The default Dockerfile builds on the NVIDIA PyTorch container, which has no ROCm
+# equivalent, so ROCm gets its own recipe rather than a branch inside that one.


Delete this line.

Suggested change

# The default Dockerfile builds on the NVIDIA PyTorch container, which has no ROCm

# equivalent, so ROCm gets its own recipe rather than a branch inside that one.

soumitra-chatterjee-tech · 2026-09-30T09:26:31Z

+# This image installs the upstream `monai` package -- it is not a separate
+# distribution and does not rename the wheel.


Suggested change

# This image installs the upstream `monai` package -- it is not a separate

# distribution and does not rename the wheel.

# This image installs the upstream `monai` package.

soumitra-chatterjee-tech · 2026-09-30T09:28:39Z

+# Select the GPU architecture with --build-arg AMDGPU_TARGETS=...:
+#   gfx942  MI300X, MI325X  (default)
+#   gfx950  MI355X


Suggested change

# Select the GPU architecture with --build-arg AMDGPU_TARGETS=...:

# gfx942 MI300X, MI325X (default)

# gfx950 MI355X

# Select the GPU architecture with '--build-arg AMDGPU_TARGETS=gfx942|gfx950'. The default is gfx942.

soumitra-chatterjee-tech · 2026-09-30T09:34:23Z

+ENV MIOPEN_USER_DB_PATH="/tmp/miopen"
+ENV MIOPEN_CUSTOM_CACHE_DIR="/tmp/miopen"
+RUN mkdir -p /tmp/miopen && chmod 1777 /tmp/miopen


These are risky and prone to clashes if multiple users are on the same node.
Suggest using standard temporary file name generator.

Removed whole block, not needed here.

soumitra-chatterjee-tech · 2026-09-30T09:35:55Z

+# The "all" extra is installed via print_dependencies.py rather than as -e .[all,testing]
+# so that cucim, nvidia-ml-py, and nni can be filtered out:
+#   cucim-cu13 → cupy-cuda13x[ctk] → cuda-toolkit → ~1.2 GB of nvidia-* wheels unusable on ROCm
+#   nvidia-ml-py loads libnvidia-ml.so.1 at import time; absent on ROCm, breaks collection
+#   nni hard-depends on nvidia-ml-py


If possible, avoid the mention of NVIDIA.

we need to filter them out.

soumitra-chatterjee-tech · 2026-09-30T09:36:57Z

+# Whole-slide-image support on ROCm is provided by hipCIM rather than cucim, which is
+# CUDA-only. It is not currently published on PyPI at a usable version, so it is an


Suggested change

# Whole-slide-image support on ROCm is provided by hipCIM rather than cucim, which is

# CUDA-only. It is not currently published on PyPI at a usable version, so it is an

# Whole-slide-image support on ROCm is provided by hipCIM.

# It is not currently published on PyPI at a usable version, so it is an

soumitra-chatterjee-tech · 2026-09-30T09:37:50Z

+# Whole-slide-image support on ROCm is provided by hipCIM rather than cucim, which is
+# CUDA-only. It is not currently published on PyPI at a usable version, so it is an


Why is it not usable? The 26.08 release is already public and should be usable.

soumitra-chatterjee-tech · 2026-09-30T09:38:43Z

+# Whole-slide-image support on ROCm is provided by hipCIM rather than cucim, which is
+# CUDA-only. It is not currently published on PyPI at a usable version, so it is an
+# optional layer: pass --build-arg HIPCIM_INDEX_URL=<index> to enable WSI workloads.
+ARG HIPCIM_INDEX_URL=""


Use the published URL based on the 26.08 release.

vcsajjan · 2026-09-30T09:53:56Z

@@ -0,0 +1,133 @@
+# Copyright (c) MONAI Consortium


Yes, we need add AMD copy right banner

MONAI runs on ROCm unmodified for the most part, since a ROCm build of PyTorch presents itself as `cuda`. Three places assume a CUDA toolkit specifically, and the Docker image has no ROCm equivalent. setup.py: `CUDA_HOME` is None on a ROCm torch, where the toolkit is found via `ROCM_HOME` instead, so BUILD_CUDA evaluated False and the C++/HIP extensions were silently skipped. Accept either. `CUDAExtension` hipifies the .cu sources transparently, so no source changes are needed. monai/_extensions/loader.py: the JIT build cache key included `torch.version.cuda`, which is None on ROCm, so every ROCm toolkit version collided on one cache entry. Fall back to `torch.version.hip`. tests/networks/nets/test_densenet.py: test_pretrain_consistency compares two separately-constructed module graphs holding identical weights for bit-exactness. The backend may select different convolution algorithms per graph, so this was never guaranteed; skip it on ROCm. Dockerfile.rocm: the default Dockerfile builds on the NVIDIA PyTorch container, so ROCm gets its own recipe. It installs the ROCm SDK and a matching PyTorch from AMD's public index, filters the CUDA-only extras (cucim-cu*, nvidia-ml-py, nni), and installs hipCIM, which provides the `cucim` module that MONAI's whole-slide-image paths need on ROCm. Signed-off-by: Patel, Nilaykumar K <NilaykumarKantibhai.Patel@amd.com>

nilapate requested a review from vcsajjan September 30, 2026 08:55

nilapate force-pushed the rocm-minimal-support branch from 0319689 to 8afbc54 Compare September 30, 2026 09:32

soumitra-chatterjee-tech requested changes Sep 30, 2026

View reviewed changes

vcsajjan requested changes Sep 30, 2026

View reviewed changes

Comment thread Dockerfile.rocm

@@ -0,0 +1,133 @@

# Copyright (c) MONAI Consortium

vcsajjan Sep 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we need add AMD copy right banner

nilapate force-pushed the rocm-minimal-support branch from 8afbc54 to 9e06576 Compare September 30, 2026 11:03

nilapate force-pushed the rocm-minimal-support branch from 9e06576 to dfcf915 Compare October 5, 2026 09:13

nilapate closed this Oct 7, 2026

nilapate deleted the rocm-minimal-support branch October 7, 2026 03:35

	# MONAI on AMD ROCm (AMD Instinct MI300X / MI325X / MI355X).
	# MONAI on AMD ROCm (AMD Instinct GPUs).

		# The default Dockerfile builds on the NVIDIA PyTorch container, which has no ROCm
		# equivalent, so ROCm gets its own recipe rather than a branch inside that one.

		# This image installs the upstream `monai` package -- it is not a separate
		# distribution and does not rename the wheel.

		# Whole-slide-image support on ROCm is provided by hipCIM rather than cucim, which is
		# CUDA-only. It is not currently published on PyPI at a usable version, so it is an

Conversation

nilapate commented Sep 30, 2026

Description

PR Checklist

Commit Message Details

What is this commit about?

Why is this commit needed?

How was this PR tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants