Skip to content

feat: add CPU and multi-GPU MPI support for larger models - #128

Open
sungho91 wants to merge 42 commits into
masterfrom
feat/mpi-gpu-gospl
Open

sungho91 wants to merge 42 commits into
masterfrom
feat/mpi-gpu-gospl

Conversation

@sungho91

@sungho91 sungho91 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Why

DES currently supports OpenMP and a single GPU. Larger future models need an execution path that can distribute timestep work across CPUs, nodes and GPUs. This adds opt-in MPI execution while keeping mesh management centralized.

What

  • Compact CPU/GPU MPI partitions with neighbor halos and overlap, partitioned by METIS (a one-rank GPU run keeps its whole mesh); one physical GPU per local rank.
  • A CI workflow (mpi-tests.yml) runs the serial-versus-MPI suite; reproducible CPU/GPU functional checks and a reviewer quickstart.

Scope: As requested in review, this PR holds only the MPI backend. GoSPL MPI coupling (independent DES/GoSPL MPI worlds, owned/ghost surface mapping, coupled continuation checkpoints) moves to a follow-up PR with its own continuation tests and "Numerical effect" section. Until then the Makefile rejects usempi=1 usegospl=1 with a message. Serial GoSPL coupling (usegospl=1 without usempi) is unchanged from master.

For the reviewer

  • Feedback wanted: full review of ownership, synchronization and failure handling.
  • Start at: mpi_simulation.cxx (execute, distribute, exchange), then mpi_partition.cxx.
  • Check closely: owned/ghost mappings, device/host buffer lifetime and collective failure cleanup.

Try it

  • Default: MPI is opt-in; build with usempi=1, which needs METIS 5 (METIS_INCLUDE_DIR, METIS_LIB_DIR if not on default paths).
  • Keys that change it: MPI_CXX; add openacc=1 openmp=0 nofma=1 GPU_CC=... for GPUs. Launcher -n selects ranks.
  • Start from: copyable CPU and GPU commands. No GoSPL installation is needed.

Numerical effect

  • Effect: none on serial builds. CPU MPI regressions pass the standard 1e-8 criterion and are bit-exact in the checks below; GPU fixed-mesh checks are measured separately.
  • Reproduce: tests/functional/mpi/run_mpi.py with a parent serial executable as --serial (see the quickstart and verification scope). Speedup depends on workload.
  • After merging current master (cd1042c, including fix: prepare poromechanics with monitoring and diagnostic fixes (1/8) #135; Linux, CPU, HDF5 + MMG): run_mpi.py in 2D and 3D at 1–4 ranks gives 56 bit-exact comparisons against the serial build, including remeshing, restart, hybrid MPI+OpenMP and empty partitions; run_pt_mpi.py (ranks 3 and 4) passes. Monitor CSVs from serial and 2- and 3-rank runs are identical with a moving mesh, with remeshing, and for grid points with rows before and after each remesh. Serial builds are bit-exact against master (test-tiny, test-core, test-rect-tiny, test-3d-tiny, test-3d-equ-tiny). The basic-build, functional-test, OpenMP and C++20 (g++, including the MPI build) steps pass locally.
  • The PT rounding-floor fix that this branch carried is now fix: give the default PT threshold a rounding floor #141; it leaves this diff when fix: give the default PT threshold a rounding floor #141 merges.

New or retired names

  • Added: usempi, MPI_CXX, METIS_INCLUDE_DIR, METIS_LIB_DIR, documented DES_MPI_* controls and MPI executables.
  • Renamed or retired: none.

Checklist

  • Follows CONTRIBUTING.md; builds for ndims=2 and ndims=3: yes, CPU/GPU builds and parent/changed regressions checked.
  • Entry under ## [Unreleased] in CHANGELOG.md: yes.
  • New parameters registered in input.cxx and documented in examples/defaults.cfg: no new cfg parameters.
  • Test under tests/functional/ or benchmarks-cores/: yes, MPI/GPU drivers and runnable fixtures.

AI disclosure

OpenAI Codex implemented the MPI backend, tests/documentation and the original description, and ran the reported pre-split checks. Claude Code split out the GoSPL coupling and plotting helpers, made the review-round changes (METIS by default, shared gather loops, one-statement-per-line MPI sources, the MPI CI workflow, the master merge, the one-rank GPU partition fix) with their checks, and revised this description.

🤖 Generated with Claude Code

sungho91 and others added 15 commits November 6, 2026 16:53
Restart output can start at frame 45. Comparing frame 50 reads its data
correctly, then crashes while printing the step because it indexes a
six-entry step list with the frame number.

Resolve the step through the actual frame list. Both three- and four-rank
50,000-step core-complex continuations report BIT-EXACT at frame 50.

Co-Authored-By: Codex <codex@openai.com>
OpenMP timesteps share a process and cannot distribute mesh storage or
physics across MPI processes. Add separate usempi=1 CPU executables with
RCB ownership, compact owned/ghost meshes and reusable neighbor buffers.

Distribute thermal, constitutive, force, motion, geometry, mass and timestep
kernels while preserving each node's global element summation order.
Rebuild ownership and halos after every remesh. Rank zero retains the global
management copy, MMG, interpolation, marker/phase work and output; this is
an experimental CPU backend with those central scaling limits.

Build 2D/3D MPI and serial/OpenMP with MMG/HDF5. Fresh parent references and
all four make cmp runs are BIT-EXACT (MPI uses 3 ranks in 2D, 4 in 3D).
The 45-check matrix at 1/3/4 ranks also covers hybrid threads, hydraulic
updates, averaged remeshing, restart, empty partitions and error exits.

Co-Authored-By: Codex <codex@openai.com>
Geometric cuts cannot optimize DES's shared-node dependencies. Add an
optional shared-node element graph with the METIS communication-volume
objective, deterministic seed zero and 3% element-balance tolerance.

Select it with usempi=1 usemetis=1 and DES_MPI_PARTITIONER=metis. RCB remains
the default and handles meshes with no more elements than ranks. Ownership
and halos use the existing builders and are rebuilt after every MMG remesh.
The objective is a proxy for real halo traffic, not a speedup guarantee.

Build MPI+METIS and serial/OpenMP in both dimensions with MMG/HDF5. All four
parent make cmp runs are BIT-EXACT. The METIS 45-check matrix at 1/3/4 ranks
also matches every physical dataset, including averaged 2D/3D remeshing.

Co-Authored-By: Codex <codex@openai.com>
Element-to-node assembly waits for ghost contributions before doing any
owned-node work, and inactive management paths collect unused global state.
Separate interior nodes from interface nodes so thermal, force and mass
assembly can post transfers, compute interior rows, then finish interface
rows without changing any node's element summation order.

Keep DES_MPI_OVERLAP=0 as the blocking path. Collect only coordinates and
volumes for mesh-quality checks; skip inactive serpentinization and monitor
collections. Share SoA pack indexing and add per-rank operation, wait and
communication counters, including reductions. Posted work windows do not
by themselves prove asynchronous MPI progress.

Both 2D/3D MPI+METIS and serial/OpenMP builds pass parent make cmp exactly.
The 45-check METIS/overlap matrix also matches every physical dataset.
Earlier pinned 64,177-element, 1,000-step measurements of these changes
reduced three-/four-rank median times from 4.9237/4.5710 s to 4.2908/4.1711 s;
small and remeshing cases do not show the same benefit.

Co-Authored-By: Codex <codex@openai.com>
Short smoke cases do not establish agreement through scientific histories.
Add isolated full/regression runs of existing core-complex/MMG, RSF and
rifting examples, checking complete physical fields, checkpoints and monitor
records with provenance and figures. Label bounded runs explicitly.

Keep the historical Maxwell input/source patch unchanged. Add a separate
current-format constitutive case and compare stress against the analytical
relaxation curve in 2D/3D. Fresh three-rank runs completed 5,000 steps through
five relaxation times: all 51 frames were bit-exact and maximum relative
analytical error was below 8e-8 (acceptance limit 1e-6).

Co-Authored-By: Codex <codex@openai.com>
Unequal CPU placement and one-off timings hide the cost of MPI ownership,
halos and repartitioning. Add verified affinity, warmups and randomized
paired/interleaved rounds for equal OpenMP/MPI budgets, before/after halo
changes, and RCB versus METIS on fixed/adaptive meshes in both dimensions.

Retain commands, executable/source hashes, sample ranges, partition costs,
communication payloads and exact parent-output comparisons. Plot actual
ownership traces and report phase times separately from complete runtime.
The RCB/METIS study contains 120 measured runs and 24 warmups at 3/4 ranks,
all bit-exact; reduced halo bytes did not establish a consistent speedup.
These local shared-memory timings do not predict multi-server performance.

Co-Authored-By: Codex <codex@openai.com>
Parallel remesher CLIs can change labels, metrics and required entities,
so importing one into the DES timestep loop would hide compatibility
failures behind a new mesh. Add pinned external builds, a test-only relink
wrapper that captures the exact existing MMG inputs, and replay/transfer
checks with retained provenance and figures.

Keep production MMG, its metric/freezing rules and field/marker remapping
unchanged. The optional ParMmg2D merge patch remains an explicit prototype;
recorded required-entity and label failures prevent production adoption.
Mark its unified-diff blank context lines as patch data, while checking the
actual added C code with git apply --whitespace=error-all.

Both 2D/3D capture wrappers compile and link against the verified serial
objects. The patch applies cleanly to its pinned ParMmg2D source. Python
entry points compile and their help paths run without starting builds.

Co-Authored-By: Codex <codex@openai.com>
CPU transport correctness cannot establish GPU pointer handling or NVLink
routing. Add standalone contiguous-buffer and shared SoA pack/unpack tests
with asymmetric empty messages, reusable buffers and irregular row maps.
OpenACC host mode is explicit; GPU-required mode rejects missing devices
or CUDA-aware MPI instead of silently using the CPU.

Host checks passed in 2D/3D at 1/2/3/4/6/8 ranks, and ordinary transport
checks passed on three/four ranks. GPU-required mode returned unsupported
status 31 on this workspace. No NVIDIA device/compiler is available here;
full GPU DES integration, multi-GPU correctness and actual NVLink routing
remain unimplemented or unverified.

Co-Authored-By: Codex <codex@openai.com>
Document owned/ghost meshes, central management limits, remesh-time
repartitioning, optional METIS and CPU halo overlap with reproducible
2D/3D correctness, scientific-example and timing commands.

Record paired timings and their sample ranges rather than treating fewer
halo bytes as a runtime guarantee. Keep RCB as the measured laptop default.
Distinguish complete scientific histories from bounded regressions, and
host transport checks from the unintegrated GPU solver/NVLink path.
Explain the parallel-remesher compatibility failures while retaining the
existing production MMG policy and remapping.

Co-Authored-By: Codex <codex@openai.com>
The MPI branch predates upstream #122's CLI11 parser and #125's C++17
baseline/C++20 compatibility checks. Integrate current master without
rewriting the nine published feature commits.

Preserve MPI wrapper compiler introspection when selecting GNU, Clang and
NVHPC flags. Keep C++17 as the default, update standalone transport checks
to that floor with an explicit C++20 option, and remove obsolete Boost
arguments/link flags from capture tooling and active build instructions.
Retain old timing reports as historical measurements of their stated base.

2D/3D serial/OpenMP and MPI+METIS build in both standards. Fresh master
references and all eight make cmp runs are BIT-EXACT. C++17 RCB passes
78 matrix checks; C++20 METIS passes 45. Both standards pass all 62 parser
cases in serial and MPI, 12 host halo checks and full 51-frame 2D/3D Maxwell
comparisons. C++20 reports no first-party deprecation warnings.

Co-Authored-By: Codex <codex@openai.com>
The upstream compatibility jobs compile only the serial code paths, so an
MPI wrapper-selection or standalone language-floor regression can escape.
Add 2D/3D MPI+METIS/HDF5 builds at the C++17 default and with the strict
C++20 checker. Trigger the workflow for MPI helper changes too.

Exercise explicit OpenACC host-mode halo checks in both standards at
1/2/3/4 ranks, oversubscribing small CI workers. Retain the build/check logs
and state clearly that these tests do not verify GPU execution. Local
replicas pass all four builds and all 16 host checks using the local METIS
prefix; CI installs the corresponding system development package.

Co-Authored-By: Codex <codex@openai.com>
The PT refactor in PR #106 replaces the legacy iteration loop with physical
trial/commit solves. Dispatching its constituent kernels alone leaves the
global trial workspace stale and provides no collective stopping decision.

Merge the upstream solver and execute both PT methods, including initial
body-force equilibrium, on compact owned/ghost meshes. Exchange trial rates
and factors, reduce owner-only residuals and ADR sums, and retain accepted
PT diagnostics and running-scale state through output, remeshing and restart.
Capture absolute element forces before boundary assembly reuses its scratch;
the former cache read can otherwise make roundoff thresholds partition dependent.

Both dimensions build as C++17/C++20 with MMG, HDF5 and MPI/METIS. Fresh
upstream make set references give eight BIT-EXACT make cmp comparisons;
123 general MPI comparisons are exact. Scoped PT checks cover both methods,
RCB/METIS, 1/3/4/8 ranks, remeshing, restart and expected failures, with
explicit field tolerances. The focused four-thread force/material/rollback
suites pass 6,852 assertions in 2D and 10,185 in 3D.

Co-Authored-By: Codex <codex@openai.com>
The generic MPI suite cannot check PT acceptance, material rollback or
whether a low residual describes the accepted physical state. Reuse the
upstream independent force/RMS, held-load/reload, Maxwell creep and exact
restart checks on serial and MPI executables, then compare every physical
HDF5 dataset with an explicit tolerance and measured byte equality.

Cover both PT methods, initial hydrostatics, two MMG boundary options,
running-scale history, dynamic timesteps, oblique supports and collective
iteration-limit failure. Retain inputs, commands, executable hashes and
worst-field error evidence. Plot the checked strain histories against the
analytical creep reference. Extend C++20 compatibility CI with PT checks.

Local C++17 RCB and C++20 METIS runs pass 192 field comparisons and 24
expected-failure checks, including ranks 1/3/4/8 and twelve option-0 hybrid
checks with two threads per rank. Each run's own restart remains exact;
130 serial/MPI comparisons are byte-exact. The maximum normalized Maxwell
reference error is 2.171e-6. Document the 50,000-iteration MMG fixture cap
and force/pressure scales used for cancellation-sensitive diagnostics.

Co-Authored-By: Codex <codex@openai.com>
Large calculations need an execution path beyond one GPU or one shared-memory OpenMP process. Extend the compact CPU MPI backend with per-rank GPU kernels and pinned host halos, and couple DES to a persistent GoSPL MPI world with independent process counts and continuation state.

Retain upstream C++17/CLI11 and PT mechanics. Current-source 2D/3D parent regressions, 34 CPU cases, four C++20 host halo cases, four NVHPC 25.7 GPU cases, coupled continuation and two-node equivalence checks pass. Historical hardware and cost matrices retain separate provenance.

Add a reviewer quickstart with tested CPU/GPU commands, an eight-step fresh coupling fixture, allocation sizes, expected results and a remeshing restart example.

Co-Authored-By: Codex <noreply@openai.com>
Retain upstream GoSPL Python/OpenMP library discovery and MMG configuration fixes. Resolve the README by keeping the current environment setup together with the MPI launch and reviewer examples. Linux numerical sources and build flags are unchanged.

Co-Authored-By: Codex <noreply@openai.com>
Comment thread gospl_driver/run_coupled.py Fixed
Comment thread tests/functional/mpi/run_pt_mpi.py Fixed
Comment thread gospl_driver/mpi_worker.py Fixed
Comment thread gospl_driver/run_coupled.py Fixed
Comment thread gospl_driver/mpi_worker.py Fixed
sungho91 and others added 4 commits November 10, 2026 09:44
Widen halo allocation products before multiplication, remove five unused imports, and use one checkpoint import style. Explain the expected cleanup races and handle worker failures and interrupts explicitly so a failing rank cannot strand its peers.

C++17/C++20 host halo checks pass in both dimensions at one and two ranks. Two-rank protocol rejection, KeyboardInterrupt/SystemExit cleanup and 14 coupled continuation comparisons pass.

Co-Authored-By: Codex <noreply@openai.com>
The MPI CI job built without MMG and crashed during its first compact partition: init_elem_size_n and refined_elem_size_n were allocated but empty, so selecting them by node ID dereferenced null storage. Serialize, select and exchange these optional fields only in MMG builds, where they are populated. The MMG-enabled field list is unchanged.

Reproduced the three-rank crash before the fix. Non-MMG parent make set / changed make cmp pass bit-exact in 2D and 3D. All 40 PT checks pass with the existing tolerances across both dimensions, relaxation methods 0/1 and ranks 3/4, including restart and collective failures.

Co-Authored-By: Codex <noreply@openai.com>
PR #128 is five commits behind master after the Docker images update in
PR #119. Merge those changes without rewriting the existing MPI history,
retaining the upstream MMG cache-path guard and Docker build/CI/docs.

Native arm64 macOS builds pass in 2D/3D with OpenMP, HDF5 and MMG, for
serial and MPI executables. Master make set / changed make cmp are
bit-exact for both dimensions and both executables. All 34 existing MPI
checks are bit-exact against current master; all 20 PT checks pass at
the existing rtol=1e-8/atol=1e-12, including four collective failures.

Co-Authored-By: Codex <noreply@openai.com>
The MPI branch has no native macOS verification record after the current
master merge. Document the Apple Silicon toolchain, CPU dependency setup,
existing reproduction commands and bounded validation results.

Both dimensions and serial/MPI builds pass. Four master regression
comparisons and 34 MPI matrix comparisons are bit-exact. All 20 PT
checks pass without relaxing tolerances; GPU and GoSPL were not tested.

Co-Authored-By: Codex <noreply@openai.com>
@sungho91
sungho91 requested review from chaseshyu, echoi and tan2 November 10, 2026 01:30
sungho91 and others added 6 commits November 10, 2026 10:48
The scheduler guard tests assume /bin/true exists. Native macOS keeps
true elsewhere, so two checks fail before reaching their intended guard.
Use the running Python executable as the existing file/hash fixture.
Every invocation is rejected before execution.

All eight scheduler guard checks now pass on macOS without contacting a
scheduler or launching MPI. Solver and launcher behavior are unchanged.

Co-Authored-By: Codex <noreply@openai.com>
The MPI PR includes dated execution diaries and standalone ParMmg
experiments outside its production backend and CI. They add internal host
details, exploratory patches and duplicate measurements to user/test docs.

Retire two dated reports and seven isolated remesher/capture tools. Keep
their originals with SHA-256 records in an ignored local archive. Retain
the reusable CPU/GPU/GoSPL tests, commands, acceptance criteria and bounded
verification conclusions. Remove obsolete links, the patch-only whitespace
exception and duplicate run history. Ignore dated reports and macOS dSYM
bundles. No solver arithmetic or executable driver logic changes.

This removes 1,749 net lines. All 33 remaining local documentation links
resolve; 55 Python files parse without retired-module dependencies. Both
MPI dimensions build on macOS and the tiny two-rank regressions remain
bit-exact against the recorded master references.

Co-Authored-By: Codex <noreply@openai.com>
The MPI feature still includes nine workload-specific timing/plotting
tools and a legacy transport harness. The latter duplicates the shared
halo test's irregular/empty-message coverage. None is called by the
production solver or CI.

Archive these ten files locally with verified hashes. Remove the old
benchmark-only options from run_mpi.py, its unused statistics import and
the transport wrapper used only by the retired harness. Preserve every
default physical check and all six shared helper functions unchanged,
verified by AST comparison. Keep the real GoSPL audit, scientific examples,
partition tracing, MPI/PT/GPU/restart drivers and CI halo test.

Remove the corresponding commands and performance-study claims from the
docs/changelog. The change removes 1,290 net lines. Both MPI dimensions
build; all 34 CPU matrix checks are bit-exact, four OpenACC host halo
checks and eight scheduler guard checks pass on macOS. Local document
links resolve and retained Python modules do not import archived tools.

Co-Authored-By: Codex <noreply@openai.com>
The MPI branch lacks the managed-memory configuration-reader fixes
merged into master in #126. Merge current master
without rewriting the existing feature history, preserving its nvc++
getline workaround and deferred GCC 8 header check.

Native macOS serial and MPI builds pass in both dimensions with OpenMP,
HDF5 and MMG. Four current-master make set / merged make cmp comparisons
are bit-exact, all 34 MPI matrix comparisons are bit-exact, and all 62
existing bad-input/configuration acceptance cases pass. GPU/NVHPC runtime
checks are outside this CPU-only repeat.

Co-Authored-By: Codex <noreply@openai.com>
Identical host/device halo loops and per-step material scans repeat work on unchanged compact state. Redistribution also invalidates an already-current root mesh, so later collections transfer it again.

Share the pack/unpack indexing and refresh composition caches at redistribution. Reuse fresh full-root fields for boundary and sample queries, and remove the extra post-step GoSPL vector copy.

The 12-step, six-remesh CPU/GPU coupling cases reduce root collection payloads from 2019488 to 1926072 bytes each. All 25 saved physical files per backend remain bit-exact against the pre-cleanup trajectory. Both-dimensional parent comparisons, 34 CPU cases, 40 PT checks, 40 GPU file comparisons and coupled restarts pass.

Co-Authored-By: Codex <noreply@openai.com>
The root management loop rescans composition after distribution has refreshed both caches. PT repeats an accepted-velocity exchange, RSF reduces its diagnostics separately, and GoSPL ranks seed the entire surface while checkpoint fields are serialized twice.

Reuse the attached-backend query, vector maximum and native PETSc DM mapping. Serialize the complete snapshot once, preserving initialization/isostasy, restored-velocity synchronization and collective checkpoint rejection.

In a four-step three-rank RSF case, root cache entries fall from 12 to 4 and Allreduce calls from 40 to 36 per rank. Two-rank GoSPL seed queries fall from 260 to 169 with correct owned/ghost values. The checkpoint fixture preserves wire bytes and rejects unsupported state.

Both-dimensional parent comparisons, 34 CPU cases, 40 PT checks and 40 standard GPU file comparisons pass. Six additional GPU PT/RSF cases preserve all 60 files bit-exact; actual PT GPU launches are recorded. CPU/GPU coupled restarts pass 14 comparisons each and retain 25 bit-exact continuous files per backend.

Co-Authored-By: Codex <noreply@openai.com>
sungho91 and others added 2 commits November 11, 2026 07:03
Compact ranks copy individual marker histories although their kernels use
only per-element material counts. MPI averaging also duplicates Output's
algorithm, and root serializes its own partition before reading it back.

Keep marker records on the management mesh and share Output averaging
with compact ranks. Reuse collection indices, scalar reduction helpers
and checkpoint buffers; remove the unreachable friction command.

In the 40-step 3D averaged-remesh case, worker state traffic falls from
3282168 to 1720230 bytes with identical fields. The hydration case falls
from 11601401 to 4372254 bytes. Both dimensions pass parent make set/cmp,
34 CPU checks, 52 PT checks and 120 GPU file comparisons. H100 coupled
continuation passes 14 comparisons; the qsub two-host run passes nine
byte-exact comparisons with four allocated slots.

Co-Authored-By: Codex <noreply@openai.com>
Clang rejects the global NDIMS/NSTR bounds in collapsed averaging loops
under default(none), while GCC accepts them. Capture local component
counts explicitly and keep the OpenACC clauses in step.

Clang 14 C++17 and Clang 18/21 C++20 pass in both dimensions. The existing
C++20 CI script also passes locally with GCC 13. Parent make set/cmp and
80 CPU plus 20 GPU averaging comparisons remain byte-exact, including
four remeshes per run. NVHPC 25.7 builds both dimensions for SM86/SM90.

Co-Authored-By: Codex <noreply@openai.com>
@tan2

tan2 commented Nov 11, 2026

Copy link
Copy Markdown
Member

I will review this PR and test its parallel scaling on our infiniteband-equipped cluster.

@tan2

tan2 commented Nov 13, 2026

Copy link
Copy Markdown
Member

Could we leave the GoSPL MPI coupling for the next PR and keep this one to the MPI backend?

The coupling is about 1.8k of the 7.4k added lines: the 18 files under gospl_driver/, tests/functional/gospl/ and doc/gospl-mpi-coupling.md, plus the GoSPL hunks in dynearthsol.cxx (continuation restart), output.cxx (.gospl checkpoint sidecar), bc.cxx and mpi_simulation.cxx. Apart from one line in mpi_simulation.cxx (velocity in the surface exchange for surface_process_option = 11), those hunks are already behind HAS_GOSPL_CPP_INTERFACE, so the split should be mechanical. Until the follow-up lands, the Makefile can reject usempi=1 usegospl=1 with a clear message.

Reasons:

  • This PR is easier to review, and I can test its scaling on our cluster without a GoSPL installation.
  • The coupling changes serial results on its own: the PETSc seed ordering, and YEAR2SEC (365.2422 d) replacing SEC_PER_YR = 3.1536e7 (365 d), which shifts GoSPL velocities and the coupling clock by 0.077 %. Those deserve their own "Numerical effect" section.
  • The coupling can then be reviewed with its own continuation tests.

Keep this PR to the CPU/GPU MPI backend so it can be reviewed and
scaled without a GoSPL installation. The coupling returns in its own PR
with its continuation tests and numerical-effect notes.

Removed here:
- gospl_driver/{checkpoint,des_gospl_client,mpi_worker,
  parallel_surface,run_coupled}.py, tests/functional/gospl/ and
  doc/gospl-mpi-coupling.md.
- Coupled continuation restart (dynearthsol.cxx), the .gospl checkpoint
  sidecar (output.cxx) and its checkpoint-reference I/O (binaryio).
- GoSPL hooks in mpi_simulation.cxx: velocity in the surface exchange
  and idle-step clock advance for surface_process_option = 11.

Restored to master:
- use_gospl() in bc.cxx and gospl_driver/gospl-driver.{cxx,hpp},
  README.md. This returns the serial coupling to SEC_PER_YR = 3.1536e7
  and the previous PETSc seed ordering, so serial GoSPL results are
  unchanged by this PR.

The Makefile now rejects usempi=1 usegospl=1 with a message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tan2

tan2 commented Nov 13, 2026

Copy link
Copy Markdown
Member

The absolute_force_rms change (pseudo_transient.cxx) also fixes a serial bug, but it is not listed under "Numerical effect".

On master, absolute_force_rms rebuilds each node's sum of absolute element forces from var.tmp_result after update_force returns. By then, apply_stress_bcs has reused rows 0..nbfacets-1 of tmp_result for facet tractions (bc.cxx, ElemCacheAccessor tmp = (*var.tmp_result)[n]). That happens whenever gravity is on and there is a Winkler bottom, a water-loaded top, or a side wall with vbc 0, 2 or 4. With PT_use_running_scale = yes, the roundoff floor 1000·ε·F_abs is therefore computed partly from facet tractions instead of element forces. This PR records the sums during assembly (element_force_magnitude), before the rows are reused, which fixes it in serial runs too. The new check in tests/residual-assembly-test ("element magnitude survives boundary scratch") covers it.

It changes results only when that floor sets the PT threshold, i.e. when it exceeds 1e-6 × the running residual reference. Could you list it under "Numerical effect", or move it into its own small PR?

Minor, same code: element_force_magnitude holds per-node sums (indexed by node), so node_force_magnitude would describe it better.

The MPI pass/fail drivers compare fields and write JSON reports; figures
are not part of the solver PR. Remove plot_partitions.py, the --plot
options of reproduce_examples.py and run_pt_mpi.py, and the matching
documentation. Partition traces (DES_MPI_PARTITION_TRACE) and the
[mpi][partition] log rows are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sungho91

sungho91 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

@tan2, Done in b55ed6f: this PR now holds only the MPI backend. The GoSPL MPI coupling (driver scripts, tests/functional/gospl/, the coupling guide, continuation restart, the .gospl checkpoint sidecar and the surface-exchange hooks) is removed. use_gospl() and gospl_driver/ are back to master, so SEC_PER_YR and the old PETSc seed order are restored and serial GoSPL results are unchanged. The Makefile rejects usempi=1 usegospl=1 with a message.

b6d504e also drops the optional plotting helpers (plot_partitions.py and the --plot options), so the MPI checks need only NumPy and h5py.

Post-split checks: run_mpi.py in 2D/3D at 1/2/4 ranks gives 45 bit-exact comparisons against the 229a48c serial build. Serial output is identical to the parent apart from walltime_sec. I've updated the description.

GoSPL MPI coupling will follow in a separate PR when requested, with its continuation tests and a "Numerical effect" section for the YEAR2SEC and seed-order changes.

🤖 Generated with Claude Code

Resolve parameters.hpp: keep the MPI branch's nullptr member defaults
together with master's ownership comments, and drop the remesh
transients that master moved into RemeshScratch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@tan2 tan2 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a big step. I ran the checks below locally (Linux, g++ 14, Open MPI 5.0.7) at 772b5e8, before the GoSPL split and the master merge, and re-checked items 1–4 against 2e59200.

What I checked

  • Non-MPI builds (2D/3D × Triangle/TetGen/MMG, -O2): all 23 regression cases (including plane strain and 3D elasto-visco-plastic, with forced remeshing) are bit-exact with its parent 6195cc2. ASan/LSan builds are clean.
  • MPI builds compile without new warnings. tests/functional/mpi/run_mpi.py --ranks 1 2 passes (34 bit-exact checks, plus the root-failure and --help exit checks).
  • My own check, 3 ranks against serial on cases the driver does not use (test-tiny-forced, test-rift, test-3d-tiny-forced with MMG, test-3d-remesh): every field in every save and checkpoint file is bit-exact.

The design reads well: root-owned mesh management, compact owned/ghost partitions, and node gathers in global element order (which is why serial and MPI match bit for bit).

Maintainability

  1. mpi_local_fields.hpp is a second list of mesh-sized fields. A per-node or per-element field added to Variables later but not to these lists stays nullptr in every rank's sub-mesh (local), and full() never gathers it back to rank 0. Code that guards it with if (var.foo) then skips it silently, and output, checkpoints and remeshing see stale values. Serial tests don't catch this. Today the only unlisted fields are unused ones (CL, dl_min, Failure_mode, MAX_shear*, stress_old, …). This rule can't be inferred from the tree, so please add it to AGENTS.md, for example:

    A new per-node or per-element field in Variables must also be listed in mpi_local_fields.hpp, or MPI runs never distribute it. Check with tests/functional/mpi/run_mpi.py.

  2. mpi_simulation.cxx is much denser than the rest of the code: about a quarter of its lines hold two or more statements (at most 2.6 % in fields.cxx, geometry.cxx, bc.cxx, remeshing.cxx), and most local names are one letter (v, m, a, c, f, p). For example, two lines in distribute() hold eight statements, building, serializing and sending one rank's state. Since you asked for a close look at ownership and synchronization, could you split it to one statement per line, with descriptive names, as in the rest of the code? mpi_partition.cxx has the same pattern on a smaller scale (14 %).

  3. update_temperature, update_force and compute_mass split the node-loop header across #ifdef USEMPI … #else … #endif, so the braces balance only within each branch. Without MPI, mpi_gather_passes() already returns 1, mpi_gather_nodes() returns {nullptr, nnode} and mpi_begin_gather() does nothing, so both builds could share the MPI loop. Only var.mpi->finish_exchange() needs a no-op helper like the others. For example, in update_temperature:

    // mpi_simulation.hpp, next to mpi_begin_gather()
    inline void mpi_finish_gather(const Variables& var) {
    #ifdef USEMPI
        if (var.mpi && var.mpi->in_kernel() && var.mpi->overlap()) var.mpi->finish_exchange();
    #endif
    }
    
    // fields.cxx, after the element loop: one loop header for every build
    #ifndef ACC
            #pragma omp master
    #endif
            mpi_begin_gather(var, tmp_result, NODES_PER_ELEM);   // no-op without MPI
    #ifndef ACC
            #pragma omp barrier
    #endif
            const int passes = mpi_gather_passes(var);            // 1 without MPI
            for (int pass=0; pass<passes; ++pass) {
                if (pass == 1) {
    #ifndef ACC
                    #pragma omp master
    #endif
                    mpi_finish_gather(var);
    #ifndef ACC
                    #pragma omp barrier
    #endif
                }
                const MpiNodeRows rows = mpi_gather_nodes(var, pass);   // {nullptr, nnode} without MPI
    #ifndef ACC
                #pragma omp for
    #endif
                #pragma acc parallel loop if(des_acc_offload()) gang vector async firstprivate(rows)
                for (int row=0; row<rows.count; ++row) {
                    const int n = rows.ids ? rows.ids[row] : row;
                    ...   // loop body unchanged
                }
            }

    update_force and compute_mass change the same way. Each node is still summed in the same order, so serial results stay bit-exact. The serial OpenMP build gains one barrier per call.

CI

  1. CI runs MPI only inside the C++20 compatibility workflow (cxx20-compat.yml): MPI builds, the host halo checks (run_accelerator_checks.py --mode host) and run_pt_mpi.py. It does not run run_mpi.py, the serial-vs-MPI suite that covers MMG remeshing, restart, hybrid MPI+OpenMP, empty partitions and root-failure termination, and the one that would catch a field missing from mpi_local_fields.hpp (item 1). Could it run in CI, preferably in its own workflow since cxx20-compat.yml is about compiling as C++20? It needs usemmg=1 hdf5=1 builds; --ranks 1 2 took about 95 s on an 8-core desktop here.

Future improvement

  1. Remeshing runs entirely on rank 0 over the whole mesh (full(), then the serial remesh(), then distribute(true)), so the field interpolation, SPR and marker updates are serial too while the other ranks wait. That is fine for this PR: it keeps the remesh identical to the serial code, and doc/mpi.md already lists it as a scaling limit. For a later PR: MMG has to stay serial, but the interpolation could be distributed, with each rank interpolating its own part of the new mesh from the nearby old-mesh data after repartitioning.

Minor: get_input_parameters runs on every rank before MPI_Init, so each input warning prints once per rank.

@echoi

echoi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@sungho91 @tan2 I think it's a reasonable move to leave out the goSPL MPI coupling part in this PR.

@sungho91 Can usemetis be an independent parameter? Shouldn't it be always 1 when usempi is 1? Maybe it's a design to allow another mesh partitioning library in the future but I think we should just provide a working combination rather than let users mix and match.

@sungho91 @tan2 Do you have performance data from MPI and MPI-OpenACC hybrid runs? If so, please share them here or somewhere (an issue?). Proven gain in performance would be the strongest motivation for this PR. Also, they can be great materials for user manual.

sungho91 and others added 9 commits November 15, 2026 19:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MPI builds took METIS only with usemetis=1 and used it only when
DES_MPI_PARTITIONER=metis, so users could mix a build and a partitioner
that were never checked together. Make METIS part of usempi=1 and use it
for every mesh; meshes with no more elements than ranks keep recursive
coordinate bisection, which supports empty partitions. The usemetis knob,
the USEMETIS macro and DES_MPI_PARTITIONER are removed.

Rename element_force_magnitude to node_force_magnitude, as in #141: it
holds per-node sums.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A field added to Variables but not to these lists stays nullptr on every rank; serial tests cannot catch it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every rank reads the input before MPI_Init, so each warning printed once per rank. Ranks other than the launcher's rank 0 now read it silently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update_temperature, update_force and the mass assembly split their node-loop header across #ifdef USEMPI ... #else, so braces balanced only within each branch. Without MPI the gather helpers already return one pass over every node, so both builds now use the MPI loop; mpi_finish_gather is a no-op without MPI. Nodes are summed in the same order, so serial results stay bit-exact (test-tiny, test-core, test-rect-tiny, test-3d-tiny, test-3d-equ-tiny against master). The serial OpenMP build gains one barrier per call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI ran MPI only inside the C++20 compatibility workflow, which does not run run_mpi.py: MMG remeshing, restart, hybrid MPI+OpenMP, empty partitions and root-failure termination, and the check that would catch a field missing from mpi_local_fields.hpp. mpi-tests.yml builds serial and MPI executables with HDF5 and MMG and runs it at ranks 1 and 2. Locally, ranks 1-4 take about one minute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rces

Review request: mpi_simulation.cxx and mpi_partition.cxx packed several
statements per line and used one-letter local names. Both files now hold
one statement per line in the style of fields.cxx and geometry.cxx, with
descriptive names for locals, parameters and local X-macros.

Code lines with two or more statements:
  mpi_simulation.cxx  25.1% (167/666)  ->  0% (0/1225)
  mpi_partition.cxx   14.6% (23/157)   ->  0% (0/237)

No operation, loop bound, MPI call order or tag changed. With the 2D and
3D MPI builds, the disassembly of both objects is unchanged apart from
one renamed internal symbol. run_mpi.py with ranks 1-4 passes with the
same 56 bit-exact checks as before, and run_pt_mpi.py with ranks 3 4
gives the same result for every case as the parent commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picks up #135. Its material-following monitor points recompute the nearest
node from each point's current position, so under MPI the rows now gather
the current coordinates first, locate every point, and then gather the
nodes and elements the rows read, including the containing element and its
nodes. The seismic moment from plastic-strain increments is summed on the
root after the fields are collected, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
usempi=1 now always links METIS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sungho91

Copy link
Copy Markdown
Collaborator Author

@tan2 Thanks for the careful review. All points are addressed, and current master, including #135, is merged in (1ac63ff). The #135 monitors needed one MPI change there: rows now gather the current coordinates before locating each point; monitor CSVs from serial and 2- and 3-rank runs are identical with a moving mesh, remeshing and grid points.

  1. AGENTS.md now states the mpi_local_fields.hpp rule (82fa08e).
  2. mpi_simulation.cxx and mpi_partition.cxx now have one statement per line and descriptive names (e8eed7b); lines with two or more statements dropped from 25% to 0% and from 15% to 0%. The generated CPU code is unchanged: the disassembly of both objects before and after is identical apart from one renamed section.
  3. update_temperature, update_force and the mass assembly share one loop header in both builds, with mpi_finish_gather as the no-op you suggested (b6aa8f3). Serial results are bit-exact against master (test-tiny, test-core, test-rect-tiny, test-3d-tiny, test-3d-equ-tiny).
  4. mpi-tests.yml runs run_mpi.py at ranks 1 and 2 with HDF5 and MMG builds (fafcb27). Locally, ranks 1–4 take about a minute: 56 bit-exact checks plus the termination checks.
  5. Agreed; distributed interpolation after remeshing is for a later PR.

Minor: only the launcher's rank 0 now prints input warnings (67375f0).

@sungho91

Copy link
Copy Markdown
Collaborator Author

@echoi Thanks.

usemetis. Agreed. usempi=1 now always builds with METIS and partitions every mesh with it; the usemetis knob and the DES_MPI_PARTITIONER switch are gone. A mesh with no more elements than ranks still uses recursive coordinate bisection, which supports empty partitions. With METIS, every serial-versus-MPI check stays bit-exact.

Performance. I have single-node timings only, from an earlier development build of this branch, and they show no speedup worth claiming yet. On one node (Xeon 8268, 2 A100), 2D box, 64,177 elements, 1,000 steps, median of three runs, simulation time only (no launcher or device startup), against one CPU core:

run speedup
OpenMP, 8 threads 3.3×
MPI, 8 ranks 2.6×
1 A100 2.4×
2 A100 1.9×

Caveats that matter:

  • GPU startup takes about 7 s. Counting it, one A100 is only 1.26× one core (a repeat with the CPUs on the GPUs' NUMA nodes: 2.5× in simulation, 1.26× in total).
  • In the table's job, the CPUs were bound to NUMA node 0, remote from the GPUs.
  • Open MPI used btl self,tcp, so ranks talked over TCP even within the node; the GPU build used managed memory with FMA off (-gpu=managed,nofma -Kieee).
  • Two GPUs are slower than one at this size, and OpenMP beats MPI at the same core count; root-side mesh management and output also stay serial.
  • The two-H100 runs on another host checked correctness only: one H100 was shared with an unrelated job, so their timings are not usable.

I'll measure the current code on a multi-node cluster and post the results in an issue; @tan2's InfiniBand runs would be a good second data point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sungho91 and others added 2 commits November 16, 2026 14:45
#141 now carries the force magnitude and the PT rounding floor that this
branch also had; the merge keeps master's form and this branch's MPI node
range.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-rank GPU run uses the compact backend and partitions the mesh.
METIS 5.1 stops with SIGFPE (integer division by zero) when asked for
one part. With one rank every element belongs to rank 0, so skip METIS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants