Repository navigation
Conversation
Restart output can start at frame 45. Comparing frame 50 reads its data correctly, then crashes while printing the step because it indexes a six-entry step list with the frame number. Resolve the step through the actual frame list. Both three- and four-rank 50,000-step core-complex continuations report BIT-EXACT at frame 50. Co-Authored-By: Codex <codex@openai.com>
OpenMP timesteps share a process and cannot distribute mesh storage or physics across MPI processes. Add separate usempi=1 CPU executables with RCB ownership, compact owned/ghost meshes and reusable neighbor buffers. Distribute thermal, constitutive, force, motion, geometry, mass and timestep kernels while preserving each node's global element summation order. Rebuild ownership and halos after every remesh. Rank zero retains the global management copy, MMG, interpolation, marker/phase work and output; this is an experimental CPU backend with those central scaling limits. Build 2D/3D MPI and serial/OpenMP with MMG/HDF5. Fresh parent references and all four make cmp runs are BIT-EXACT (MPI uses 3 ranks in 2D, 4 in 3D). The 45-check matrix at 1/3/4 ranks also covers hybrid threads, hydraulic updates, averaged remeshing, restart, empty partitions and error exits. Co-Authored-By: Codex <codex@openai.com>
Geometric cuts cannot optimize DES's shared-node dependencies. Add an optional shared-node element graph with the METIS communication-volume objective, deterministic seed zero and 3% element-balance tolerance. Select it with usempi=1 usemetis=1 and DES_MPI_PARTITIONER=metis. RCB remains the default and handles meshes with no more elements than ranks. Ownership and halos use the existing builders and are rebuilt after every MMG remesh. The objective is a proxy for real halo traffic, not a speedup guarantee. Build MPI+METIS and serial/OpenMP in both dimensions with MMG/HDF5. All four parent make cmp runs are BIT-EXACT. The METIS 45-check matrix at 1/3/4 ranks also matches every physical dataset, including averaged 2D/3D remeshing. Co-Authored-By: Codex <codex@openai.com>
Element-to-node assembly waits for ghost contributions before doing any owned-node work, and inactive management paths collect unused global state. Separate interior nodes from interface nodes so thermal, force and mass assembly can post transfers, compute interior rows, then finish interface rows without changing any node's element summation order. Keep DES_MPI_OVERLAP=0 as the blocking path. Collect only coordinates and volumes for mesh-quality checks; skip inactive serpentinization and monitor collections. Share SoA pack indexing and add per-rank operation, wait and communication counters, including reductions. Posted work windows do not by themselves prove asynchronous MPI progress. Both 2D/3D MPI+METIS and serial/OpenMP builds pass parent make cmp exactly. The 45-check METIS/overlap matrix also matches every physical dataset. Earlier pinned 64,177-element, 1,000-step measurements of these changes reduced three-/four-rank median times from 4.9237/4.5710 s to 4.2908/4.1711 s; small and remeshing cases do not show the same benefit. Co-Authored-By: Codex <codex@openai.com>
Short smoke cases do not establish agreement through scientific histories. Add isolated full/regression runs of existing core-complex/MMG, RSF and rifting examples, checking complete physical fields, checkpoints and monitor records with provenance and figures. Label bounded runs explicitly. Keep the historical Maxwell input/source patch unchanged. Add a separate current-format constitutive case and compare stress against the analytical relaxation curve in 2D/3D. Fresh three-rank runs completed 5,000 steps through five relaxation times: all 51 frames were bit-exact and maximum relative analytical error was below 8e-8 (acceptance limit 1e-6). Co-Authored-By: Codex <codex@openai.com>
Unequal CPU placement and one-off timings hide the cost of MPI ownership, halos and repartitioning. Add verified affinity, warmups and randomized paired/interleaved rounds for equal OpenMP/MPI budgets, before/after halo changes, and RCB versus METIS on fixed/adaptive meshes in both dimensions. Retain commands, executable/source hashes, sample ranges, partition costs, communication payloads and exact parent-output comparisons. Plot actual ownership traces and report phase times separately from complete runtime. The RCB/METIS study contains 120 measured runs and 24 warmups at 3/4 ranks, all bit-exact; reduced halo bytes did not establish a consistent speedup. These local shared-memory timings do not predict multi-server performance. Co-Authored-By: Codex <codex@openai.com>
Parallel remesher CLIs can change labels, metrics and required entities, so importing one into the DES timestep loop would hide compatibility failures behind a new mesh. Add pinned external builds, a test-only relink wrapper that captures the exact existing MMG inputs, and replay/transfer checks with retained provenance and figures. Keep production MMG, its metric/freezing rules and field/marker remapping unchanged. The optional ParMmg2D merge patch remains an explicit prototype; recorded required-entity and label failures prevent production adoption. Mark its unified-diff blank context lines as patch data, while checking the actual added C code with git apply --whitespace=error-all. Both 2D/3D capture wrappers compile and link against the verified serial objects. The patch applies cleanly to its pinned ParMmg2D source. Python entry points compile and their help paths run without starting builds. Co-Authored-By: Codex <codex@openai.com>
CPU transport correctness cannot establish GPU pointer handling or NVLink routing. Add standalone contiguous-buffer and shared SoA pack/unpack tests with asymmetric empty messages, reusable buffers and irregular row maps. OpenACC host mode is explicit; GPU-required mode rejects missing devices or CUDA-aware MPI instead of silently using the CPU. Host checks passed in 2D/3D at 1/2/3/4/6/8 ranks, and ordinary transport checks passed on three/four ranks. GPU-required mode returned unsupported status 31 on this workspace. No NVIDIA device/compiler is available here; full GPU DES integration, multi-GPU correctness and actual NVLink routing remain unimplemented or unverified. Co-Authored-By: Codex <codex@openai.com>
Document owned/ghost meshes, central management limits, remesh-time repartitioning, optional METIS and CPU halo overlap with reproducible 2D/3D correctness, scientific-example and timing commands. Record paired timings and their sample ranges rather than treating fewer halo bytes as a runtime guarantee. Keep RCB as the measured laptop default. Distinguish complete scientific histories from bounded regressions, and host transport checks from the unintegrated GPU solver/NVLink path. Explain the parallel-remesher compatibility failures while retaining the existing production MMG policy and remapping. Co-Authored-By: Codex <codex@openai.com>
The MPI branch predates upstream #122's CLI11 parser and #125's C++17 baseline/C++20 compatibility checks. Integrate current master without rewriting the nine published feature commits. Preserve MPI wrapper compiler introspection when selecting GNU, Clang and NVHPC flags. Keep C++17 as the default, update standalone transport checks to that floor with an explicit C++20 option, and remove obsolete Boost arguments/link flags from capture tooling and active build instructions. Retain old timing reports as historical measurements of their stated base. 2D/3D serial/OpenMP and MPI+METIS build in both standards. Fresh master references and all eight make cmp runs are BIT-EXACT. C++17 RCB passes 78 matrix checks; C++20 METIS passes 45. Both standards pass all 62 parser cases in serial and MPI, 12 host halo checks and full 51-frame 2D/3D Maxwell comparisons. C++20 reports no first-party deprecation warnings. Co-Authored-By: Codex <codex@openai.com>
The upstream compatibility jobs compile only the serial code paths, so an MPI wrapper-selection or standalone language-floor regression can escape. Add 2D/3D MPI+METIS/HDF5 builds at the C++17 default and with the strict C++20 checker. Trigger the workflow for MPI helper changes too. Exercise explicit OpenACC host-mode halo checks in both standards at 1/2/3/4 ranks, oversubscribing small CI workers. Retain the build/check logs and state clearly that these tests do not verify GPU execution. Local replicas pass all four builds and all 16 host checks using the local METIS prefix; CI installs the corresponding system development package. Co-Authored-By: Codex <codex@openai.com>
The PT refactor in PR #106 replaces the legacy iteration loop with physical trial/commit solves. Dispatching its constituent kernels alone leaves the global trial workspace stale and provides no collective stopping decision. Merge the upstream solver and execute both PT methods, including initial body-force equilibrium, on compact owned/ghost meshes. Exchange trial rates and factors, reduce owner-only residuals and ADR sums, and retain accepted PT diagnostics and running-scale state through output, remeshing and restart. Capture absolute element forces before boundary assembly reuses its scratch; the former cache read can otherwise make roundoff thresholds partition dependent. Both dimensions build as C++17/C++20 with MMG, HDF5 and MPI/METIS. Fresh upstream make set references give eight BIT-EXACT make cmp comparisons; 123 general MPI comparisons are exact. Scoped PT checks cover both methods, RCB/METIS, 1/3/4/8 ranks, remeshing, restart and expected failures, with explicit field tolerances. The focused four-thread force/material/rollback suites pass 6,852 assertions in 2D and 10,185 in 3D. Co-Authored-By: Codex <codex@openai.com>
The generic MPI suite cannot check PT acceptance, material rollback or whether a low residual describes the accepted physical state. Reuse the upstream independent force/RMS, held-load/reload, Maxwell creep and exact restart checks on serial and MPI executables, then compare every physical HDF5 dataset with an explicit tolerance and measured byte equality. Cover both PT methods, initial hydrostatics, two MMG boundary options, running-scale history, dynamic timesteps, oblique supports and collective iteration-limit failure. Retain inputs, commands, executable hashes and worst-field error evidence. Plot the checked strain histories against the analytical creep reference. Extend C++20 compatibility CI with PT checks. Local C++17 RCB and C++20 METIS runs pass 192 field comparisons and 24 expected-failure checks, including ranks 1/3/4/8 and twelve option-0 hybrid checks with two threads per rank. Each run's own restart remains exact; 130 serial/MPI comparisons are byte-exact. The maximum normalized Maxwell reference error is 2.171e-6. Document the 50,000-iteration MMG fixture cap and force/pressure scales used for cancellation-sensitive diagnostics. Co-Authored-By: Codex <codex@openai.com>
Large calculations need an execution path beyond one GPU or one shared-memory OpenMP process. Extend the compact CPU MPI backend with per-rank GPU kernels and pinned host halos, and couple DES to a persistent GoSPL MPI world with independent process counts and continuation state. Retain upstream C++17/CLI11 and PT mechanics. Current-source 2D/3D parent regressions, 34 CPU cases, four C++20 host halo cases, four NVHPC 25.7 GPU cases, coupled continuation and two-node equivalence checks pass. Historical hardware and cost matrices retain separate provenance. Add a reviewer quickstart with tested CPU/GPU commands, an eight-step fresh coupling fixture, allocation sizes, expected results and a remeshing restart example. Co-Authored-By: Codex <noreply@openai.com>
Retain upstream GoSPL Python/OpenMP library discovery and MMG configuration fixes. Resolve the README by keeping the current environment setup together with the MPI launch and reviewer examples. Linux numerical sources and build flags are unchanged. Co-Authored-By: Codex <noreply@openai.com>
Widen halo allocation products before multiplication, remove five unused imports, and use one checkpoint import style. Explain the expected cleanup races and handle worker failures and interrupts explicitly so a failing rank cannot strand its peers. C++17/C++20 host halo checks pass in both dimensions at one and two ranks. Two-rank protocol rejection, KeyboardInterrupt/SystemExit cleanup and 14 coupled continuation comparisons pass. Co-Authored-By: Codex <noreply@openai.com>
The MPI CI job built without MMG and crashed during its first compact partition: init_elem_size_n and refined_elem_size_n were allocated but empty, so selecting them by node ID dereferenced null storage. Serialize, select and exchange these optional fields only in MMG builds, where they are populated. The MMG-enabled field list is unchanged. Reproduced the three-rank crash before the fix. Non-MMG parent make set / changed make cmp pass bit-exact in 2D and 3D. All 40 PT checks pass with the existing tolerances across both dimensions, relaxation methods 0/1 and ranks 3/4, including restart and collective failures. Co-Authored-By: Codex <noreply@openai.com>
PR #128 is five commits behind master after the Docker images update in PR #119. Merge those changes without rewriting the existing MPI history, retaining the upstream MMG cache-path guard and Docker build/CI/docs. Native arm64 macOS builds pass in 2D/3D with OpenMP, HDF5 and MMG, for serial and MPI executables. Master make set / changed make cmp are bit-exact for both dimensions and both executables. All 34 existing MPI checks are bit-exact against current master; all 20 PT checks pass at the existing rtol=1e-8/atol=1e-12, including four collective failures. Co-Authored-By: Codex <noreply@openai.com>
The MPI branch has no native macOS verification record after the current master merge. Document the Apple Silicon toolchain, CPU dependency setup, existing reproduction commands and bounded validation results. Both dimensions and serial/MPI builds pass. Four master regression comparisons and 34 MPI matrix comparisons are bit-exact. All 20 PT checks pass without relaxing tolerances; GPU and GoSPL were not tested. Co-Authored-By: Codex <noreply@openai.com>
The scheduler guard tests assume /bin/true exists. Native macOS keeps true elsewhere, so two checks fail before reaching their intended guard. Use the running Python executable as the existing file/hash fixture. Every invocation is rejected before execution. All eight scheduler guard checks now pass on macOS without contacting a scheduler or launching MPI. Solver and launcher behavior are unchanged. Co-Authored-By: Codex <noreply@openai.com>
The MPI PR includes dated execution diaries and standalone ParMmg experiments outside its production backend and CI. They add internal host details, exploratory patches and duplicate measurements to user/test docs. Retire two dated reports and seven isolated remesher/capture tools. Keep their originals with SHA-256 records in an ignored local archive. Retain the reusable CPU/GPU/GoSPL tests, commands, acceptance criteria and bounded verification conclusions. Remove obsolete links, the patch-only whitespace exception and duplicate run history. Ignore dated reports and macOS dSYM bundles. No solver arithmetic or executable driver logic changes. This removes 1,749 net lines. All 33 remaining local documentation links resolve; 55 Python files parse without retired-module dependencies. Both MPI dimensions build on macOS and the tiny two-rank regressions remain bit-exact against the recorded master references. Co-Authored-By: Codex <noreply@openai.com>
The MPI feature still includes nine workload-specific timing/plotting tools and a legacy transport harness. The latter duplicates the shared halo test's irregular/empty-message coverage. None is called by the production solver or CI. Archive these ten files locally with verified hashes. Remove the old benchmark-only options from run_mpi.py, its unused statistics import and the transport wrapper used only by the retired harness. Preserve every default physical check and all six shared helper functions unchanged, verified by AST comparison. Keep the real GoSPL audit, scientific examples, partition tracing, MPI/PT/GPU/restart drivers and CI halo test. Remove the corresponding commands and performance-study claims from the docs/changelog. The change removes 1,290 net lines. Both MPI dimensions build; all 34 CPU matrix checks are bit-exact, four OpenACC host halo checks and eight scheduler guard checks pass on macOS. Local document links resolve and retained Python modules do not import archived tools. Co-Authored-By: Codex <noreply@openai.com>
The MPI branch lacks the managed-memory configuration-reader fixes merged into master in #126. Merge current master without rewriting the existing feature history, preserving its nvc++ getline workaround and deferred GCC 8 header check. Native macOS serial and MPI builds pass in both dimensions with OpenMP, HDF5 and MMG. Four current-master make set / merged make cmp comparisons are bit-exact, all 34 MPI matrix comparisons are bit-exact, and all 62 existing bad-input/configuration acceptance cases pass. GPU/NVHPC runtime checks are outside this CPU-only repeat. Co-Authored-By: Codex <noreply@openai.com>
Identical host/device halo loops and per-step material scans repeat work on unchanged compact state. Redistribution also invalidates an already-current root mesh, so later collections transfer it again. Share the pack/unpack indexing and refresh composition caches at redistribution. Reuse fresh full-root fields for boundary and sample queries, and remove the extra post-step GoSPL vector copy. The 12-step, six-remesh CPU/GPU coupling cases reduce root collection payloads from 2019488 to 1926072 bytes each. All 25 saved physical files per backend remain bit-exact against the pre-cleanup trajectory. Both-dimensional parent comparisons, 34 CPU cases, 40 PT checks, 40 GPU file comparisons and coupled restarts pass. Co-Authored-By: Codex <noreply@openai.com>
The root management loop rescans composition after distribution has refreshed both caches. PT repeats an accepted-velocity exchange, RSF reduces its diagnostics separately, and GoSPL ranks seed the entire surface while checkpoint fields are serialized twice. Reuse the attached-backend query, vector maximum and native PETSc DM mapping. Serialize the complete snapshot once, preserving initialization/isostasy, restored-velocity synchronization and collective checkpoint rejection. In a four-step three-rank RSF case, root cache entries fall from 12 to 4 and Allreduce calls from 40 to 36 per rank. Two-rank GoSPL seed queries fall from 260 to 169 with correct owned/ghost values. The checkpoint fixture preserves wire bytes and rejects unsupported state. Both-dimensional parent comparisons, 34 CPU cases, 40 PT checks and 40 standard GPU file comparisons pass. Six additional GPU PT/RSF cases preserve all 60 files bit-exact; actual PT GPU launches are recorded. CPU/GPU coupled restarts pass 14 comparisons each and retain 25 bit-exact continuous files per backend. Co-Authored-By: Codex <noreply@openai.com>
Compact ranks copy individual marker histories although their kernels use only per-element material counts. MPI averaging also duplicates Output's algorithm, and root serializes its own partition before reading it back. Keep marker records on the management mesh and share Output averaging with compact ranks. Reuse collection indices, scalar reduction helpers and checkpoint buffers; remove the unreachable friction command. In the 40-step 3D averaged-remesh case, worker state traffic falls from 3282168 to 1720230 bytes with identical fields. The hydration case falls from 11601401 to 4372254 bytes. Both dimensions pass parent make set/cmp, 34 CPU checks, 52 PT checks and 120 GPU file comparisons. H100 coupled continuation passes 14 comparisons; the qsub two-host run passes nine byte-exact comparisons with four allocated slots. Co-Authored-By: Codex <noreply@openai.com>
Clang rejects the global NDIMS/NSTR bounds in collapsed averaging loops under default(none), while GCC accepts them. Capture local component counts explicitly and keep the OpenACC clauses in step. Clang 14 C++17 and Clang 18/21 C++20 pass in both dimensions. The existing C++20 CI script also passes locally with GCC 13. Parent make set/cmp and 80 CPU plus 20 GPU averaging comparisons remain byte-exact, including four remeshes per run. NVHPC 25.7 builds both dimensions for SM86/SM90. Co-Authored-By: Codex <noreply@openai.com>
|
I will review this PR and test its parallel scaling on our infiniteband-equipped cluster. |
|
Could we leave the GoSPL MPI coupling for the next PR and keep this one to the MPI backend? The coupling is about 1.8k of the 7.4k added lines: the 18 files under Reasons:
|
Keep this PR to the CPU/GPU MPI backend so it can be reviewed and
scaled without a GoSPL installation. The coupling returns in its own PR
with its continuation tests and numerical-effect notes.
Removed here:
- gospl_driver/{checkpoint,des_gospl_client,mpi_worker,
parallel_surface,run_coupled}.py, tests/functional/gospl/ and
doc/gospl-mpi-coupling.md.
- Coupled continuation restart (dynearthsol.cxx), the .gospl checkpoint
sidecar (output.cxx) and its checkpoint-reference I/O (binaryio).
- GoSPL hooks in mpi_simulation.cxx: velocity in the surface exchange
and idle-step clock advance for surface_process_option = 11.
Restored to master:
- use_gospl() in bc.cxx and gospl_driver/gospl-driver.{cxx,hpp},
README.md. This returns the serial coupling to SEC_PER_YR = 3.1536e7
and the previous PETSc seed ordering, so serial GoSPL results are
unchanged by this PR.
The Makefile now rejects usempi=1 usegospl=1 with a message.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The On master, It changes results only when that floor sets the PT threshold, i.e. when it exceeds Minor, same code: |
The MPI pass/fail drivers compare fields and write JSON reports; figures are not part of the solver PR. Remove plot_partitions.py, the --plot options of reproduce_examples.py and run_pt_mpi.py, and the matching documentation. Partition traces (DES_MPI_PARTITION_TRACE) and the [mpi][partition] log rows are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@tan2, Done in b55ed6f: this PR now holds only the MPI backend. The GoSPL MPI coupling (driver scripts, b6d504e also drops the optional plotting helpers ( Post-split checks: GoSPL MPI coupling will follow in a separate PR when requested, with its continuation tests and a "Numerical effect" section for the 🤖 Generated with Claude Code |
Resolve parameters.hpp: keep the MPI branch's nullptr member defaults together with master's ownership comments, and drop the remesh transients that master moved into RemeshScratch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tan2
left a comment
There was a problem hiding this comment.
Thanks, this is a big step. I ran the checks below locally (Linux, g++ 14, Open MPI 5.0.7) at 772b5e8, before the GoSPL split and the master merge, and re-checked items 1–4 against 2e59200.
What I checked
- Non-MPI builds (2D/3D × Triangle/TetGen/MMG,
-O2): all 23 regression cases (including plane strain and 3D elasto-visco-plastic, with forced remeshing) are bit-exact with its parent6195cc2. ASan/LSan builds are clean. - MPI builds compile without new warnings.
tests/functional/mpi/run_mpi.py --ranks 1 2passes (34 bit-exact checks, plus the root-failure and--helpexit checks). - My own check, 3 ranks against serial on cases the driver does not use (
test-tiny-forced,test-rift,test-3d-tiny-forcedwith MMG,test-3d-remesh): every field in every save and checkpoint file is bit-exact.
The design reads well: root-owned mesh management, compact owned/ghost partitions, and node gathers in global element order (which is why serial and MPI match bit for bit).
Maintainability
-
mpi_local_fields.hppis a second list of mesh-sized fields. A per-node or per-element field added toVariableslater but not to these lists staysnullptrin every rank's sub-mesh (local), andfull()never gathers it back to rank 0. Code that guards it withif (var.foo)then skips it silently, and output, checkpoints and remeshing see stale values. Serial tests don't catch this. Today the only unlisted fields are unused ones (CL,dl_min,Failure_mode,MAX_shear*,stress_old, …). This rule can't be inferred from the tree, so please add it toAGENTS.md, for example:A new per-node or per-element field in
Variablesmust also be listed inmpi_local_fields.hpp, or MPI runs never distribute it. Check withtests/functional/mpi/run_mpi.py. -
mpi_simulation.cxxis much denser than the rest of the code: about a quarter of its lines hold two or more statements (at most 2.6 % infields.cxx,geometry.cxx,bc.cxx,remeshing.cxx), and most local names are one letter (v,m,a,c,f,p). For example, two lines indistribute()hold eight statements, building, serializing and sending one rank's state. Since you asked for a close look at ownership and synchronization, could you split it to one statement per line, with descriptive names, as in the rest of the code?mpi_partition.cxxhas the same pattern on a smaller scale (14 %). -
update_temperature,update_forceandcompute_masssplit the node-loop header across#ifdef USEMPI … #else … #endif, so the braces balance only within each branch. Without MPI,mpi_gather_passes()already returns 1,mpi_gather_nodes()returns{nullptr, nnode}andmpi_begin_gather()does nothing, so both builds could share the MPI loop. Onlyvar.mpi->finish_exchange()needs a no-op helper like the others. For example, inupdate_temperature:// mpi_simulation.hpp, next to mpi_begin_gather() inline void mpi_finish_gather(const Variables& var) { #ifdef USEMPI if (var.mpi && var.mpi->in_kernel() && var.mpi->overlap()) var.mpi->finish_exchange(); #endif } // fields.cxx, after the element loop: one loop header for every build #ifndef ACC #pragma omp master #endif mpi_begin_gather(var, tmp_result, NODES_PER_ELEM); // no-op without MPI #ifndef ACC #pragma omp barrier #endif const int passes = mpi_gather_passes(var); // 1 without MPI for (int pass=0; pass<passes; ++pass) { if (pass == 1) { #ifndef ACC #pragma omp master #endif mpi_finish_gather(var); #ifndef ACC #pragma omp barrier #endif } const MpiNodeRows rows = mpi_gather_nodes(var, pass); // {nullptr, nnode} without MPI #ifndef ACC #pragma omp for #endif #pragma acc parallel loop if(des_acc_offload()) gang vector async firstprivate(rows) for (int row=0; row<rows.count; ++row) { const int n = rows.ids ? rows.ids[row] : row; ... // loop body unchanged } }
update_forceandcompute_masschange the same way. Each node is still summed in the same order, so serial results stay bit-exact. The serial OpenMP build gains one barrier per call.
CI
- CI runs MPI only inside the C++20 compatibility workflow (
cxx20-compat.yml): MPI builds, the host halo checks (run_accelerator_checks.py --mode host) andrun_pt_mpi.py. It does not runrun_mpi.py, the serial-vs-MPI suite that covers MMG remeshing, restart, hybrid MPI+OpenMP, empty partitions and root-failure termination, and the one that would catch a field missing frommpi_local_fields.hpp(item 1). Could it run in CI, preferably in its own workflow sincecxx20-compat.ymlis about compiling as C++20? It needsusemmg=1 hdf5=1builds;--ranks 1 2took about 95 s on an 8-core desktop here.
Future improvement
- Remeshing runs entirely on rank 0 over the whole mesh (
full(), then the serialremesh(), thendistribute(true)), so the field interpolation, SPR and marker updates are serial too while the other ranks wait. That is fine for this PR: it keeps the remesh identical to the serial code, anddoc/mpi.mdalready lists it as a scaling limit. For a later PR: MMG has to stay serial, but the interpolation could be distributed, with each rank interpolating its own part of the new mesh from the nearby old-mesh data after repartitioning.
Minor: get_input_parameters runs on every rank before MPI_Init, so each input warning prints once per rank.
|
@sungho91 @tan2 I think it's a reasonable move to leave out the goSPL MPI coupling part in this PR. @sungho91 Can @sungho91 @tan2 Do you have performance data from MPI and MPI-OpenACC hybrid runs? If so, please share them here or somewhere (an issue?). Proven gain in performance would be the strongest motivation for this PR. Also, they can be great materials for user manual. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MPI builds took METIS only with usemetis=1 and used it only when DES_MPI_PARTITIONER=metis, so users could mix a build and a partitioner that were never checked together. Make METIS part of usempi=1 and use it for every mesh; meshes with no more elements than ranks keep recursive coordinate bisection, which supports empty partitions. The usemetis knob, the USEMETIS macro and DES_MPI_PARTITIONER are removed. Rename element_force_magnitude to node_force_magnitude, as in #141: it holds per-node sums. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A field added to Variables but not to these lists stays nullptr on every rank; serial tests cannot catch it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every rank reads the input before MPI_Init, so each warning printed once per rank. Ranks other than the launcher's rank 0 now read it silently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update_temperature, update_force and the mass assembly split their node-loop header across #ifdef USEMPI ... #else, so braces balanced only within each branch. Without MPI the gather helpers already return one pass over every node, so both builds now use the MPI loop; mpi_finish_gather is a no-op without MPI. Nodes are summed in the same order, so serial results stay bit-exact (test-tiny, test-core, test-rect-tiny, test-3d-tiny, test-3d-equ-tiny against master). The serial OpenMP build gains one barrier per call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI ran MPI only inside the C++20 compatibility workflow, which does not run run_mpi.py: MMG remeshing, restart, hybrid MPI+OpenMP, empty partitions and root-failure termination, and the check that would catch a field missing from mpi_local_fields.hpp. mpi-tests.yml builds serial and MPI executables with HDF5 and MMG and runs it at ranks 1 and 2. Locally, ranks 1-4 take about one minute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rces Review request: mpi_simulation.cxx and mpi_partition.cxx packed several statements per line and used one-letter local names. Both files now hold one statement per line in the style of fields.cxx and geometry.cxx, with descriptive names for locals, parameters and local X-macros. Code lines with two or more statements: mpi_simulation.cxx 25.1% (167/666) -> 0% (0/1225) mpi_partition.cxx 14.6% (23/157) -> 0% (0/237) No operation, loop bound, MPI call order or tag changed. With the 2D and 3D MPI builds, the disassembly of both objects is unchanged apart from one renamed internal symbol. run_mpi.py with ranks 1-4 passes with the same 56 bit-exact checks as before, and run_pt_mpi.py with ranks 3 4 gives the same result for every case as the parent commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picks up #135. Its material-following monitor points recompute the nearest node from each point's current position, so under MPI the rows now gather the current coordinates first, locate every point, and then gather the nodes and elements the rows read, including the containing element and its nodes. The seismic moment from plastic-strain increments is summed on the root after the fields are collected, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
usempi=1 now always links METIS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@tan2 Thanks for the careful review. All points are addressed, and current master, including #135, is merged in (1ac63ff). The #135 monitors needed one MPI change there: rows now gather the current coordinates before locating each point; monitor CSVs from serial and 2- and 3-rank runs are identical with a moving mesh, remeshing and grid points.
Minor: only the launcher's rank 0 now prints input warnings (67375f0). |
|
@echoi Thanks.
Performance. I have single-node timings only, from an earlier development build of this branch, and they show no speedup worth claiming yet. On one node (Xeon 8268, 2 A100), 2D box, 64,177 elements, 1,000 steps, median of three runs, simulation time only (no launcher or device startup), against one CPU core:
Caveats that matter:
I'll measure the current code on a multi-node cluster and post the results in an issue; @tan2's InfiniBand runs would be a good second data point. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#141 now carries the force magnitude and the PT rounding floor that this branch also had; the merge keeps master's form and this branch's MPI node range. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-rank GPU run uses the compact backend and partitions the mesh. METIS 5.1 stops with SIGFPE (integer division by zero) when asked for one part. With one rank every element belongs to rank 0, so skip METIS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why
DES currently supports OpenMP and a single GPU. Larger future models need an execution path that can distribute timestep work across CPUs, nodes and GPUs. This adds opt-in MPI execution while keeping mesh management centralized.
What
mpi-tests.yml) runs the serial-versus-MPI suite; reproducible CPU/GPU functional checks and a reviewer quickstart.Scope: As requested in review, this PR holds only the MPI backend. GoSPL MPI coupling (independent DES/GoSPL MPI worlds, owned/ghost surface mapping, coupled continuation checkpoints) moves to a follow-up PR with its own continuation tests and "Numerical effect" section. Until then the Makefile rejects
usempi=1 usegospl=1with a message. Serial GoSPL coupling (usegospl=1withoutusempi) is unchanged from master.For the reviewer
mpi_simulation.cxx(execute,distribute,exchange), thenmpi_partition.cxx.Try it
usempi=1, which needs METIS 5 (METIS_INCLUDE_DIR,METIS_LIB_DIRif not on default paths).MPI_CXX; addopenacc=1 openmp=0 nofma=1 GPU_CC=...for GPUs. Launcher-nselects ranks.Numerical effect
1e-8criterion and are bit-exact in the checks below; GPU fixed-mesh checks are measured separately.tests/functional/mpi/run_mpi.pywith a parent serial executable as--serial(see the quickstart and verification scope). Speedup depends on workload.cd1042c, including fix: prepare poromechanics with monitoring and diagnostic fixes (1/8) #135; Linux, CPU, HDF5 + MMG):run_mpi.pyin 2D and 3D at 1–4 ranks gives 56 bit-exact comparisons against the serial build, including remeshing, restart, hybrid MPI+OpenMP and empty partitions;run_pt_mpi.py(ranks 3 and 4) passes. Monitor CSVs from serial and 2- and 3-rank runs are identical with a moving mesh, with remeshing, and for grid points with rows before and after each remesh. Serial builds are bit-exact against master (test-tiny,test-core,test-rect-tiny,test-3d-tiny,test-3d-equ-tiny). The basic-build, functional-test, OpenMP and C++20 (g++, including the MPI build) steps pass locally.New or retired names
usempi,MPI_CXX,METIS_INCLUDE_DIR,METIS_LIB_DIR, documentedDES_MPI_*controls and MPI executables.Checklist
ndims=2andndims=3: yes, CPU/GPU builds and parent/changed regressions checked.## [Unreleased]in CHANGELOG.md: yes.input.cxxand documented inexamples/defaults.cfg: no new cfg parameters.tests/functional/orbenchmarks-cores/: yes, MPI/GPU drivers and runnable fixtures.AI disclosure
OpenAI Codex implemented the MPI backend, tests/documentation and the original description, and ran the reported pre-split checks. Claude Code split out the GoSPL coupling and plotting helpers, made the review-round changes (METIS by default, shared gather loops, one-statement-per-line MPI sources, the MPI CI workflow, the master merge, the one-rank GPU partition fix) with their checks, and revised this description.
🤖 Generated with Claude Code