Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 935 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 935 commits into
sbryngelson wants to merge 935 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
- Coverage 61.26% 60.60% -0.67%
==========================================
Files 84 96 +12
Lines 22330 31022 +8692
Branches 3265 5040 +1775
==========================================
+ Hits 13680 18800 +5120
- Misses 6207 8423 +2216
- Partials 2443 3799 +1356 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
sbryngelson
changed the title
Member
Author
|
2 tasks
fahnab666
added a commit
to fahnab666/mfc-nabid
that referenced
this pull request
Aug 29, 2026
…isting lint debt Merges the 20 commits added to sbryngelson/up-mega since ee7758b: the CCE descriptor and GPU_DECLARE fixes for the Frontier AMR abort, the NVHPC declare-ordering fix, restart format v2 (per-block owner/extents), amr_blocking_factor defaulting to 4, and the level-order clustering reductions. The delta is confined to AMR: m_amr, m_amr_regrid, m_amr_restart, m_constants, m_global_parameters, cmake/GPU.cmake, 14 AMR goldens, and two AMR-only toolchain hunks. No JWL, IBM, Riemann, or EOS source file is touched and no non-AMR golden changes. Merge was conflict-free; the local static-AMR GPU allocation cap for IBM survives intact. Also clears lint debt that predates this merge and blocked any commit: formatter line-wrapping in six JWL/IBM files, the duplicated rxn_val PROHIBITs in pre_process/m_checker.fpp (already enforced in case_validator.check_stiffened_eos, as its own docstring states), the stale hyperelasticity row in case.md (removed upstream), the missing sec-mhd anchor, and m_jwl/m_jwl_sources missing from module_categories.json.
The simulation path raises MFCException on a non-zero return code; the --test-all post_process path captured stdout and ignored the result entirely (common.system runs with check=False). post_process could abort, segfault or fail outright and the test still reported PASS as long as the simulation goldens matched. That is how a total break of the AMR post-process reader shipped unnoticed: --test-all ran post_process on every AMR case and threw the answer away. Verified both directions: an AMR test now fails where it previously passed, and non-AMR cases are unaffected.
Two defects, both in the AMR fine-block reader, both invisible until the test suite started checking post_process's exit code. First: format v2 was never taught to post. The writer marks v2 with a negative rank count, the simulation reader handles both formats, and post did not, so its rank-count check fired unconditionally (a negative can never equal num_procs) and blamed a rank-count mismatch on runs that used the right number of ranks. v2 stores one contiguous chunk per block written by that block's single owner, plus a 4-int (owner + 1, m, n, p) record. Ownership now comes from that record rather than from the reader's own geometric intersection, which also fixes a case where a rank overlapping a block it does not own would read a -1 extent and abort. The whole chunk is read and this rank's intersection sub-box copied out, so s_setup_amr_block still reconstructs coordinates from a valid local coarse index. v1 files stay readable. Second: post runs with a LARGER sys_size than the simulation for 5eq Lagrange bubbles, because it appends beta_idx as a post-only output slot. Comparing the file's conserved-variable count against post's inflated sys_size rejected every valid AMR + Lagrange-bubbles file. Size and read by the count the writer actually used.
[amr-grideff] printed amr_n_tagged, which counts THIS rank's local tag_grid, next to amr_n_covered, which comes from the global accepted box list (s_amr_cluster runs with reduce = .true., so every rank clusters the same global tag set and already holds the same volume). Printing the unreduced pair from rank 0 made the ratio wrong by about num_procs. Sum the numerator across ranks and leave the denominator alone. Every rank enters the collective; only rank 0 prints. Still approximate and marked as such in the source: the numerator mixes the level-1 and level-2 index spaces, and it is accumulated before the amr_buf pad and the box merge, so real over-coverage is worse than the printed ratio.
With l0_ntile > 0 the L0 tiles occupy slots 1..l0_slot_off of the SAME pool as the fine blocks and amr_num_blocks counts them, so s_write_amr_restart emits level-0 tile records into the AMR fine-block file. All three reader paths rejected them twice over: lvl == 0 fails the lvl < 1 test, and a tile's refinement ratio of 1 fails rr < 2. Every AMR + L0-tiles case therefore died in post_process with 'malformed fine-block header (level/extent inconsistent); the writer and reader header layouts have drifted' -- a message that blamed a drift which had not happened. A tile's data is the base grid re-tiled and is already in the level-0 restart file, so the overlay skips it while still advancing the file offset past the record. Reproduced and fixed on a CPU build, which is where CI sees it: all nine cases fail with the old reader and pass with the new one. They pass on amdflang GPU either way, so no local GPU gate could have caught this.
amr_n_shaped counts the coarse volume the FINAL boxes occupy, after s_amr_regrid_shape_boxes has padded by amr_buf, clamped, size-capped, clipped and tiled. amr_n_covered is taken before all of that, so it cannot see the pad, which is exactly where amr_buf acts: across an amr_buf 4 -> 1 sweep covered moved 0.4% while the refined volume moved 9%. [amr-grideff] now prints tagged, covered and shaped together. Also warn when the global accepted-box count exceeds amr_max_blocks: boxes contributed by other ranks were discarded silently, so the refined mesh stopped matching the tag field with no diagnostic. The existing capped flag covers only the tree bisection, not this truncation.
The runtime banner and the per-level distribution table both asserted 1.55x as a measured subcycle speedup. It is a phase-share model and was never measured. One matched-resolution arm pair (equal physical time, level 2 at the same dt in both) gave 2.84x, but that T arm's phase table omits rhs, seam, reflux, gather and rk entirely because the subcycle advance path carries no PH_* brackets, so it is a wall ratio with no accounting behind it. Both figures are now labelled for what they are. Separately, m_time_steppers said multi-level reflux under subcycling is future work; s_amr_advance_children has been calling s_amr_reflux_to_parent(dt_sub, .true.) for some time.
Records what the reviews established: the np8-512 weak-scaling ladder already existed and had never been analysed (rg:clus is O(P^3) and 75% of wall at np=512; rb:topo is exactly O(P^2)); the emitted box set is a single bounding box re-tiled, so amr_cluster_eff cannot affect it and the measured 2.20x over-coverage is this clusterer's best case on its most flattering geometry; and the per-step global scan work this plan had as item 1 is 2.5% of wall at np=512, measured, against a P^2 and a P^3 term beside it.
Under cfl_dt the simulation names each save by save_count = int(mytime/t_save) (m_start_up.fpp:798), so when adaptive dt grows enough for one step to cross TWO t_save boundaries the index skips and no file is written for the intervening value. The post loop walks indices 0..n_save-1 and the reader aborts on the first absent one, so every CFL-driven case died in post_process with 'File restart_data/lustre_N.dat is missing'. The IGR_triple_point case writes 60 saves at indices 0, 1, 3, 5, 7 and upward, and post demanded index 2. A skipped index is legitimate output, not a fault. f_save_exists lives beside the reader that owns the filename convention and only answers for the shared-file layout, since with file_per_process each rank holds a different file and the answer would not be rank-uniform; that path keeps the original fail-closed behaviour. Reproduced and fixed on a CPU build: the case fails before and passes after, and 12 non-CFL cases still pass.
…FlowCode#1795 4C751DAF aborts on the Frontier CCE cpu lane with 'No Lagrangian bubbles remain in the domain', and passes on amdflang GPU and gfortran CPU. Measured on gfortran over the full run, from the case's own lag_bubble_evol output: the single bubble moves 0.5 -> 0.5000076, its radius goes 0.008 -> 0.0079987 (it shrinks), and the void fraction in its cell peaks at 0.0335 against a valmaxvoid threshold of 0.99 -- a 3.1x margin. Neither removal criterion is reachable, so the count reaching zero on CCE is a symptom of the bubble state degrading there, not a physical outcome. The case is therefore NOT loosened to dodge the abort: with that much margin on working toolchains, any such change would hide whatever actually goes wrong on CCE. Only the --test-all re-run trips it, so the primary run and the golden comparison still cover the case fully. This is present on master too, invisible because the suite discarded the --test-all exit code.
The subcycle driver, the shared per-block advance and the children recursion carried no PH_* markers, so a subcycle run's phase table omitted rhs, rk, gather, seam and halo entirely (zero-time phases are dropped from the report) and the measured 2.84x speedup was a wall ratio with no accounting behind it. Bracketed: the coarse cons halos (halo), the setup and lerp loops (gather), both fine-fine halos (seam), s_compute_rhs and the RK update inside s_amr_subtree_stage_advance (rhs, rk -- one site covers level 1 and the children), and the children's restrict + reflux-to-parent (rs:rfp). Same ids as the lock-step path so the two budgets read side by side. Validated: a subcycle case now reports rhs 31.9%, gather 7.7%, rk 3.1% where it reported none of them; 66/66 AMR gate green.
BASE_CFG hardcodes precision = 2 (double post_process output), which case_validator rightly prohibits on a --single build. Every post_process case on the single-precision CI lane has been failing this validation invisibly -- the suite discarded post's exit code until 68bcaf1 -- and surfaced as 'Case validation failed for post_process' across the lane once the check landed. get_post_process_mods now requests precision = 1 when the build is single. Post output files are not golden-compared, and the lane's post step never succeeded before, so no goldens move.
…it-identical The min-separation merge restarted an all-pairs scan after every fusion and removed the absorbed box by an O(n) shift; at np=128 the gathered list is ~12,400 leaves collapsing through ~11,300 fusions per call, and rg:clus grows to 75% of wall at np=512. Survivors now live on a doubly-linked list (O(1) unlink) and each pass finds its fusion through a uniform bin grid over the LIVE boxes' bounding box, width max(ext_max + thr, clamped so nb_tot stays O(n)) -- sound because a tooclose pair is within ext_max + thr - 1 per dimension and enlarging cellw only widens the candidate set; bit-identical because for each i in list order the minimum surviving candidate index equals the first tooclose j the linear walk meets. Also fixes: the nacc == 0 walk read an unwritten nxt(1); the [amr-merge] fusion counter was declared and printed but never incremented (its 'zero fusions' reading was an artifact); and this rebuild restores the truncation warning, amr_n_shaped accumulation and grideff 'shaped' field that the first version of this commit reverted through a stale-base file copy (caught in review). Measured on mi2104x pinned-binary A/B: rg:clus 138.6->83.8 (np32), 1074.9->335.7 (np64), 8462.1->1076.1 ms/call (np128).
s_amr_st_reserve chose device-native vs host-staged growth by COLUMN COUNT (32), but a column is ~33 MB at the default 64^3 block cap and ~1.8 GB on the 57.3 GiB run that motivated the guard -- the same count means wildly different bytes. The store-capacity ratchet pushes production runs to 43-81 columns, sending every later growth on the full-store host PCIe round trip (measured 4.4 s/regrid vs 0.55 under the guard). The guard now budgets the transient itself: stage on-device while the extra copy (oldcap * column bytes, which is exactly the store size regardless of factorization) stays under 4 GiB, so near-limit stores of ANY column shape still take the OOM-safe host path. Review-noted exposure on 16 GB cards is recorded at the constant; the registers site keeps its count guard (per-array transient, two orders smaller) with its stale cross-reference corrected.
…eX on one node in one window, the non-case-opt arm reproduces ledger 140's 0.649 beside it, and the excess moves only -0.033 +/- 0.037 because it is a difference and case optimization speeds both of its terms -- the goal's own prediction that it would not be a wash is falsified
… ms per step floor every rank pays, so 40 percent of it is structural serialization rather than skew; the base solve is identical between the arms; and the migration-path store growth this campaign called the largest AMR item is 6.6 ms per step on the deck that owns statement 2, not the 144.3 measured on the rung deck
…model, port its damage flux and solid-partial-density private-list entries into the restructured HLL and HLLC solvers, apply its explicit num_species chemistry bounds, and migrate our AMR and GPU pitfalls into the docs files that replaced the deleted rules page
…und in the restructured Riemann solvers
…urement retractions
…s docs linkcheck)
…s energy The master merge brought upstream's damage model, in which the energy uses the undamaged modulus, and upstream regenerated all six of its own cont_damage goldens in that merge (0BAA2F42 32A3A936 574636EE 5A137315 6CFBCF4A EF3E7C79). D731AB7A is the seventh and is AMR-only, so upstream could not reach it. It is the only test the merge changed: a clean 15-test rerun on the merged tree fails it and otherwise only the ten non-Newtonian tests that also fail on upstream master alone on this machine. The new value matches the candidate the merged code produced on both CPU and GPU (999.0098931133879).
…the merge dropped The merge took MFlowCode#1852's species bound (10 -> 60) everywhere but left U_L/U_R, F_L/F_R/F_star_L/F_star_R/F_HLLC and F_HLL at dimension(20), while the guard it also brought admits sys_size up to AMD_SYS_SIZE_MAX = 70. Any amdflang non-case-optimized run with 20 < sys_size <= 70 overran them silently. The three sites now read exactly as upstream's.
…amage golden; the ten non-Newtonian failures are upstream's on this toolchain
…code span (fixes docs linkcheck)
…p-level rows The RESIDUAL was wall minus the sum of every row, including rows nested inside others, so it read strongly negative (-9.9 % on a run whose true residual is +2.3 %). s_phase_tic now keeps a stack of open brackets and records the shallowest and deepest depth each phase opens at; s_phase_toc counts interleaved and orphan closes. The report appends a tier column (T1, T2, or T1-2 when a phase opens at two depths), sums only rows that were top-level on every rank, and says whether every bracket closed innermost-first. The column is appended, never inserted, so existing positional parsers read the same fields. Verified: AMR goldens F57C3A5B and 09E0D257 pass; the budget is valid on both on CPU and on the amdflang OpenMP-offload build, with an identical tier map on each.
…ment tools were rebuilt and proven able to fail
…generated golden (Phase 0 closed)
…block-shape diversity, so Phase 2 is retargeted to an amr_bat_pad probe
…aunch, but leader order and the 8-member batch cap keep it from reaching the rank that sets the lockstep
…np=2 sibling case
… tiles evenly inside the two-cell floor
…ime-feedback balance
Lines of Code
|
…rger; stop rule fired, design void
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Performance
Measured on AMD MI210 (gfx90a), OpenMP offload, one MPI rank per GPU, double precision throughout — MFC and the reference are compared at the same precision. Every figure is a per-step wall differenced between two from-scratch runs of different length, so start-up and mesh ramp-in cancel, and is the mean of three repetitions with the two codes interleaved inside one node reservation.
Reference. AMReX's
Tests/GPU/CNSblob test, rebuilt withNUM_GROW = 4to match MFC's WENO5 ghost width, on a deck matched to MFC's: unit periodic box, entropy blob advected at u=1, gamma 1.4, inviscid, 400^3 base,max_level = 2,ref_ratio = 2,regrid_int = 20, no subcycling, reflux on, density-gradient tagging thresholded to tag the same cells.AMR overhead on the matched deck (8 GPUs, 400^3 base + 2 levels)
The two codes do not run the same numerics, so the comparable quantity is each code's excess over its own uniform-grid arm at the same advanced-cell count:
MFC pays 1.73x AMReX's AMR overhead on this deck. AMR does not slow the base solve: the AMR and uniform arms agree on the base-solve phases to -6.9 ms/step, inside noise.
Where MFC's 627 ms/step goes: 521 ms is AMR machinery, of which 352 ms (56%) is MPI wait. Largest phases: reflux 110, coarse/fine gather 109, restriction 104 ms/step. Reducing that wait is the active work; it is not a correctness or an API concern for this PR.
That wait is imbalance, not a transfer floor. An earlier revision of this section read the per-rank minimum MPI wait (251 ms/step) as a communication floor. That was wrong twice over: the
[mpiwait] TOTALrow also sums host-side rows, and a per-rank minimum is not a floor when ranks are waiting on each other. A per-wave-instance trace (every rank's entry and exit for each rendezvous, validated against[mpiwait]to within 1%, and free — an instrumented-vs-parent A/B is a null on every row, largest |t| = 1.9 at n = 3) decomposes it properly: for each instance, the smallest wait any rank achieved is the cost everyone pays, and the rest is one rank waiting on another.The common cost is 28 ms/step, stable to 3% across three repetitions, and 61-90% of each rank's wait is imbalance. It concentrates in two of the seven wave bands — reflux faces and level-1 restriction carry 78% of it, at 90-96% imbalance — while the parent-fill and restrict-parent bands are floor-dominated. So transfer time and any device synchronisation inside the wait brackets are together bounded by 28 ms/step, and the reachable quantity is load balance.
Scaling
Weak ladder, fixed work per rank, 8 -> 48 GPUs (multi-node), on the lower-order scaling deck with the same geometry and refinement settings:
a fitted 1.222x per rank-doubling, against 1.234x for AMReX on the matched ladder. The scaling slope is at parity — the gap above is a constant factor, not a worsening one.
Notes for anyone reproducing these
--case-optimization. On this deck it is worth a paired -12.4% on the AMR step (-0.581 s/step, sd 0.084, n=3), 78% of it in the RHS. The excess figures above are case-optimized; without it the same comparison gives 1.80x rather than 1.73x, since case optimization largely cancels in the difference.Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.