Decorates inline helpers in blackoil*modules.hh, blackoilprimaryvariables.hh,
directionalmobility.hh, tpfalinearizer.hh, NewTranFluxModule.hpp,
TemperatureModel.hpp and BlackOilEnergyIntensiveQuantitiesGlobalIndex.hpp
with OPM_HOST_DEVICE, replaces a few raw `throw`s with OPM_THROW, and
switches static `FluidSystem::` accesses to instance-based
`fluidState.fluidSystem()` / `getFluidSystem()` so the same code works
when the fluid system lives on the device. Test
test_gpu_linear_two_phase_material.cu picks up the matching template
adjustments.
When a reservoir-coupling master group's guide-rate-distributed
production target exceeds the slave's reported potential, the slave
cannot deliver the requested rate and the shortfall is lost from the
parent group's total. Cap each capacity-limited group's target at
the slave's potential and redistribute the surplus to sibling master
groups via the standard FractionCalculator / localReduction
machinery, so the parent (typically FIELD) can still meet its
overall target.
The redistribution is implemented in
RescoupConstraintsCalculator::calculateMasterGroupConstraintsAndSendToSlaves
and is structured as four phases:
Pre-phase: restore master-group control modes from the Schedule
(restoreMasterGroupControlsFromSchedule_); switch master groups
of inactive slaves to individual control so they do not consume
a share of the parent's target
(excludeInactiveSlaveMasterGroupsFromDistribution_); recompute
GCW and FIELD target reduction (updateGCWAndTargetReductions_).
Phase 1: compute initial guide-rate-distributed targets per
master group via GroupConstraintCalculator.
Phase 2: cap-and-redistribute. Groups whose target exceeds the
slave-reported potential are capped, switched to individual
control, and the FIELD reduction is recomputed. Uncapped
groups are then re-evaluated and absorb the surplus. Finally
all master groups are switched to individual control with their
final allocated targets.
Phase 3: send the (target, cmode, per-rate-type limits) triples
to each activated slave.
The driver runs on every rank of the master communicator: its
helpers call collective MPI operations
(updateGroupControlledWells does comm_.sum, GroupConstraintCalculator
and updateGroupTargetReduction walk GroupStateHelper which is
collective). The slave-facing MPI sends in
sendSlaveGroupConstraintsToSlave_ are rank-0-only inside the
underlying helpers, so the Phase-3 loop is safe to enter from every
rank. A trailing GroupState::communicate_rates(comm) aggregates any
per-rank-partial reduction rates so the post-redistribution state is
consistent across master ranks.
Supporting changes:
* GroupStateHelper::updateGroupControlledWells -- conditional GCW
for master groups: GCW=1 under FLD/NONE control, GCW=0 under
individual control. Without this the calculator cannot
distinguish between capped groups (excluded from distribution)
and uncapped groups (absorbing the surplus).
* BlackoilWellModel::updateGroupControls -- guard at the top that
skips master groups, plus a new isReservoirCouplingMasterGroup
forwarding method on the RescoupProxy facade. Master-group
controls and targets must be read-only outside the constraint
calculator; the guard prevents checkGroupHigherConstraints and
updateGroupIndividualControl from modifying them during the
assemble() Newton iteration.
Known limitations, deferred to follow-up work:
* A master group with GCONPROD CONTROL_MODE set to a
rate mode (ORAT, WRAT, ...) AND RESPOND_TO_PARENT = YES has both
an own rate limit and parent control available. The current GCW
logic returns 0 in this case and excludes the group from the
FractionCalculator guide-rate sum, so Phase 1's localFraction
yields a wrong target. Typical RC setups have master groups
acting as pure routing nodes (CONTROL_MODE = FLD or NONE) and are
unaffected.
* Injection -- the cap-and-redistribute pass currently applies only
to production master groups. Per-phase injection extension
requires a slave-reported injection potential on
SlaveGroupInjectionData (currently absent) plus the parallel set
of helpers.
* Finished slaves -- the inactive-slave handling covers only the
not-yet-activated case. A slave run can finish before the master;
detecting that finish is a separate PR with its own MPI-protocol
changes.
Adds host-side allocator helpers, host/device-decorated raw-pointer
accessors and small additions to MiniVector that the rest of the GPU
intensive-quantities work depends on. Updates test_gpu_smart_pointers.cu
to cover the new behaviour.
and make sure that we always search for ParMETIS in the *prereq.cmake file
This file gets processed when running CMake for opm-simulators and
during a find_package(opm-simulators) call.
Previously this was done via config.h, but due to the recent build
system refactoring these defines became MIA and some code is never
activated even if it would be supported by the hardware.
We now set the necessary compile definitois as properties for the relevant
source files to fix this.
Document the class, the constructor, and the member functions whose
semantics are not self-evident from the name; add brief annotations
to the member variables describing role, ownership, and lifecycle.
Class-level docs cover the per-report-step loop, what the class owns
(solver, tuning-update callback, report aggregation, convergence-output
thread, OPMRST, reservoir-coupling bring-up), and how it is instantiated.
Member functions documented individually: the constructor, destructor,
registerParameters, both run() overloads (RC and non-RC),
checkRunningAsReservoirCouplingMaster, both init() overloads,
updateTUNING, updateTUNINGDP, runStep, finalize, serializeOp,
loadState, saveState, getHeader, createSolver.
Outdated comments removed/rewritten.
No behavior change.
This is what we did before refactoring the build system. Somehow this
was accidentally skipped during the refactoring and this commit
reintroduces it.
This is needed, because without it we will always directly agglomerate
the data on the coase levels to on process instead of fewer and fewer.
This should make a difference in scalability of the linear solver when
using many processes.
Move all template-method definitions out of
SimulatorFullyImplicitBlackoil.hpp into a new
SimulatorFullyImplicitBlackoil_impl.hpp, mirroring the layout
already used by AdaptiveTimeStepping, BlackoilModel, and most
template classes under opm/simulators/.
The .hpp now contains only declarations (with Doxygen) plus
trivial inline accessors (grid, model, eclState, schedule,
isRestart, wellModel_, getCellMapping). The header shrinks from 754
to 223 lines, making the public API of the class easier to skim
as new methods are added.
Pure mechanical refactor: method bodies are moved verbatim
modulo template<class TypeTag> and qualifier prefixes.
Document the three classes, the TuningUpdateCallback alias, and the
member functions whose semantics are not self-evident from the name.
No behavior change.
Rename the `dt` and `timeStep` parameters of the tuning-update
callback and its passthroughs to `substep_length` and
`sub_step_number`. The old names were generic enough to hide the
"substep vs report-step" distinction whose confusion caused the
bug fixed in #6998: the callback fires once per substep and its
length/index arguments are specifically substep-scoped, not
report-step-scoped.
Restructure the rescoup master's outer substep loop so that each
outer iteration is one master actual time step. The master's
adaptive timestepping now drives the sync cadence instead of
saturating at the slave report-step size. This matches the
coupling semantics of the reference simulator, where
master<->slave exchanges are tied to master actual time steps
rather than to slave report steps.
Introduce two sync modes, selected by the
--rescoup-sync-at-report-steps CLI flag added in the previous
commit:
TSYNC (default, flag=false)
Each outer iteration is one master time step. Sync-step
length is chosen fresh each iteration from the master's
adaptive suggestion, capped at the max time step and at the
remaining report-step time, then chopped against slave
report dates.
RSYNC (flag=true)
Each chunk between slave report-step boundaries is one sync
step. Preserves the pre-change OPM behavior.
Both modes share a single outer-loop function; the mode-specific
bits (step-length selection, user-visible log line, chop against
slave reports) are encapsulated in the new private helper
`getRcMasterSyncStepLength_`.
Add a new boolean parameter `RescoupSyncAtReportSteps` (default `false`)
to the time-stepping parameter set, registered with a developer-facing
description. Read it in the `ReservoirCouplingMaster` constructor,
store it as a member, expose it via a `syncAtReportSteps()` getter, and
log the chosen mode once at simulation start.
This is a no-op precursor for the upcoming change to the master's outer
sync loop. The flag is read and made available, but no code consults
it yet, so behavior is unchanged on both values.
Future use: when `true`, the master will sync with its slaves at slave
report-step boundaries (the current OPM behavior). When `false`, the
master will sync at every master actual time step.
The tuning_updater callback expects (elapsed, dt, sub_step_number)
where dt is the length of the upcoming substep and sub_step_number
is 0 at the start of a report step. All three call sites in
runStepOriginal_ and the reservoir coupling variants were passing
outer SimulatorTimer values instead: The report-step length (Sites
A, C), the sync-step length (Site B), and the report-step index
(Site A).
Use suggestedNextTimestep_() and /*substep=*/0 at all three sites,
matching the pattern already in use at the initial tuningUpdater
invocation in SimulatorFullyImplicitBlackoil::runStep and inside
maybeUpdateTuningAndTimeStep_.
Impact is gated by WCYCLE being active in the deck. The TUNING
branch of the callback is unaffected (TUNING_CHANGE and
TUNINGDP_CHANGE are already cleared by the initial call in
SimulatorFullyImplicitBlackoil::runStep). The WCYCLE branch
re-runs with a too-wide look-ahead horizon, causing
WCYCLE::nextTimeStep to over-chop and invoke
updateNEXTSTEP(wcycle_time_step) when no chop was warranted. The
wrong sub_step_number in runStepOriginal_ also bypasses the
REQUEST_OPEN_WELL guard for every report step after the first.
Regression introduced in b94868633 when the callback signature
grew three parameters as part of reconciling the impl file with
the WCYCLE-support PR (#5792).
Add OPM_TIMEFUNCTION() to the eight MPI exchange entry points between
the master and slave processes in reservoir coupling, so the per-call
cost of each exchange is visible in Tracy traces.
Functions instrumented (four pairs):
master -> slave sendMasterGroupConstraintsToSlaves
slave -> master receiveGroupConstraintsFromMaster
slave -> master sendSlaveGroupDataToMaster
master -> slave receiveSlaveGroupData
slave -> master sendNextReportDateToMasterProcess
master -> slave receiveNextReportDateFromSlaves
master -> slave sendNextTimeStepToSlaves
slave -> master receiveNextTimeStepFromMaster
The macros are no-ops unless built with USE_TRACY_PROFILER=ON, so this
adds no overhead in normal builds.
Motivated by upcoming work to change the rescoup sync cadence from
per-slave-report-step to per-master-actual-step. The new cadence
increases the number of MPI exchange events per simulation, so the
ability to measure per-exchange cost is needed to verify the overhead
stays acceptable.
Kjetil Olsen Lye
Tobias Meyer Andersen
Tor Harald Sandve
Håkon Hægland
Bård Skaflestad
Kai Bao
Markus Blatt
Arne Morten Kvarving
Atgeirr Flø Rasmussen
hnil
Tor Harald Sandve