and make sure that we always search for ParMETIS in the *prereq.cmake file
This file gets processed when running CMake for opm-simulators and
during a find_package(opm-simulators) call.
Previously this was done via config.h, but due to the recent build
system refactoring these defines became MIA and some code is never
activated even if it would be supported by the hardware.
We now set the necessary compile definitois as properties for the relevant
source files to fix this.
Document the class, the constructor, and the member functions whose
semantics are not self-evident from the name; add brief annotations
to the member variables describing role, ownership, and lifecycle.
Class-level docs cover the per-report-step loop, what the class owns
(solver, tuning-update callback, report aggregation, convergence-output
thread, OPMRST, reservoir-coupling bring-up), and how it is instantiated.
Member functions documented individually: the constructor, destructor,
registerParameters, both run() overloads (RC and non-RC),
checkRunningAsReservoirCouplingMaster, both init() overloads,
updateTUNING, updateTUNINGDP, runStep, finalize, serializeOp,
loadState, saveState, getHeader, createSolver.
Outdated comments removed/rewritten.
No behavior change.
This is what we did before refactoring the build system. Somehow this
was accidentally skipped during the refactoring and this commit
reintroduces it.
This is needed, because without it we will always directly agglomerate
the data on the coase levels to on process instead of fewer and fewer.
This should make a difference in scalability of the linear solver when
using many processes.
Move all template-method definitions out of
SimulatorFullyImplicitBlackoil.hpp into a new
SimulatorFullyImplicitBlackoil_impl.hpp, mirroring the layout
already used by AdaptiveTimeStepping, BlackoilModel, and most
template classes under opm/simulators/.
The .hpp now contains only declarations (with Doxygen) plus
trivial inline accessors (grid, model, eclState, schedule,
isRestart, wellModel_, getCellMapping). The header shrinks from 754
to 223 lines, making the public API of the class easier to skim
as new methods are added.
Pure mechanical refactor: method bodies are moved verbatim
modulo template<class TypeTag> and qualifier prefixes.
Document the three classes, the TuningUpdateCallback alias, and the
member functions whose semantics are not self-evident from the name.
No behavior change.
Rename the `dt` and `timeStep` parameters of the tuning-update
callback and its passthroughs to `substep_length` and
`sub_step_number`. The old names were generic enough to hide the
"substep vs report-step" distinction whose confusion caused the
bug fixed in #6998: the callback fires once per substep and its
length/index arguments are specifically substep-scoped, not
report-step-scoped.
Restructure the rescoup master's outer substep loop so that each
outer iteration is one master actual time step. The master's
adaptive timestepping now drives the sync cadence instead of
saturating at the slave report-step size. This matches the
coupling semantics of the reference simulator, where
master<->slave exchanges are tied to master actual time steps
rather than to slave report steps.
Introduce two sync modes, selected by the
--rescoup-sync-at-report-steps CLI flag added in the previous
commit:
TSYNC (default, flag=false)
Each outer iteration is one master time step. Sync-step
length is chosen fresh each iteration from the master's
adaptive suggestion, capped at the max time step and at the
remaining report-step time, then chopped against slave
report dates.
RSYNC (flag=true)
Each chunk between slave report-step boundaries is one sync
step. Preserves the pre-change OPM behavior.
Both modes share a single outer-loop function; the mode-specific
bits (step-length selection, user-visible log line, chop against
slave reports) are encapsulated in the new private helper
`getRcMasterSyncStepLength_`.
Add a new boolean parameter `RescoupSyncAtReportSteps` (default `false`)
to the time-stepping parameter set, registered with a developer-facing
description. Read it in the `ReservoirCouplingMaster` constructor,
store it as a member, expose it via a `syncAtReportSteps()` getter, and
log the chosen mode once at simulation start.
This is a no-op precursor for the upcoming change to the master's outer
sync loop. The flag is read and made available, but no code consults
it yet, so behavior is unchanged on both values.
Future use: when `true`, the master will sync with its slaves at slave
report-step boundaries (the current OPM behavior). When `false`, the
master will sync at every master actual time step.
The tuning_updater callback expects (elapsed, dt, sub_step_number)
where dt is the length of the upcoming substep and sub_step_number
is 0 at the start of a report step. All three call sites in
runStepOriginal_ and the reservoir coupling variants were passing
outer SimulatorTimer values instead: The report-step length (Sites
A, C), the sync-step length (Site B), and the report-step index
(Site A).
Use suggestedNextTimestep_() and /*substep=*/0 at all three sites,
matching the pattern already in use at the initial tuningUpdater
invocation in SimulatorFullyImplicitBlackoil::runStep and inside
maybeUpdateTuningAndTimeStep_.
Impact is gated by WCYCLE being active in the deck. The TUNING
branch of the callback is unaffected (TUNING_CHANGE and
TUNINGDP_CHANGE are already cleared by the initial call in
SimulatorFullyImplicitBlackoil::runStep). The WCYCLE branch
re-runs with a too-wide look-ahead horizon, causing
WCYCLE::nextTimeStep to over-chop and invoke
updateNEXTSTEP(wcycle_time_step) when no chop was warranted. The
wrong sub_step_number in runStepOriginal_ also bypasses the
REQUEST_OPEN_WELL guard for every report step after the first.
Regression introduced in b94868633 when the callback signature
grew three parameters as part of reconciling the impl file with
the WCYCLE-support PR (#5792).
Add OPM_TIMEFUNCTION() to the eight MPI exchange entry points between
the master and slave processes in reservoir coupling, so the per-call
cost of each exchange is visible in Tracy traces.
Functions instrumented (four pairs):
master -> slave sendMasterGroupConstraintsToSlaves
slave -> master receiveGroupConstraintsFromMaster
slave -> master sendSlaveGroupDataToMaster
master -> slave receiveSlaveGroupData
slave -> master sendNextReportDateToMasterProcess
master -> slave receiveNextReportDateFromSlaves
master -> slave sendNextTimeStepToSlaves
slave -> master receiveNextTimeStepFromMaster
The macros are no-ops unless built with USE_TRACY_PROFILER=ON, so this
adds no overhead in normal builds.
Motivated by upcoming work to change the rescoup sync cadence from
per-slave-report-step to per-master-actual-step. The new cadence
increases the number of MPI exchange events per simulation, so the
ability to measure per-exchange cost is needed to verify the overhead
stays acceptable.
Replace updateSlaveGroupTargetsInSchedule() (which created
synthetic GCONPROD/GCONINJE) with markSlaveGroupsInSchedule()
which sets lightweight boolean flags on slave groups.
The flags make isProductionGroup()/isInjectionGroup() return
true so the existing constraint enforcement gates pass. The
actual target values are still provided by the master via
hasMasterProductionTarget()/hasMasterInjectionTarget().
Requires the companion opm-common commit that adds the flag
mechanism to Group and Schedule.
We do some special things in the last write out (e.g. writing RSM
file). Previously, that was only done if we reached the last report
step. With an EXIT in an executed ACTIONX the simulation end is
reached earlier and we missed some output, though.
With this change we perform the same write out, if there was an
ACTIONX/EXIT, as for a regular write out at the last report step.
After receiving constraints from the master, the slave
updates its Schedule with synthetic GCONPROD/GCONINJE entries
so the existing constraint enforcement mechanisms recognize
the groups and apply the master's targets. Without this,
slave groups without deck-level GCONPROD/GCONINJE would be
skipped by isProductionGroup()/isInjectionGroup() gates in
checkGroupHigherConstraints, getWellGroupTargetProducer, etc.
Requires the companion opm-common PR that adds
Schedule::updateSlaveGroupProductionTarget() and
Schedule::updateSlaveGroupInjectionTarget().
Kjetil Olsen Lye
Bård Skaflestad
Markus Blatt
Kai Bao
Arne Morten Kvarving
Håkon Hægland
Tor Harald Sandve
Atgeirr Flø Rasmussen
hnil
Tor Harald Sandve
Antonella Ritorto
Stein Krogstad