When running Norne with the interleaved solver sometimes exceptions
(diagonal matrix block is not invertible) occur for some rows in the
ILU decomposition. In a parallel run this means that some, but not all
processes will see the exceptions. This leads to a classic deadlock.
With this commit we catch the exception during the setup of the preconditioner,
determine with the other processes whether there were any exceptions, and
in this case all the processes will throw an exception.
Currently this limited to Dune::MatrixBlockError, but we could extend this.
Previously we initialized a variable for a global
(i.e. sum over all processes) value with the local
value first and the overwrote it with the computed
global. This meant the name did not reflect the value
during the first initialization.
With this commit we fix this by using an additional
variable for the local value that is used to compute
the global one.
If this option was set there were some branches in
the code that did depend on the local number of wells
but should depend on the number of wells in the reservoir
no matter on which process they are stored.
With this commit we introduce BlackOilModelBase::localWellsActive()
which only takes local wells into account. The function now
BlackOilModelBase::wellsActive() considers all active wells in the
reservoir.
The last parameter of this functions specifies how vector/martix
entries there are per index/cell of the grid. For the interleaved
versions all components end up in one block. Therefore this number
needs to be one here (in contrast to the number of phases for the
non-interleaved version).
This commit new uses the correct number.
It holds a reference to the sequential code. Previously this reference
point to a temporary object that was deleted upon exit of
constructPrecond.
With this commit use a unique_ptr to store the parallel
preconditioner. It also gets a custom deleter that will delete the
nested sequential iterator during destruction.
The class is not limited to parallel preconditioners. To reflect this
we rename it to AdditionalObjectDeleter. As it will also be used for
the parallel interleaved version we move the class to a separate header.
Previously, we tried to make an empty container bigger by
resizing it with its current size. Of course this is wrong and does
not change anything. With this commit we use the size of another
container which already has the correct size.
For g++-4.4 support for nullptr is not complete. Using it
to initialize an empty shared_ptr breaks compilation:
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/AutoDiffHelpers.hpp: In constructor ‘Opm::HelperOps::HelperOps(const Grid&, Opm::EclipseStateConstPtr)’:
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/AutoDiffHelpers.hpp:87: error: no match for ternary ‘operator?:’ in ‘(bool)eclState ? eclState->.Opm::EclipseState::getNNC() : nullptr’
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/AutoDiffHelpers.hpp:87: note: candidates are: operator?:(bool, const Opm::NNC* std::__shared_ptr<const Opm::NNC, (__gnu_cxx::_Lock_policy)2u>::*, const Opm::NNC* std::__shared_ptr<const Opm::NNC, (__gnu_cxx::_Lock_policy)2u>::*) <built-in>
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/AutoDiffHelpers.hpp:87: note: operator?:(bool, const Opm::NNC* std::__shared_ptr<const Opm::NNC, (__gnu_cxx::_Lock_policy)2u>::*, T*) <built-in>
Here no conversion to shared_ptr happens.
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/ImpesTPFAAD.cpp: In constructor ‘Opm::HelperOps::HelperOps(const Grid&, Opm::EclipseStateConstPtr) [with Grid = UnstructuredGrid]’:
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/ImpesTPFAAD.cpp:160: error: no matching function for call to ‘std::shared_ptr<const Opm::EclipseState>::shared_ptr(const dune_nullptr_t&)’
/usr/include/c++/4.4/bits/shared_ptr.h:1263: note: candidates are: std::shared_ptr<_Tp>::shared_ptr(std::shared_ptr<_Tp>&&) [with _Tp = const Opm::EclipseState]
/usr/include/c++/4.4/bits/shared_ptr.h:1238: note: std::shared_ptr<_Tp>::shared_ptr() [with _Tp = const Opm::EclipseState]
/usr/include/c++/4.4/bits/shared_ptr.h:1236: note: std::shared_ptr<const Opm::EclipseState>::shared_ptr(const std::shared_ptr<const Opm::EclipseState>&)
Here the DUNE's nullptr implementation is used with std::shared_ptr which are not
compatible.
We fix this by using the emptry constructor of std::shared_ptr to create a nullptr.
The whole system consists out of three equations per cell. Using
the parallel index set of the grid cells for it is wrong. Therefore
we use PR OPM/opm-core#803 to set up an additional parallel index set
for the whole system and use this for the communication that is needed e.g.
by the ILU preconditioner.
This commit adapts the PR #375 for parallel runs. That is, the norms are
calculated over all wells, not just the ones that perforate the local grid
cells.
As this is a reduction, too, we move the computation to convergenceReduction
method.
In a parallel the well handling needs to be slightly
adapted, see PR OPM/opm-core#799. Therefore, we need to
pass a boolean flag to the well manager indicating whether
this is a parallel run or not.
With this commit we do this.
This method seems to be some leftover from refactoring.
In flow we now use computeResidualNorms() instead and
residualNorm() is not used anywhere. Therefore this commit
removes it to prevent confusion.
As it turns out initializing the Geology on a distributed grid
result in wrong values for e.g. saturation. Therefore with this
commit we resort to initializing the global geology and distribute
it using communication.
Previously, we only did send the information for the
first phase which led to computing wrong values in
a simulation. With this patch we now send all the data
for all the phases.
Previously only passed the parallel information to the ILU preconditioner,
but of course needs this information to set up the communication, too.
With this commit we pass the parallel information object to AMG's constructor.
Previously we used the size of the communicator within the CpGrid to check
whether we are running in parallel and need to redistribute the grid.
Unfortunately, this is MPI_COMM_SELF until we actually loadbalance and redistribute.
Therefore we now use the size of the MPI_Helper (i.e. MPI_COMM_WORLD) which
gives us the number of all available processes.
Note that the wrong behaviour was provoked with 656e5de331. Before that we
redistributed in any case which luckily included runs with more than 1 process.
During BlackoilStateDataHandle::gather we did read values from the
state where we should only receive values to. With this commit we
read from the state where we should send values.
Kudos to Bard for noticing this.
As a result we only have one faceCentroid(int) function the returns const double* and
FieldVector<double,3> for UnstructuredGrid and CpGrid, respectively. The codes is
adapted to this.
For historic (or no apparent) reason the free function grid interface
was added to opm-autodiff. As it depends on whether or not dune-cornerpoint
is present this seems rather unnatural. Therefore this commit removes the
functionality unconditionally from opm-autodiff.
Note that there is a corresponding commit in dune-cornerpoint that adds it
there.
If oscillating then the same measures have to be taken on all
processes. This was not the case before the commit. Instead only
process 0 would have adapted omegas and we have produced wrong
results.
Kudos to Atgeirr for detecting this while reading the code.
With now generic implementation of the initStateEquil in opm-core
we added the necessary grid helper functionlality for CpGrid and activated
the processing if the EQUIL keyword is there.
Previously BlackoilPropsDataHandle did hold a grid for sending
and receiving that were either not used or we could prevent their
usage. Therefore this commit removes them from the class and queries
all needed information from the property objects.
Previously, we had to use two layers of overlap cells such the
innermost layer contains the rightvalues automatically (as it is
surrounded by internal edges). No we use communication to get
the correct values in the whole overlap region and one layer
suffices as it should.
With this commit we add the possibility to start with a global representation
of a simulator that is read on each process and afterwards this presentation
is redistributed among the processors together with the properties and
state data needed to initialize the simulation.
There still is no parallel well handling and no parallel output. But with the
equilibrium example of @dr-robertk and deactivated output we can already
perform parallel runs.
In the parallel simulator we will have to be able adress only poperties on
some part of the global grid. To create thos properties we need to be able
to copy the grid independant data of the properties object and resize the rest.
This commit adds a construct taking a properties object for reading and a
new number of cells to accomplish this.
During the constructor the underlying object only holds smart
pointers and an empty vector. The FullyImplicitBlackoilSolver
obtains a reference to it from the NewtInterationInterface instances.
Therefore copying boost::any and storing it by value should be cheap
and safe.
We need it serveral places and all of them seem to have access to
NewtonIterationBlackoilInterface. This makes it natural to give access
to it and prevent users from having to forward it manually at several
places in the simulator driver.
As with opm-core we use boost::any to provide additional
information about a parallel run. It is used to set a
ParallelISTLInformation object and and fill it with the
information obtained from a parallel Cpgrid.
Note that the simulator currently compiles sucessfully. Still,
we have to test the runs and do debugging.
Apparently Eigen cannot handle empty containers during reserve correctly.
Therfore we check for the size of the vector and if it is zero simply create
empty jacobians.
Closes#290
Previously, we just called the default constructor. Of course this does not allocate
any memory and we experienced segmentation faults. With this patch we correctly
allocate the arrays by passing the number of row and columns to the constructor.
It took me quite some time to understand the computations done
e.g. during the detection of oscillations, where the stuff returned
by residuals() is used as a vector of doubles. It turns out that
residuals() actually returns the norm of the residuals. To clarify this
we rename residuals() to computeResidualNorms() and residuals to
residual_norms. Having my dare devil day today, I even try to document the
method. (This documented method might feel kind of lonely between the others,
now;). Hopefully this saves others some time.
The computations made to check the convergence are the same for all existing
phases. Therefore this patch uses loops over the phase indices when cmputing
them,
In the convergence check there are several reductions (maxCoeff(), sum())
that will trigger communication in a parallel run. This patch seperates the
reductions from the other computations. The idea is to one reduction for the
reductions that need to done as global communication is expensive.
The initial definition of the phase indices seems to be in
opm/core/props/BlackoilPhases.hpp. Nevertheless there were
several redefinitions of the same or similar enums (either
Aqua, Liquid, and Vapor, or Water, Oil, and Gas). Surprisingly
most often these definitions did not use the original values.
This is bound to break if there is a change upstream.
This patch limits the definition to one place in opm-autodiff,
namely opm/autodiff/BlackoilPropsAdInterface.hpp. To avoid
downstream confusion we define both the Water and Aqua triplets.
In addition we define the maximum number of phases to use at compile
time.
Previously, there were a completely different check for UnstructuredGrid
and CpGrid. The latter probably not working properly due to missing
support in the DUNE grid interface.
This commit resorts the code to use the UgGridHelpers abtraction instead of
UnstructuredGrid's legacy grid interface. Thus one code can be used to do
the checks for both grids. For UG the code compiles and the test is successful.
For CpGrid it compiles but of course the test still fails due to missing
transmissibility multiplier support. This will be sorted out in a different PR.
Markus Blatt