Previously, local averages were calculated and used in the
well equations. With this commit we add versions of defineState and
calcAverages that take into account the parallel domain decomposition
and calculate correct averages.
Function calcAverages has a boolean template parameter
indicating whether this is a parallel run. Additionally we introduce
AverageIncrementCalculator with the same boolean template parameter.
In a parallel run we check whether the cell is owned by the process and
only in this case return an increment bigger than zero. In a sequential run
(no MPI or just one process -> empty boost::any parameter) no overhead is
introduced.
As it is just used to compute the tempV value there is no need to
copy it to a new vector. With commit we use a reference to
geo.poreVolume() instead to prevent the copy.
It needs a material law manager that works on the processes local
grid partition. Therfore copying the one from the global representation
does not work. Instead we now create an new SaturationPropsFromDeck and
initialize it with the local material law manager.
Since the refactoring to using opm-material a material law manager for
the global grid was used. This meant that the properties used for
elements of the local grid were wrong. With this commit we set up a
manager that is based on the local grid only.
When running parallel one cannot use Eigen::Array::mean() for this
as the it is just a local part of the complete array. With this commit
we correctly compute the number of global cells in the grid and use this
together with a parallel reduction to compute a global mean value.
Somehow I was mislead by the name rock properties that the data stored in
them is not associated to grid cells, but to rock regions. With this commit
we determine the porosity and permeability by communication.
There is a using `BaseType::wellMap` directive that redirects all the
well map accesses to the base class. In consequence the local wellMap_
is alway empty and just makes debugging harder. Therefore it is
removed in this commit.
In the rare occasion that there are no wells int the model the wells_ pointer in
BlackoilModelBase class is a null pointer. Therefore we need to test whether it is
null and only process well information if it is not.
This problem was reintroduced with PR #460 and gets fixed by this
one. No we can run the equilibrium examples without wells again.
Sorry for the inconvenience.
There might be wells without any perforations. It it happens
to be the last well will supscript over the bounds. In other
cases we actually return the correction for the next well.
Not sure whether returning 0 makes any sense, though.
When running Norne with the interleaved solver sometimes exceptions
(diagonal matrix block is not invertible) occur for some rows in the
ILU decomposition. In a parallel run this means that some, but not all
processes will see the exceptions. This leads to a classic deadlock.
With this commit we catch the exception during the setup of the preconditioner,
determine with the other processes whether there were any exceptions, and
in this case all the processes will throw an exception.
Currently this limited to Dune::MatrixBlockError, but we could extend this.
Previously we initialized a variable for a global
(i.e. sum over all processes) value with the local
value first and the overwrote it with the computed
global. This meant the name did not reflect the value
during the first initialization.
With this commit we fix this by using an additional
variable for the local value that is used to compute
the global one.
If this option was set there were some branches in
the code that did depend on the local number of wells
but should depend on the number of wells in the reservoir
no matter on which process they are stored.
With this commit we introduce BlackOilModelBase::localWellsActive()
which only takes local wells into account. The function now
BlackOilModelBase::wellsActive() considers all active wells in the
reservoir.
The last parameter of this functions specifies how vector/martix
entries there are per index/cell of the grid. For the interleaved
versions all components end up in one block. Therefore this number
needs to be one here (in contrast to the number of phases for the
non-interleaved version).
This commit new uses the correct number.
It holds a reference to the sequential code. Previously this reference
point to a temporary object that was deleted upon exit of
constructPrecond.
With this commit use a unique_ptr to store the parallel
preconditioner. It also gets a custom deleter that will delete the
nested sequential iterator during destruction.
The class is not limited to parallel preconditioners. To reflect this
we rename it to AdditionalObjectDeleter. As it will also be used for
the parallel interleaved version we move the class to a separate header.
Previously, we tried to make an empty container bigger by
resizing it with its current size. Of course this is wrong and does
not change anything. With this commit we use the size of another
container which already has the correct size.
For g++-4.4 support for nullptr is not complete. Using it
to initialize an empty shared_ptr breaks compilation:
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/AutoDiffHelpers.hpp: In constructor ‘Opm::HelperOps::HelperOps(const Grid&, Opm::EclipseStateConstPtr)’:
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/AutoDiffHelpers.hpp:87: error: no match for ternary ‘operator?:’ in ‘(bool)eclState ? eclState->.Opm::EclipseState::getNNC() : nullptr’
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/AutoDiffHelpers.hpp:87: note: candidates are: operator?:(bool, const Opm::NNC* std::__shared_ptr<const Opm::NNC, (__gnu_cxx::_Lock_policy)2u>::*, const Opm::NNC* std::__shared_ptr<const Opm::NNC, (__gnu_cxx::_Lock_policy)2u>::*) <built-in>
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/AutoDiffHelpers.hpp:87: note: operator?:(bool, const Opm::NNC* std::__shared_ptr<const Opm::NNC, (__gnu_cxx::_Lock_policy)2u>::*, T*) <built-in>
Here no conversion to shared_ptr happens.
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/ImpesTPFAAD.cpp: In constructor ‘Opm::HelperOps::HelperOps(const Grid&, Opm::EclipseStateConstPtr) [with Grid = UnstructuredGrid]’:
/home/mblatt/src/dune/opm/opm-autodiff/opm/autodiff/ImpesTPFAAD.cpp:160: error: no matching function for call to ‘std::shared_ptr<const Opm::EclipseState>::shared_ptr(const dune_nullptr_t&)’
/usr/include/c++/4.4/bits/shared_ptr.h:1263: note: candidates are: std::shared_ptr<_Tp>::shared_ptr(std::shared_ptr<_Tp>&&) [with _Tp = const Opm::EclipseState]
/usr/include/c++/4.4/bits/shared_ptr.h:1238: note: std::shared_ptr<_Tp>::shared_ptr() [with _Tp = const Opm::EclipseState]
/usr/include/c++/4.4/bits/shared_ptr.h:1236: note: std::shared_ptr<const Opm::EclipseState>::shared_ptr(const std::shared_ptr<const Opm::EclipseState>&)
Here the DUNE's nullptr implementation is used with std::shared_ptr which are not
compatible.
We fix this by using the emptry constructor of std::shared_ptr to create a nullptr.
The whole system consists out of three equations per cell. Using
the parallel index set of the grid cells for it is wrong. Therefore
we use PR OPM/opm-core#803 to set up an additional parallel index set
for the whole system and use this for the communication that is needed e.g.
by the ILU preconditioner.
This commit adapts the PR #375 for parallel runs. That is, the norms are
calculated over all wells, not just the ones that perforate the local grid
cells.
As this is a reduction, too, we move the computation to convergenceReduction
method.
In a parallel the well handling needs to be slightly
adapted, see PR OPM/opm-core#799. Therefore, we need to
pass a boolean flag to the well manager indicating whether
this is a parallel run or not.
With this commit we do this.
This method seems to be some leftover from refactoring.
In flow we now use computeResidualNorms() instead and
residualNorm() is not used anywhere. Therefore this commit
removes it to prevent confusion.
As it turns out initializing the Geology on a distributed grid
result in wrong values for e.g. saturation. Therefore with this
commit we resort to initializing the global geology and distribute
it using communication.
Previously, we only did send the information for the
first phase which led to computing wrong values in
a simulation. With this patch we now send all the data
for all the phases.
Previously only passed the parallel information to the ILU preconditioner,
but of course needs this information to set up the communication, too.
With this commit we pass the parallel information object to AMG's constructor.
Previously we used the size of the communicator within the CpGrid to check
whether we are running in parallel and need to redistribute the grid.
Unfortunately, this is MPI_COMM_SELF until we actually loadbalance and redistribute.
Therefore we now use the size of the MPI_Helper (i.e. MPI_COMM_WORLD) which
gives us the number of all available processes.
Note that the wrong behaviour was provoked with 656e5de331. Before that we
redistributed in any case which luckily included runs with more than 1 process.
During BlackoilStateDataHandle::gather we did read values from the
state where we should only receive values to. With this commit we
read from the state where we should send values.
Kudos to Bard for noticing this.
As a result we only have one faceCentroid(int) function the returns const double* and
FieldVector<double,3> for UnstructuredGrid and CpGrid, respectively. The codes is
adapted to this.
For historic (or no apparent) reason the free function grid interface
was added to opm-autodiff. As it depends on whether or not dune-cornerpoint
is present this seems rather unnatural. Therefore this commit removes the
functionality unconditionally from opm-autodiff.
Note that there is a corresponding commit in dune-cornerpoint that adds it
there.
If oscillating then the same measures have to be taken on all
processes. This was not the case before the commit. Instead only
process 0 would have adapted omegas and we have produced wrong
results.
Kudos to Atgeirr for detecting this while reading the code.
With now generic implementation of the initStateEquil in opm-core
we added the necessary grid helper functionlality for CpGrid and activated
the processing if the EQUIL keyword is there.
Previously BlackoilPropsDataHandle did hold a grid for sending
and receiving that were either not used or we could prevent their
usage. Therefore this commit removes them from the class and queries
all needed information from the property objects.
Previously, we had to use two layers of overlap cells such the
innermost layer contains the rightvalues automatically (as it is
surrounded by internal edges). No we use communication to get
the correct values in the whole overlap region and one layer
suffices as it should.
With this commit we add the possibility to start with a global representation
of a simulator that is read on each process and afterwards this presentation
is redistributed among the processors together with the properties and
state data needed to initialize the simulation.
There still is no parallel well handling and no parallel output. But with the
equilibrium example of @dr-robertk and deactivated output we can already
perform parallel runs.
In the parallel simulator we will have to be able adress only poperties on
some part of the global grid. To create thos properties we need to be able
to copy the grid independant data of the properties object and resize the rest.
This commit adds a construct taking a properties object for reading and a
new number of cells to accomplish this.
During the constructor the underlying object only holds smart
pointers and an empty vector. The FullyImplicitBlackoilSolver
obtains a reference to it from the NewtInterationInterface instances.
Therefore copying boost::any and storing it by value should be cheap
and safe.
We need it serveral places and all of them seem to have access to
NewtonIterationBlackoilInterface. This makes it natural to give access
to it and prevent users from having to forward it manually at several
places in the simulator driver.
As with opm-core we use boost::any to provide additional
information about a parallel run. It is used to set a
ParallelISTLInformation object and and fill it with the
information obtained from a parallel Cpgrid.
Note that the simulator currently compiles sucessfully. Still,
we have to test the runs and do debugging.
Markus Blatt