This computation is serial and needs a complete representation
of data attached to all preforations (even those stored on
another process). This commit uses the newly created factory to
correctly compute the connection densities for distributed wells.
Some of our computations are heavily serial and need a complete
representation of the data attached to all perforation no matter
whether a perforation lives on the local partition or not. This commit
adds a factory that allows to easily create such a representaion and
helps writing data back to the local representation.
As this is as sequential (ordering matters!) as it can get we need to
communicate all perforations, do the partial sum with them and save
result back to the local perforations.
Well rates of distributed wells might be stored on multiple processes
but should be summed only once. Hence only the owner does the
summation with this commit.
This adds an utility that creates a vector of all above values for
the local perforations. For distributed wells this is needed as the
perforation above might live on another processor. We use the parallel
index sets together with the global index of the cells that are
perforated.
The B matrix is basically a component-wise multiplication
with a vector followed by a parallel reduction. We do that
reduction to all ranks computing for the well to save the
broadcast when applying C^T.
BlackoilWellModel now stores an instance of this class for each
well. Inside that class there is a custom communicator that only
contains ranks that will have local cells perforated by the well.
This will be used in the application of the distributed well operator.
This is another small step in the direction of distributed wells,
but it should be safe to merge this (note creation of the custom
communicators is a collective operation in MPI but done only once).
IMHO this might have happened if perf_data_ is empty
or if the last connection is closed. (Discovered while
working on distributed wells but might happen already
before!)
that simplifies the code a bit and will work with
distributed wells. Previously, we assumed that all
non-shut perforations are stored locally. That does
not hold any more.
Only after rank zero does the filtering the schedule the well
definitions in there are guarateed to have no perforations to inactive
cells. Therefore we broadcast the schedule another time to publish
this to all processes.
Previously, we did the filtering locally on these processes bit that
did also remove perforations to cells that are active globally but
not locally. That seems very hard to work with when allowing
distributed wells.
It would remove perforated cells from wells that cross the local
domain's border. That would make it impossible to figure out the
first connection. In addition we would not be able to check that
the connections exist (as rank 0 would have the complete information
-> inconsistency).
In serial we use the first cell of the first well to determine the
pvt region index for a group. Previously, we used the first cell of
the first local well in a parallel run. Unfortunately that may lead
to different pvt region indices being used for the same goup on
different processes.
We fix this by using the same approach in parallel as we already use
in serial. For this we use Well::seqIndex() to determine the needed
ordering.
and use it in the WellInterface instead of creating a vector
with these indices there. The original approach recreates
information in another path of the well and assumes that all
connections are in a process's local partition. That assumption
does not hold any more for distributed wells.
In OPM the matrix graph might be unsymmetric as we do not store
the full sparsity pattern for copy rows but only the diagonal.
Unfortunately, DUNE assumes that matrices from finite elements and
finite volumes have a symmetric sparsity pattern for copy rows to
and uses this assumption to create the graphs for PTScotch/ParMETIS
more easily. But PTScotch/ParMetis assume a symmetric graph.
nvcc exits compilation if the header dune/istl/basearray.hh (form DUNE
2.6) is included as it does not seem to understand the friend declaration
there (friend class for a struct).
```
/usr/include/dune/istl/basearray.hh:101:49: error: ‘typename Dune::Imp::base_array_unmanaged<B, A>::RealIterator’ names ‘template<class B, class A> template<class T> struct Dune::Imp::base_array_unmanaged<B, A>::RealIterator’, which is not a type
friend class RealIterator<const ValueType>;
^
```
Previously, we exported an unordered map containing all names of
wells that are not present in the local part of the grid.
As we envision to have wells that are distributed across multiple
processors, this information does not seem to be enough. We need
to be able to set up communication for each well. To do this we need
to find out who handles perforations of each well.
We now export a full list of well name together with a boolean
indicating whether it perforates local cells (vector of pair of string
and bool).
Markus Blatt