Executable is named flow_distribute_z and uses the external
loadbalancing information. It can be used to test the distributed
standard wells on SPE9 with 4 or more processes.
You can use EclCpGridVanguard::setExternalLoadBalancer() to
set an external funtion that creates a vector of integers (containing
the partition for each cell) from the grid. If it is set then this
information will be used for loadbalancing, otherwise ZOLTAN.
We introduce a new parameter --enable-distributed-wells=<true|false>
for this. During startup we check that the model either only has
standard wells or that multisegement wells are actively interpreted
as standard wells (by way of passing --enable-multisegment-wells=false
as an option).
This computation is serial and needs a complete representation
of data attached to all preforations (even those stored on
another process). This commit uses the newly created factory to
correctly compute the connection densities for distributed wells.
Some of our computations are heavily serial and need a complete
representation of the data attached to all perforation no matter
whether a perforation lives on the local partition or not. This commit
adds a factory that allows to easily create such a representaion and
helps writing data back to the local representation.
As this is as sequential (ordering matters!) as it can get we need to
communicate all perforations, do the partial sum with them and save
result back to the local perforations.
Well rates of distributed wells might be stored on multiple processes
but should be summed only once. Hence only the owner does the
summation with this commit.
This adds an utility that creates a vector of all above values for
the local perforations. For distributed wells this is needed as the
perforation above might live on another processor. We use the parallel
index sets together with the global index of the cells that are
perforated.
The B matrix is basically a component-wise multiplication
with a vector followed by a parallel reduction. We do that
reduction to all ranks computing for the well to save the
broadcast when applying C^T.
BlackoilWellModel now stores an instance of this class for each
well. Inside that class there is a custom communicator that only
contains ranks that will have local cells perforated by the well.
This will be used in the application of the distributed well operator.
This is another small step in the direction of distributed wells,
but it should be safe to merge this (note creation of the custom
communicators is a collective operation in MPI but done only once).
IMHO this might have happened if perf_data_ is empty
or if the last connection is closed. (Discovered while
working on distributed wells but might happen already
before!)
that simplifies the code a bit and will work with
distributed wells. Previously, we assumed that all
non-shut perforations are stored locally. That does
not hold any more.
Only after rank zero does the filtering the schedule the well
definitions in there are guarateed to have no perforations to inactive
cells. Therefore we broadcast the schedule another time to publish
this to all processes.
Previously, we did the filtering locally on these processes bit that
did also remove perforations to cells that are active globally but
not locally. That seems very hard to work with when allowing
distributed wells.
It would remove perforated cells from wells that cross the local
domain's border. That would make it impossible to figure out the
first connection. In addition we would not be able to check that
the connections exist (as rank 0 would have the complete information
-> inconsistency).
In serial we use the first cell of the first well to determine the
pvt region index for a group. Previously, we used the first cell of
the first local well in a parallel run. Unfortunately that may lead
to different pvt region indices being used for the same goup on
different processes.
We fix this by using the same approach in parallel as we already use
in serial. For this we use Well::seqIndex() to determine the needed
ordering.
and use it in the WellInterface instead of creating a vector
with these indices there. The original approach recreates
information in another path of the well and assumes that all
connections are in a process's local partition. That assumption
does not hold any more for distributed wells.
In OPM the matrix graph might be unsymmetric as we do not store
the full sparsity pattern for copy rows but only the diagonal.
Unfortunately, DUNE assumes that matrices from finite elements and
finite volumes have a symmetric sparsity pattern for copy rows to
and uses this assumption to create the graphs for PTScotch/ParMETIS
more easily. But PTScotch/ParMetis assume a symmetric graph.
nvcc exits compilation if the header dune/istl/basearray.hh (form DUNE
2.6) is included as it does not seem to understand the friend declaration
there (friend class for a struct).
```
/usr/include/dune/istl/basearray.hh:101:49: error: ‘typename Dune::Imp::base_array_unmanaged<B, A>::RealIterator’ names ‘template<class B, class A> template<class T> struct Dune::Imp::base_array_unmanaged<B, A>::RealIterator’, which is not a type
friend class RealIterator<const ValueType>;
^
```
Markus Blatt