That the local ids were consecutive and starting from 0 was just
a coincidence and they should never be used to access linear systems
or vectors. This commit fixes this by using the correct mappers instead.
Note the we removed some computations from the constructor of
ISTLSolverEbosCpr as it inherits from ISTLSolverEbos and the operations
already happnen in constructor of the base class.
There are field properties that can usually be queried even if they
are not explicitly specified in the input
file (e.g. PVTNUM). Unfortunately, the ParallelEclipseState cannot
forsee which of these will be queried at startup and after the
loadbalancing only the master process is able to auto creates
these (easily). Hence this commit uses a fall-back if an unstored
keyword is queried. In this case we use get_global-* to auto create
the keyword and use functions of the cartesian mapper to extract the
relevant values on the process.
Of course this temporarily wastes space and we might want to resort to
a more memory savy approach later.
With the arrival of compressed field properties there is no need
to extract the global arrays just to compress them manually
afterwards. This change should remove commununication and
synchronization points.
There is not reason to it as a vector of cartesian size as we access
its entries by the compressed element index. This should save space
and speedup the lookup.
They are attached to the cells as well and are now distributed
during CpGrid::loadBalance. Due to this change we also rename
FieldPropsDataHandle to PropsCentroidsDataHandle.
The created data handle for the communication could in theory be used
with other DUNE grids, too. In reality we will need to merge with the
handle that ALUGrid already uses to communicate the cartesian indices.
This PR gets rid of using the get_global_(double|int) method in
ParallelEclipseState and reduces the amount of boilerplate code there.
It seems like the VERSION_GREATER_EQUAL operator for boolean
expressions was introduced after CMake 3.6 and hence the current
check whether to activate CUDA or not is broken in version 3.6 and
below.
This PR fixes this by using VERSION_GREATER.
Closes#2375.
We experienced weired linker errors when using host compiler version for
compilation that were not supported by the nvcc used to compile the
cuda code:
```
[ 15%] Linking CXX executable bin/test_timer
/usr/bin/ld: /home/mblatt/src/dune/opm-2.6/opm-common/opm-seq/lib/libopmcommon.a(Parser.cpp.o): in function `Opm::(anonymous namespace)::file& std::vector<Opm::(anonymous namespace)::file, std::allocator<Opm::(anonymous namespace)::file> >::emplace_back<std::filesystem::__cxx11::path&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::filesystem::__cxx11::path&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)':
Parser.cpp:(.text+0x1096): undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
/usr/bin/ld: Parser.cpp:(.text+0x10ad): undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'
/usr/bin/ld: /home/mblatt/src/dune/opm-2.6/opm-common/opm-seq/lib/libopmcommon.a(Parser.cpp.o): in function `Opm::(anonymous namespace)::ParserState::loadFile(std::filesystem::__cxx11::path const&)':
Parser.cpp:(.text+0x23a1): undefined reference to `std::filesystem::canonical(std::filesystem::__cxx11::path const&)'
/usr/bin/ld: Parser.cpp:(.text+0x24e0): undefined reference to
`std::filesystem::__cxx11::path::_M_split_cmpts()'
```
The reason turned out to be that the library path was build up by
paths of the old (g++-7) compiler used by nvcc and the actual (newer) compiler
g++-8. This completely messed up the linker paths for CMake.
To detect this situation already when running cmake we have resorted
to first setting the CMAKE_CUDA_FLAGS to force cmake to make nvcc use
the host compiler and to activate CUDA (if available) before calling
`find_package(CUDA)`. If the host compiler is not supported CMake will
error out during `enable(CUDA)`
Note that we still use (deprecated) FindCUDA later to determine the
libraries to link to.
The users has the option to either deactivate CUDA by setting
`-DCMAKE_DISABLE_FIND_PACKAGE_CUDA=ON` or to use a compiler supported
by nvcc (setting `-DCMAKE_CXX_COMPILER=compiler`).
Additionally we do not try to activate CUDA the CMake version is <
3.8. Please note that previously CMake would have errored out here
anyway since we used the unsupported `enable_language(CUDA)` even in
this case.
Closes#2363.
This fixes name clashes with DUNE's own factory that is introduced
in DUNE 2.7. Hence it closes issue #2266.
BTW: Dune's factory has more template parameters than ours.
... instead of a reference This is needed as we only want to read the
full deck and construct the EclipseGrid only on the master process
with rank 0. Hence all other ranks will pass a nullptr to this
function. This will be possible with this move from a reference to a
pointer.
packSize now uses a reference to map/unordered_map to prevent copying.
In addition we always specify all template parameters for the containers to
allow for sending data if non-default ones are used.
At that stage flow did not set the number of thread in the case where
OMP_NUM_THREADS is not defined. That happens in setupParallelism which
is called when the simulator is run. Hence we have to manually compute
the number of threads similar to there.
When filterConnections_ is called the grid is not load
balanced, yet. Currently that means that grid() will also return the
unbalanced grid and all processes will see the whole global grid.
We will change semantics of the unbalanced grid soon: Only the root
process will see the whole grid and the others will see an empty
partition of it. Hence filtering on this partition will remove all
connections on all wells in the schedule for non-root processes and
produce wrong results.
For non-root process the filtering needs to be done on the load
balanced grid. This is accomplished by this commit.
In the case with active solvent (BlackoilWellModel::has_solvent_ is
true, e.g. case 2D_OILWATER_POLYMER.DATA) and one process have no
wells while another has some this lead to numComponents returning
different values. Later this lead to truncated message in MPI_Allgather.
The underlying problem was the usage of numWells() which returns just
the number of local wells. We resort to using wellsActive which
returns whether there any well active regardless which process knows it.
This removes a deadlock experienced for some models
where we have specified connections to non-active cells.
On non-IO ranks we are using the local grid since in the
future there will be no global grid available. Wells connecting
cells not on these processors are neglected anyway.
Closes#2101
This is needed to support dune-fem where the local mapper might
be
`MultipleCodimMultipleGeomTypeMapper<GridView<GridPart2GridViewTraits<AdaptiveLeafGridPart<CpGrid, (PartitionIteratorType)4,
false> > >, Dune::Impl::MCMGFailLayout>`
as opposed to the global one being
`MultipleCodimMultipleGeomTypeMapper<GridView<DefaultLeafGridViewTraits<CpGrid>>, Dune::Impl::MCMGFailLayout>`.
Closes#2095.
Since the indexMaps do not contain the global element index anymore
(but the global id). The old code did not work anymore.
Unfortunately, we are using CpGrid specific functions (scatterData)
to get the mapping. Therefore this might be broken if other grids are
used.
Previously, it was still assumed that all ranks knew the global grid
and each map on CollectDataToIORank::indexMaps_ was a mapping of
send/receive index to the index of the cell using the mapper of the
corresponding global grid.
With this patch inside of CollectDataToIORank::DistributeIndexMapping
indexMaps is a mapping from send/receive index to global cartesian
index until the destructor is run. Inside of the destructor of the
iorank the remapping to mapped index of the global grid happens and
the ranks array is computed.
This at least slightly improves the old design. In that design the
subclass had no own constructor but inherited the one of the base class.
That base class constructor called certain subclass
functions (createGrids_, filterConnections_, updateOutputDir_, and
finalizeInit_)that would initialize raw pointers of the
subclass. Hence subclasses where not allowed to have non-pod members
and those used later (e.g. deleted in the destructor) had to be
initialized in these functions.
The new (still ugly) design introduces constructors into the
subclasses and skips inheriting constructors. Now one must call a base
class function classImplementationInit which will still call the
functions createGrids_, filterConnections_, updateOutputDir_, and
finalizeInit_, but at least at this point the baseclass is fully
constructed and the subclass is constructed as much as
possible/needed (non-pod types will be initialized now.)
With the change of CPgrid to only holding the grid on one process
it will be an empty grid on all other processes. This has really
strange side effects like Schedule::filterConnections removing all
well perforations on theses processes.
This at least slightly improves the old design. In that design the
subclass had no own constructor but inherited the one of the base class.
That base class constructor called certain subclass
functions (createGrids_, filterConnections_, updateOutputDir_, and
finalizeInit_)that would initialize raw pointers of the
subclass. Hence subclasses where not allowed to have non-pod members
and those used later (e.g. deleted in the destructor) had to be
initialized in these functions.
The new (still ugly) design introduces constructors into the
subclasses and skips inheriting constructors. Now one must call a base
class function classImplementationInit which will still call the
functions createGrids_, filterConnections_, updateOutputDir_, and
finalizeInit_, but at least at this point the baseclass is fully
constructed and the subclass is constructed as much as
possible/needed (non-pod types will be initialized now.)
In the case where two were direct vertical neighbors in the grid
but not in the underlying cartesian grid (e.g. because of MINPV or
pinch outs), we treated them as NNCs and wrote the transmissibilty
to TRANNC.
With this patch we detect this situation (two neighbor cells with identical
i and i and no active cells between them) and do not create an NNC
in the eclipse output files but write the transmissibility to TRANZ.
This change will print the encountered unknown keywords to std::cerr
(albeit in the format ThisKeyword and not --this-keyword) and abort
before running the simulator.
It seems like eclipse ignores NNCs with small transmissibility.
Small means less than 1e-6 for Eclipse (Even if it says that it
is ignoring values below 1e-5 and/or zero values)!.
This commit now implements the same threshold during IO.
Also fixes a bug when applying EDITNNC, it needs to have cell1<=cell2 to work.
Previously the vector of NNCData was passed in as a reference and sorted.
Unfortunately, it needed to be transformed later to meet all prerequisites.
With this commit we do this transformations in sortNncAndApplyEditnnc.
Furthermore EDITNNC data is passed by value as it is not needed
outside and should usually not be to big. It was copied outside anyway!
We first add all NNCs specified in the deck to the ouput
and then determine additional NNCs by iterating over all faces that
connect cells that are not connected in the underlying cartesian grid.
Therefore we need to make sure that we do not output NNCs twice
and for faults that also have a specified NNC we need to substract
the transmissibility specified via NNC.
Unfortunately, we first created NNC with applied EDITNNC and then
still used the original NNC data to set the transmissibility. Thus
we were actually ignoring EDITNNC.
This commit fixes this by using the data structure that has EDITNNC
applied.
There is a problem compiling upstream
Hierarchy::recalculateGalerkin() without MPI. Therefore we
reimplement that method in our AMG version. Fixes will travel
upstream.
Apart from that we made sure that all types have a meaning even
if there is no MPI found and that we at least have redistribute method
to call even if using SequentialInformation. In that case the method
does nothing but prevents compilation errors complaining about missing
parallel functionality.
Markus Blatt