We are experiencing singular matrices when solving mulisegment wells
sometimes. In that case (here during
BlackoilModelEbos::assembleReservoir <-
BlackoilModelEbos::initialLinerization <-
BlackoilModelEbos:::nonlinearIterationNewton ) an exception is thrown
when updating the controls of a well.
The problem here is that this exception only happens on one
process. That one goes to the catch block in
NonLinearSolverEbos::step, marks the nonlinear solve as failed and
cuts the time step. The others move to the collective communication
below. Somehow and somewhen all end up in a non-matching collective
communication with different data types and we get an MPI Error that
the message was truncated.
Now all processes will throw, terminate the nonlinear solver and cut
the timestep as it should be.
Using bool here is at least frowned upon. To be honest, I have no idea
what happens underneath here if we pass a bool. In contrast to other
pod types we do not associate it with a builtin type of MPI (not even
sure what to use). Hence we probably create a custom type for sending
and receiving. That should work. But I have no idea what will be used
for summation.
BTW: I am debugging a case that previously crashed and now suddenly
works and this seems to be the only relevant change I made in the
meantime.
We did only break in the loop for rank 0 and not the other ones. Hence
all other processes kept iterating beyond the maximum number of
allowed iterations. This lead to hard to find crashes because of
non-matching MPI communication.
Previously, we did a global summation of the size of the
well_perf_data vector to determine the number of perforations
of a well. In the case of distributed wells this will try to access
more perforations than stored for the well in well_perf_data and hence
might use data from cells that actually are not perforated by this
cell. Note that for well not distributed the code worked as the
summation has no effect.
This commit changes this to only query peforations on the
local process. This should be enough to fix this problem.
In addition it removes the computation of connpos which is never used.
A correct MPI program should do that.
As MPI_Finalize is part of the destructor of MainObject we need to
make sure that it's destructor is called before the return statement.
We do that manually by resetting the unique_ptr that we now use to
store the MainObject,
Instead of unconditionally issuing MPI_Abort if we encounter a fatal
exception, we try to test whether all processes have experienced this
exception and if this is the case just terminate nomally with a exit
code that signals an error. We still use MPI_Abort if not all
processes get an exception as this is the only way to make sure that
the program aborts.
This approach also works around issues in some MPI implementations
that might not correctly return the error.
Multiple messages like this are gone now:
```
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[smaug.dr-blatt.de:129359] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[smaug.dr-blatt.de:129359] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
```
Bu we still see something like this:
```
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[35057,1],0]
Exit code: 1
--------------------------------------------------------------------------
```
Changes
```
Program threw an exception: [/home/mblatt/src/dune/opm/opm-simulators/opm/simulators/timestepping/AdaptiveTimeSteppingEbos.hpp:586] Solver failed to converge after cutting timestep 11 times.
```
to
```
Simulation aborted: Solver failed to converge after cutting timestep 11 times.
```
Which seems more user friendly.
We now give a different (more explanatory) error if the input grid has
no active cells at all. Often this points to a user error and having a
more precise error message might help here.
There is a strange interaction when using MPI and OpenMP on some
hardware/MPI implementations. I a serial run omp_get_num_procs() would
return the number of processors but when started under mpirun it would
always return 1.
With this we now allow users to use any amount of threads.
While we reported that we used the number of threads that were passed
on the command line, we never really used it for OpenMP but always
sticked to two unless environment variable OMP_NUM_THREADS was set.
Note that because the ThreadManager in opm-models would always use the
command line option and hence the linearizer would use that number of
threads.
Please note that the only use of OpenMP in opm-common (volume
calculation in EclipseGrid) is not effected by this as it happens
before we set the number of OpenMP threads.
This hopefully will prevent breaking things when further work for
PINCH is done.
The following tests are still known to fail with current master:
T2A_NOPINCH T2A_GAP T2A_NOGAP T4A1_NOGAP T4A1_GAP
We need to make sure that all pods are initialzed when using
the default constructor. Fixes
```
RateConverter.hpp:83:50: warning: ‘phaseUsage’ may be used uninitialized in this function [-Wmaybe-uninitialized]
83 | , attr_ (rmap_, Attributes())
| ^
```
We rely on parallel hdf5 for a parallel build of OPM flow. Otherwise
compilation will fail. Hence if this is a parallel build but only a
serial hdf5 library is found, we now issue an informative warning and
deactivate hdf5/parallel restart.
ClosesOPM/opm-common#3405
Markus Blatt