These are not aware whether they are parallel or not. Therefore
the user should only specify a verbose verbosity only on one rank.
With this commit we do this on the process responsible for terminal
output.
Previously, we kind of hard coded the problem using the TypeTag system.
Instead of this we now simply pass the only additional thing needed, the
ElementContext, as an additional template parameter.
Removes the include of removed header BlackoilModelEbosTypeTags.hpp.
detail::getConvergence computes a lot more than we actually need.
We only need the average over B and the maximum of the well residual.
Therefore we now compute these quantities in getWellConvergence and
do the global reductions manually.
We will need the typetag information also for the wells.
If it is not in a separate header we get problems
with recursive inclusion of the headers (BlackoilEbos.hpp
includes the header that also needs the typetag information).
Using cachedIntensiveQuantities on parallel grids will cause/is causing
dereferencing a null pointer here. Therefore we resort to iterating over
the grid and using the element Context.
If this turns out ot be performance regression @andlaus owes me a beer!
Using cachedIntensiveQuantities on parallel grids will cause/is causing
dereferencing a null pointer here. Therefore we resort to iterating over
the grid and using the element Context.
If this turns out ot be performance regression @andlaus owes me a beer!
Closes#1110
The cat must have dragged that in during some of the various rebases of this branch.
This introduced a segmentation fault as for the second setup eclIO was already null.
Currently, all parallel DUNE grid store some cells in addition to
interior cells. Therefore assuming that the global number of cells
(i.e. the number of cells a sequential grid needs to cover the same
whole domain with indentical cells) is not the sum of the number of
cells of the local grid. Previously, the latter was used.
Before this commit only the solution of process 0 was written.
To fix this we make the equilGrid of Ebos available. It is used
for the output writer. The properties written initially are gathered from
all processes first using the new gather/scatter utility.
Dense case was handled in PR #1002. I was not aware that the same
regression was in the legacy code. wells_manager.c_wells() might return
a null pointer if there are no wells in the domain of a process.
This was (accidentally?) introduced in c880efae5b (labelled
"mostly eliminate Eigen in the FIP and VFP code"). As it is neither
present in flow_ebos nor in flow_mpi and was previously removed in
ccec17f8 (labelled "Fixed missing unit conversion for total fluid in place")
I assume that this is a bug.
Up to now parallel runs aborted due to an exception with
the message "FIP not yet implemented for MPI". With this
commit we do the computation in parallel, too. And flow_ebos
runs a bit longer...
In that case `wells_manager.c_wells()` returns a null pointer
which made `updateListEconLimited` segfault. With this commit
we treat a null wells_struct as having zero wells.
With this this and #1001 opm-data/equilibrium at least runs
through in parallel. Results will be checked next.
In upstream they expect to be used with FielMatrix as
the block type only. Thus it it is impossible to use AMG
with a direct coarse solver within OPM. With this patch
it would be possible if we did not use field type float.
Previously only the cell data registered with SimulationDataContainer
war gathered during parallel output. User requested data was neglected
and a warning was issued. With this commit we intialize the local view
of data::Solution on all processes with the data registered in
SimulationDataContainer and add cell data requested by the user. This is
then gathered on the IO process, and used for the output layer. To
rudimentarily support matlab we also create a global view of
SimulationDataContainer for it.
There is already a generic one with the Grid implementation
being the template parameter in the header file. The other
implementation using UnstructuredGrid does exactly the same.
Therefore we remove it with this commit.
The regex we are using might also consider a file named bla.2.blub.
In that case it is not nice to throw an exception. Instead we print
a message to std::cerr.
Previously, the eclipseGrid used by EclipseWriter was constructed from
the one in the EclipseState with the current CpGrid. Unfortunately the
latter was the distributed version resembling only the local part that
the processor works on. Therefore the information about the active cells
was wrong when writing results (which raised an exception in the writer).
With this commit we construct the EclipseWriter before distributing the grid
and use this writer later on in the OutputWriter.
All ranks were still writing to step_timing.txt at the same time.
This made it unusable for parallel runs. With this commit only
one processes writes to this file.
This is currently still happening due to the implementation of
OPM_THROW whenever the linear solver does not converge. This
happens quite often and we might not want to get overwhelmed by
the issue tracker.
The new WellSwitchingLogger within updateWellControls uses
collective communication with all processes. Therefore all
of them need to enter the function as other flow_mpi will deadlock.
Therefore this commit calls the method even with non local wells
active.
That version does not provide a default constructor for
CollectiveCommunication, Therefore we now use
MPIHelper::getCollectiveCommunication() for the default
constructor argument.
This completes f94459d5ed
Each process with rank >0 will use .<deckname>.<rank>.DEBUG, and
<deckname>-<rank>.PRT for logging (instead of <file>.<rank>as before.
After the simulator has finished running we will append the content
of those files to the usual log files. If these files have a non-zero
size we will omit a warning as this should not happen if logging is
done right.
These files will be empty unless we fail to to log messages
only on the root process. Currently that is the case for
the messages about switching the well controls.
Only the root process did set the output_dir correctly. Others
used the default. Therefore all messages logged by non-root
processes did end up in the current directory even if an
output_dir was passed to flow_mpi.
Previously, for all step zero was reported. With this commit
we set these numbers in the SimulatorReport and now they end
up correctly in step_timings.txt
Previously, we also called it when the full time step was done.
As the simulator writes that information anyway and we cannot call
it a sub step, we omit the final write in the adaptive time stepper.
Its first implementation computed wrong results in parallel. With this commit
we noe have completely parallelized the computations and the results seem correct
for parallel runs with norne.
Both hcpv and res will be used to save only dims elements. As dims
will most likely be much smaller than the number of cells, we only
allocate containers of size dims with this commit.
With GCC version (Debian 4.9.2-10) 4.9.2 we get the following error
when compiling with -std=c++11 (default for dune 2.4):
converting to ‘const std::unordered_set<std::basic_string<char> >’ from initializer list would use explicit constructor
Instead of the WellsManager guessing which wells are handled by other
processes we now use tha ouput of the load balancer to compute wells
that are handled by other processes.
With the previous approach it was not possible to calculate this information
correctly. Wells with only one completion next to the border of the
processes' partition were represented on multiple processes. In additition
wells that the eclipse schedule section defined with completions on non-active
cells in sequential runs were not at all calculated in parallel runs.
With the new approach the CpGrid::loaBalance routine returns the set names of
wells that are not handled by this process when setting up the simulation. This
information is then used throughout the simulation.
When running in parallel a well state object with the well information
of the whole grid needs to constructed to gather the information from all
processes. Previously, this was done with the report step exported by the
timer. This was wrong for the following reason:
The output occurs after solving the time step and the timer is already
incremented. This means that we constructed the well state for gathering the
data for the next report step, already. Unfortunately, at that step some
wells that we have computed results for might have been shut. In that case
an exception with message "global state does not contain well ..." was thrown.
This problem occured for Model number 2 and might have been due to shut wells
because of banned cross flow.
With this commit we use the last report step if this is not an initial write
and not a substep.
The order of an unordered_map is quite unpredictable. (The same
order will only hold with the same hash function, comparison
operator, and insertion order). Therefore we cannot assume that
the global SimulationDataContainer uses the same order for the
cell data as the local one (This was done before this commit).
But we can assume that the local one uses the same order on every
process.
Before this commit data got mixed up (e.g. gasoilratio with surfacevol)
when gathering local data for writing eclipse files on the master
process. This commit fixes this.
Instead of iterating over the cell data of the global state when
writing the data received, we again iterate over the cell data of
the local state and simply use the key to request the correct data
for writing from the global state.
Previously, the call was made after the grid was distributed.
This means that each process wrote it, but only with his cells
active which was just a part of the whole domain.
With this commit we make the writeInit call before distributing the
grid and make sure that only one process calls it.
This is used to compute the Euclidian product for the saturations.
Thes are ordered in an interleaved manner (all saturations for cell
with index 0, the all for index 1, ...). Up to now the implementation
assumed a different ordering: blockwise (all saturations for phase 0 first,
then all saturations phase 1, ...).
With this commit the computation uses the right assumption.
Markus Blatt