The motivation for this PR is that currently the build fails on my
Ubuntu 17.10 laptop with two processes because that machine "only" has
8 GB of RAM (granted, the optimization options may have been a bit too
excessive). under the new scheme, each specialization of the simulator
is put into a separate compile unit which is part of
libopmsimulators. this has the advantages that the specialized
simulators and the main binary automatically stay consistent, the
compilation is faster (2m25s vs 4m16s on my machine) because all
compile units can be built in parallel and that compilation takes up
less RAM because there is no need to instantiate all specializations
in a single compile unit.
on the minus side, all specializations must now always be compiled,
the approach means slightly more work for the maintainers and the
flow_* startup code gets even more complicated.
in particular, this implied some changes to the MPI initialization
code. since dune-fem's GridPart class currently has issues with
CpGrid's implementation of loadBalance(), parallel computations still
do not work if dune-fem is around, but at least sequential ones now
do even if MPI is enabled.
this is needed to avoid linker errors if this class ought to be used
in multiple compile units. IMO the main problem here is the use of an
_impl.hpp file.
the only thing that was used of this class was the phase usage object,
but the phase usage object can be accessed via much leaner interfaces.
The old BlackoilPropsFromDeck (without "Ad") is still required to
compute the initial condition, but the init code should be refactored
soon anyway.
that parameter is called "max_strict_iter". This increases the
flexibility of this slightly and it avoids screwing up the default
value for the "max_iter" parameter in the future. The credits for this
patch go to [at]atgeirr for proposing it.
i.e. give up after 10 instead of 15 Newton iterations. since this now
also causes the CNV convergence criterion to be ignored after 8
instead of after 13 Newton iterations, and because CNV stagnation
seems to be the cause for the vast majority of the failures, it does
not lead to worse results or more time step chopping on Norne but
causes fewer wasted iterations.
now, the dune APIs are used whereever possible and the data is
computed for the global grid, i.e. for parallel runs it does not need
to be gathered across the processes anymore. Also, the INIT file is
now only written once instead of twice.
I've verified that the sequential and the parallel INIT files stay
identical for the Norne case and that the INIT file does not change
w.r.t. before this patch.
the fact that faces may point into the opposite Cartesian direction
was not considered and these output fields are specified
w.r.t. logically Cartesian cell indices. (not compressed ones!)
the corresponding code was shamelessly lifted from the DerivedGeology
class. it has been substantially modified to adapt it to the flow_ebos
specifics, though.
it was already almost unused (except for output). Besides making the
overall flow_ebos code leaner because it reduces redundancies, this
patch also implies a small reduduction of memory consumption and a
minor performance improvement. the latter is due to the fact that the
transmissibilities now do not need to be calculated more often than
necessary anymore.
since flow_ebos is the new silverback simulator of the opm-simulators
clan, it should itself as "flow"(the old silverback announces itself
as "flow_legacy" already).
Also some people seem to have been confused by the eWoms version and
codename. Since this information only exhibits limited value IMI,
let's remove it here.
this prevents GCC-7's address sanitizer from complaining about a
delete for a new of a different type. (i.e., the new is for the
derived class, but if the destructor is non-virtual only the dtor of
the base class is called if a pointer to the base class is deleted.)
finding this was quite some fun, mainly because it took a while what
the issue was.
`flow_ebos` should be now capable of doing everything that
`flow_legacy` can and recently all testing seems to have been centered
on `flow_ebos`. note that `flow_ebos` does not yet support some more
advanced/exotic features like MPI, solvents, polymer, etc., but
neither does the plain `flow_legacy` binary. until `flow_ebos`
supports these features, the specialized simulators thus continue to
use the legacy code paths.
the performance summary at the end of a Norne run which are printed by
`flow_ebos` now looks like this on my machine:
```
Total time (seconds): 773.757
Solver time (seconds): 753.349
Assembly time (seconds): 377.218 (Failed: 23.537; 6.23965%)
Linear solve time (seconds): 352.022 (Failed: 23.2757; 6.61201%)
Update time (seconds): 16.3658 (Failed: 1.13149; 6.91375%)
Output write time (seconds): 22.5991
Overall Well Iterations: 870 (Failed: 35; 4.02299%)
Overall Linearizations: 2098 (Failed: 136; 6.48236%)
Overall Newton Iterations: 1756 (Failed: 136; 7.74487%)
Overall Linear Iterations: 26572 (Failed: 1786; 6.72136%)
```
for the flow_legacy family, nothing changes.
or are these called integration tests? anyway, these are the tests
which use the decks from opm-data. This should put flow_ebos testing
on the same level as flow_legacy.
the results of `flow_legacy` and `flow_ebos` are not bit-wise
identical. The same tests using `flow_ebos` should to be added when
re-organizing the unit tests the next time.
This used to be done in solveJacobianSystem(), but this method is only
supposed to solve the linearized system of equations, not to modify it
IMO.
I tested this patch with Norne: It did not change anything.
this unifies the code paths of the code that calculates the FIP field
totals for the parallel and the sequential cases and makes the code
more robust because it does not hard-code the presence of an intensive
quantities cache anymore. also, rock compressibility is now also
included in the field totals instead of just the FIP regions. this was
forgotten in the last FIP PR because the region values are calculated
in a different class using completely different code. (i.e., regions
are done by the model, field totals by the simulator. that design
should win an award, IMO.)
with this patch, the field totals for Norne and SPE1 seem to match
those produced by E100 _very_ closely and also parallel and sequential
runs for Norne and SPE1 of flow_ebos produce exactly the same
numbers. (This is probably the case for all decks, but I haven't
tested anything else.)
mainly this should now work properly in parallel, because non-interior
cells are not counted multiple times anymore. also, the number of
loops over the global arrays has been reduced, some variables have
been renamed and some comments were added.
finally this fixes the average pressure for regions that do not
contain hydrocarbons (or at least it unifies it with the approach for
regions that contain hydrocarbons).
for now "all pore volume multipliers" means compressibility. the
storage term of the simulator includes them, so they need to be
considered when calculating the fluid in place as well.
for Norne the numbers are now very close to those of the ECL
reference. ("very close" means < 0.01 % deviation of pressure and
initial fluid volume for the field totals.)
in particular, the rock compressibility effects are not considered in
the FIP numbers anymore. While I'm not sure if this is correct or not,
it at least makes the results consistent with those produced by
'flow_legacy'.
this information is already part of the EclipseState. The reason why
this should IMO be avoided is that this enforces an implementation
detail (ordering of the permeability matrices) of the simulator on the
well model. If this needs to be done for performance reasons, IMO it
would be smarter to pass an array of matrices instead of passing a raw
array of doubles. I doubt that this is necessary, though: completing
the full Norne deck takes about 0.25 seconds longer on my machine,
that's substantially less than 0.1% of the total runtime.
this information is already part of the EclipseState. The reason why
this should IMO be avoided is that this enforces an implementation
(ordering of the permeability matrices) the simulator on the well
model. If this needs to be done for performance reasons, IMO it would
be smarter to pass an array of matrices, instead of passing a raw
array of doubles. I doubt that this is necessary, though: completing
the full Norne deck takes about 0.25 seconds longer on my machine,
that's substantially less than 0.1% of the total runtime.
in order to avoid code duplication, the permeability extraction
function of the RockFromDeck class is now made a public static
function and used as an implementation detail of the WellsManager.
finally, the permfield_valid_ attribute is removed from the
RockFromDeck class because this data was unused and not accessible via
the class' public API.
this is has become necessary by the recent SWATINIT changes and is
needed to calculate the fluid-in-place consistently as well as for
ebos to use the correct initial solution for the initial
linearization.
the canonical source is now the EclProblem object.
v2: don't change the API of the DerivedGeology class. thanks to
[at]atgeirr for noticing that this can be avoided.
the most significant change is that only a "PhaseUsage" object must be
passed to its constructor instead of a full "fluid object". also, the
pointers to the vector of "active" phases has been converted into a
full object. (this helps to avoid potential use-after-free errors.)
the problem here is that some types used as the `BVector` template
parameters return signed integers for the `size()` method, others
return unsigned ones.
this makes the RateConverter stuff independent of Eigen and it
simplifies some things because the the old PVT API is designed as a
"bulk-with-derivatives" API while the rate converter code used it in
"single shot" mode without derivatives.
this code mostly used the Eigen vectors as arrays anyway, so let's use
`std::vector`.
also, this patch only "mostly eliminates" Eigen from from these parts
of the code because the source files of the VFP code still use
AutoDiffBlock; Unfortunately this cannot easily be changed because
`flow_legacy` depends on these methods. (`flow_ebos` does not use the
incriminating methods.)
I consider the blackoil model enums to be pretty low level while the
"FluidProperties" code is IMO quite high level. this, using the latter
to define the former constitutes a layering violation IMO. note that
the fix is to simply use the enums from opm-core directly.
now we have BlackoilDetails.hpp which contains all stuff that is used
by flow_ebos as well as flow and which does not include anything from
Eigen, and we have BlackoilLegacyDetails.hpp which contains all stuff
that depends on Eigen (and is thus not required by flow_ebos)
I probably did not see this warning when testing f7910af7d7 because
I'm currently flooded by deprecation warnings stemming from
Eigen. (these warnings are caused because I use Ubuntu 16.04 and an
old version of Eigen is cloned by the build system that is not system
installed.)
* https://github.com/OPM/opm-simulators:
[bugfix] add defunct_well_names to BlackoilModelEbos.
[bugfix] fix ownerMask for parallel FIP code.
flow_ebos: tell the ebos in ourselves to not handle SWATINIT
[bugfix] Make initialization work in parallel for flow_ebos.
[bugfix] make flow_ebos work when no wells are present.
flow_ebos: only instantiate the grid once
this patch also includes code to make FlowMainEbos more autonomous
(i.e. it does not derive from FlowMainBase anymore). this allows more
flexibility how and what stuff gets created.
it now uses the grid object which gets created by ebos for everything
which should make the parallelization efforts easier. I also tried to
cut back the use of the legacy property objects (i.e., for the fluid,
geologic and rock properties), but this effort ran aground because of
the initialization and output code. (also, if those two were fixed,
there would probably be issues with the Newton update.)
I ran Norne with this and there did not seem to be any notable
performance regressions or benefits.
the compiler suggested braces around the conditions, and by look of
it, it was right: these asserts most likely want express that only the
threephase case or the twophase case without gas are handled...
this changes the conversion of the results to hopefully make it work
with twophase simulations. Note that flow_ebos still does not work
because there is a crash in the well model that I don't understand
(and also, I've disabled an assertation in the well model plus I'm not
completely sure if the conversion code need to do something about the
primary variables in the twophase case).
some files (e.g., thresholdPressures.hpp) are already missing in the
master version of this file, but most of them were specific to the
`frankenstein` branch.
thanks to [at]atgeirr for noticing this.
this broke after the latest master merge (again, the reason were API
changes. this time they did not lead to merge conflicts.)
I've verified that flow_ebos works with Norne and that the performance
did not regress.
* origin/master:
Start with a clean globalCellData_ for each data gathering.
Added assertion that entry was added to globalCellData_ for pack/unpack
Use data::Solution to gather cell data including user requested data.
Warn about broken parallel matlab output.
a few warning fixing.
Removed second implementation of matlab output.
Fix spelling in warning about unhandled parallel ouput.
Andreas Lauser