OilfieldToolsNet/opm-simulators
4
0
Fork 0
mirror of https://github.com/OPM/opm-simulators.git synced 2025-02-14 08:13:27 -06:00
Code Issues Packages Projects Releases Wiki Activity
c7c4bffa7c
opm-simulators/opm/simulators/utils/DamarisKeywords.cpp

241 lines
9.0 KiB
C++
Raw Normal View History

Changes needed to add Damaris functionality. The current state is that we output the pressure field and we use both HDF5 and parallel HDF5. Damaris initialization is added after InitMpi but before starting the simulation. Damaris will invoke a separate core for writing in parallel and leave the rest of cores for the simulator. The main changes are in main where start_damaris and then in eclwriterm where we use damaris to output the PRESSURE. To test Damaris one can use --enable-damaris-output=true and to use parallel HDF5 one can use --enable-async-damaris-output=true (false is the default choice)
2021-10-22 04:09:12 -05:00
/*
Copyright 2021 Equinor.
This file is part of the Open Porous Media project (OPM).
OPM is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
OPM is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with OPM. If not, see <http://www.gnu.org/licenses/>.
*/
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
#include <config.h>
#include <opm/simulators/flow/Main.hpp>
#include <opm/models/utils/propertysystem.hh>
#include <opm/models/utils/parametersystem.hh>
Changes needed to add Damaris functionality. The current state is that we output the pressure field and we use both HDF5 and parallel HDF5. Damaris initialization is added after InitMpi but before starting the simulation. Damaris will invoke a separate core for writing in parallel and leave the rest of cores for the simulator. The main changes are in main where start_damaris and then in eclwriterm where we use damaris to output the PRESSURE. To test Damaris one can use --enable-damaris-output=true and to use parallel HDF5 one can use --enable-async-damaris-output=true (false is the default choice)
2021-10-22 04:09:12 -05:00
#include <opm/simulators/utils/DamarisKeywords.hpp>
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
#include <damaris/env/Environment.hpp>
Changes needed to add Damaris functionality. The current state is that we output the pressure field and we use both HDF5 and parallel HDF5. Damaris initialization is added after InitMpi but before starting the simulation. Damaris will invoke a separate core for writing in parallel and leave the rest of cores for the simulator. The main changes are in main where start_damaris and then in eclwriterm where we use damaris to output the PRESSURE. To test Damaris one can use --enable-damaris-output=true and to use parallel HDF5 one can use --enable-async-damaris-output=true (false is the default choice)
2021-10-22 04:09:12 -05:00
#include <string>
#include <map>
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
#include <random>
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
#include <fstream>
Changes needed to add Damaris functionality. The current state is that we output the pressure field and we use both HDF5 and parallel HDF5. Damaris initialization is added after InitMpi but before starting the simulation. Damaris will invoke a separate core for writing in parallel and leave the rest of cores for the simulator. The main changes are in main where start_damaris and then in eclwriterm where we use damaris to output the PRESSURE. To test Damaris one can use --enable-damaris-output=true and to use parallel HDF5 one can use --enable-async-damaris-output=true (false is the default choice)
2021-10-22 04:09:12 -05:00
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
/**
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
Below are the Damaris Keywords supported by Damaris to be filled
Set _PATH_REGEX_ for non-async, also reformat.
2022-10-06 09:34:44 -05:00
in the built-in XML file.
Changes needed to add Damaris functionality. The current state is that we output the pressure field and we use both HDF5 and parallel HDF5. Damaris initialization is added after InitMpi but before starting the simulation. Damaris will invoke a separate core for writing in parallel and leave the rest of cores for the simulator. The main changes are in main where start_damaris and then in eclwriterm where we use damaris to output the PRESSURE. To test Damaris one can use --enable-damaris-output=true and to use parallel HDF5 one can use --enable-async-damaris-output=true (false is the default choice)
2021-10-22 04:09:12 -05:00
Set _PATH_REGEX_ for non-async, also reformat.
2022-10-06 09:34:44 -05:00
The entries in the map below will be filled by the corresponding
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
Damaris Keywords.
The command line arguments are defined in ebos/damariswriter.hh
and defaults are set in ebos/eclproblem_properties.hh
Changes needed to add Damaris functionality. The current state is that we output the pressure field and we use both HDF5 and parallel HDF5. Damaris initialization is added after InitMpi but before starting the simulation. Damaris will invoke a separate core for writing in parallel and leave the rest of cores for the simulator. The main changes are in main where start_damaris and then in eclwriterm where we use damaris to output the PRESSURE. To test Damaris one can use --enable-damaris-output=true and to use parallel HDF5 one can use --enable-async-damaris-output=true (false is the default choice)
2021-10-22 04:09:12 -05:00
*/
namespace Opm::DamarisOutput
{
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
bool FileExists(const std::string filename_in, const MPI_Comm comm, const int rank) {
// From c++17 : std::filesystem::exists(filename_in);
int retint = 0 ;
std::ifstream filestr ;
bool file_exists = false ;
if ((filename_in.length() == 0) || (filename_in == "#") ) {
return file_exists ;
}
if (rank == 0) {
filestr.open(filename_in);
file_exists = true ;
if(filestr.fail()) {
retint = 0 ;
} else {
retint = 1 ;
filestr.close() ;
}
MPI_Bcast(&retint,1,MPI_INT,0,comm);
} else {
MPI_Bcast(&retint,1,MPI_INT,0,comm); // recieve the value from rank 0
}
if (retint == 1) {
file_exists = true ;
}
return (file_exists) ;
}
Set _PATH_REGEX_ for non-async, also reformat.
2022-10-06 09:34:44 -05:00
std::map<std::string, std::string>
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
DamarisKeywords(MPI_Comm comm, std::string OutputDir)
Set _PATH_REGEX_ for non-async, also reformat.
2022-10-06 09:34:44 -05:00
{
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
typedef Properties::TTag::FlowEarlyBird PreTypeTag;
bool enableDamarisOutputCollective = true ;
bool saveToDamarisHDF5 = true ;
std::string pythonFilename = "" ;
std::string paraviewPythonFilename = "" ;
std::string damarisSimName = "" ; // empty defaults to opm-sim-<magic_number>
std::string damarisLogLevel = "info" ;
int nDamarisCores = 1 ;
int nDamarisNodes = 0 ;
long shmemSizeBytes = 536870912 ;
// Get all of the Damaris keywords (except for --enable-damaris, which is used in simulators/flow/Main.hpp)
// These command line arguments are defined in ebos/damariswriter.hh and defaults are set in ebos/eclproblem_properties.hh
enableDamarisOutputCollective = EWOMS_GET_PARAM(PreTypeTag, bool, EnableDamarisOutputCollective) ;
saveToDamarisHDF5 = EWOMS_GET_PARAM(PreTypeTag, bool, DamarisSaveToHdf);
pythonFilename = EWOMS_GET_PARAM(PreTypeTag, std::string, DamarisPythonScript);
paraviewPythonFilename = EWOMS_GET_PARAM(PreTypeTag, std::string, DamarisPythonParaviewScript);
damarisSimName = EWOMS_GET_PARAM(PreTypeTag, std::string, DamarisSimName);
nDamarisCores = EWOMS_GET_PARAM(PreTypeTag, int, DamarisDedicatedCores);
nDamarisNodes = EWOMS_GET_PARAM(PreTypeTag, int, DamarisDedicatedNodes);
shmemSizeBytes = EWOMS_GET_PARAM(PreTypeTag, long, DamarisSharedMemeorySizeBytes);
damarisLogLevel = EWOMS_GET_PARAM(PreTypeTag, std::string, DamarisLogLevel);
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
int rank ;
MPI_Comm_rank(comm, &rank) ;
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
std::string saveToHDF5_str("MyStore") ;
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
if (! saveToDamarisHDF5 ) saveToHDF5_str = "#" ;
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
// These strings are used to comment out an XML element if it is not reqired
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
std::string disablePythonXMLstart("!--") ;
std::string disablePythonXMLfin("--") ;
std::string disableParaviewXMLstart("!--") ;
std::string disableParaviewXMLfin("--") ;
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
// Test if input Python file exists and set the name of the script for <variable ... script="" > )XML elements
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
std::string publishToPython_str("") ;
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
if (pythonFilename != ""){
if (FileExists(pythonFilename, comm, rank)) {
publishToPython_str="PythonScript" ; // the name of the PyScript XML element
disablePythonXMLstart = std::string("") ;
disablePythonXMLfin = std::string("") ;
} else {
pythonFilename = "" ; // set to empty if it does not exist
std::string disablePythonXMLstart("!--") ;
std::string disablePythonXMLfin("--") ;
}
}
// Test if input Paraview Python file exists
if (paraviewPythonFilename != ""){
if (FileExists(paraviewPythonFilename, comm, rank)) {
disableParaviewXMLstart = std::string("") ;
disableParaviewXMLfin = std::string("") ;
} else {
paraviewPythonFilename = "" ; // set to empty if it does not exist
disableParaviewXMLstart = std::string("!--") ;
disableParaviewXMLfin = std::string("--") ;
}
}
// Flag error if both scripts are enabled
// It would be good to know if Damaris has either one compiled into the library - this is currently not possible
if ((pythonFilename.size() > 0) && (paraviewPythonFilename.size() > 0) )
{
std::cerr << "ERROR: Both the Python (--damaris-python-script command line argument) and Paraview Python " <<
"(--damaris-python-paraview-script command line argument) scripts are valid, however only one type "
"of analysis is supported in a single simulation. Please choose one. Exiting." << std::endl ;
std::exit(-1) ;
}
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
std::string damarisOutputCollective_str("") ;
Rename option to EnableDamarisOutputCollective.
2022-10-10 07:28:09 -05:00
if (enableDamarisOutputCollective) {
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
damarisOutputCollective_str="Collective" ;
} else {
damarisOutputCollective_str="FilePerCore" ;
}
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
std::string simName_str("") ;
if (damarisSimName == "") {
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
// Having a different simulation name is important if multiple simulations
// are running on the same node, as it is used to name the simulations shmem area
// and when one sim finishes it removes its shmem file.
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
// simName_str = damaris::Environment::GetMagicNumber(comm) ;
if (simName_str == "") {
// We will add a random value as GetMagicNumber(comm) requires latest Damaris
// Seed with a real random value, if available
std::random_device r;
// Choose a random number between 0 and MAX_INT
std::default_random_engine e1(r());
std::uniform_int_distribution<int> uniform_dist(0, std::numeric_limits<int>::max());
int rand_int = uniform_dist(e1);
simName_str = std::to_string(rand_int) ;
} else {
simName_str = "opm-flow-" + simName_str ;
}
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
} else {
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
simName_str = damarisSimName ;
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
}
if ((nDamarisCores > 0) && (nDamarisNodes > 0))
{
nDamarisNodes = 0 ; // Default is to use Damaris Cores
}
std::string nDamarisCores_str ;
if ( nDamarisCores != 0 ) {
nDamarisCores_str = std::to_string(nDamarisCores);
} else {
nDamarisCores_str = "0" ;
}
std::string nDamarisNodes_str ;
if ( nDamarisNodes != 0 ) {
nDamarisNodes_str = std::to_string(nDamarisNodes);
} else {
nDamarisNodes_str = "0" ;
}
std::string shmemSizeBytes_str ;
if ( shmemSizeBytes != 0 ) {
shmemSizeBytes_str = std::to_string(shmemSizeBytes);
Set _PATH_REGEX_ for non-async, also reformat.
2022-10-06 09:34:44 -05:00
} else {
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
shmemSizeBytes_str = "536870912" ;
Set _PATH_REGEX_ for non-async, also reformat.
2022-10-06 09:34:44 -05:00
}
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
Refactored the Damaris command line functionality so that commandline arguments are accessed where they are needed in the DamarisKeywords() method
2023-09-24 15:23:31 -05:00
std::string logLevel_str(damarisLogLevel) ;
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
// _MAKE_AVAILABLE_IN_PYTHON_
// _PYTHON_SCRIPT_
std::map<std::string, std::string> damaris_keywords = {
{"_SHMEM_BUFFER_BYTES_REGEX_", shmemSizeBytes_str},
{"_DC_REGEX_", nDamarisCores_str},
{"_DN_REGEX_", nDamarisNodes_str},
{"_File_Mode", damarisOutputCollective_str},
{"_MORE_VARIABLES_REGEX_", ""},
{"_PATH_REGEX_", OutputDir}, /* Do Not change the string "_PATH_REGEX_" as it is used to search for the output path */
{"_MYSTORE_OR_EMPTY_REGEX_", saveToHDF5_str},
{"_PARAVIEW_PYTHON_SCRIPT_",paraviewPythonFilename}, /* this has to be before _PYTHON_SCRIPT_ entry */
{"_PYTHON_SCRIPT_",pythonFilename}, /* if a Python script is specified then assume that we want to publish the data to Python */
{"_PRESSURE_UNIT_","Pa"},
{"_MAKE_AVAILABLE_IN_PYTHON_",publishToPython_str}, /* must match <pyscript name="PythonScript" */
{"_SIM_NAME_",simName_str},
{"_LOG_LEVEL_",logLevel_str},
Update for support of Damaris mesh based in situ processing Moved damaris command line parameter accessors to Main.cpp and fixed support for both Python and Paraview Python scripts, as they both cannot be present in the same simulation (seems to be an initialization conflict or double initialization)
2023-09-23 15:24:11 -05:00
{"_DISABLEPYTHONSTART_",disablePythonXMLstart},
{"_DISABLEPYTHONFIN_",disablePythonXMLfin},
{"_DISABLEPARAVIEWSTART_",disableParaviewXMLstart},
{"_DISABLEPARAVIEWFIN_",disableParaviewXMLfin},
Geometric mesh data added for in-situ visualisation added access to DUNE mesh geometry and passing through data to Damaris; Updated command line so users can specifiy Python or Paraview script names and other paramaters that control Damaris - Simulation name - Number of dedicated cores or dedicated nodes - Shared memory region size - switch to turn off HDF5 output. - Damaris logging level
2023-09-22 16:23:29 -05:00
};
return damaris_keywords;
Changes needed to add Damaris functionality. The current state is that we output the pressure field and we use both HDF5 and parallel HDF5. Damaris initialization is added after InitMpi but before starting the simulation. Damaris will invoke a separate core for writing in parallel and leave the rest of cores for the simulator. The main changes are in main where start_damaris and then in eclwriterm where we use damaris to output the PRESSURE. To test Damaris one can use --enable-damaris-output=true and to use parallel HDF5 one can use --enable-async-damaris-output=true (false is the default choice)
2021-10-22 04:09:12 -05:00
}
Set _PATH_REGEX_ for non-async, also reformat.
2022-10-06 09:34:44 -05:00
} // namespace Opm::DamarisOutput
Reference in New Issue Copy Permalink