Files
LBPM/models/MultiPhysController.cpp

214 lines
8.3 KiB
C++
Raw Normal View History

2020-08-14 14:23:22 -04:00
#include "models/MultiPhysController.h"
ScaLBL_Multiphys_Controller::ScaLBL_Multiphys_Controller(
int RANK, int NP, const Utilities::MPI &COMM)
: rank(RANK), nprocs(NP), Restart(0), timestepMax(0), num_iter_Stokes(0),
num_iter_Ion(0), analysis_interval(0), visualization_interval(0),
Rc 08.15.2022 (#70) * summit configure * add spock scripts to FOM * get new models to build with hip * add hip slipping bc * testing communication on spock * update spock build based on olcf docs * update configure & test scripts for spock * Fixing potential bugs with communication * Adding simple test of GPU aware MPI * some changes to configure for spock * Modifying GPU aware MPI test to send multiple messages * playing with spock Gpu test * added gpu wrapper test * Cleaning up some compiler warnings * add barrier between pack / MPI send * Updating build to support HIP as a language * fixing gpu mpi sync * Adding script * local spock changes * add membrane class * update membrane structure * membrane communications * working on new comm data structures * add membrane unit test * membrane compiles * membrane test * Updating hip port to match cuda * update summit config * update summit config * add configure script for crusher * update membrane test * update membrane test * convention for inside / outside membrane link direction * working on membrane comm * try to fix time conversion factor for Poisson solver; to be built and tested * fix dumb typo * update summit config * tune launch for crusher * tune launch for mrt on crusher * update color * summit script with specific module versions * update crusher config * add crusher examples * add dense case for crusher * Fixing some quick annoying compile warnings * fix binding in example * working on fix * Adding simple crusher test * Adding new crusher MPI test * disable MPI thread multiple for crusher * updates to crusher configure * cpu test for crusher * Working on standalone reproducer for MPI bug * More work on creating standalone test * More work on creating standalone test * More work on creating standalone test * Reverting TestCrusher2, standalone version passes (TestCrusher3.cpp), need to figure out why * Working on standalone MPI test on crusher * Working on standalone MPI test on crusher * Getting closer to stand alone test * Still trying to create standalone reproducer * hang fix / workaround * Created standalone MPI failure test * Removing TestCrusher tests, the bug deals with the StackTrace which we disable the multistack trace for now. Moving the test out of LBPM * fix sendcount / recvcount * Testing persistent communication * Updating calculation of bandwidth * crusher hackathon final version * working on membrane communication structures * add cell simulator * added cell simulator * make sure halo is filled when measuring object * add membrane transport function for d3q7 * add membrane unpack function * poking at MF issue * update crusher build * membrane data structures compiling * update to membrane capability * update comments in ion model * fix dumb print bug * clean up relabel * adding membrane functions * move membrane to common folder * membrane structure in IonModel * membrane structure in IonModel * try at membrane simulator * add python script to generate bubble * add python script to generate bubble * cell simulator runs * read input files * add single cell example * refining cell example * start on cuda function * werkin * start on cuda function * start on hip function * updates and fix for user input reader * update cell example * add sigmoid to ion equilibrium dist * cuda build succeeds * update crusher script * getting ready to merge gpu * refactor compact AA routines for testing * add testing functions to ScaLBL * testing membrane ion transport * membrane transport test passing * membrane starts working ok... * original wang poisson solver (broken) * rex d3q19 (broken) * tau from wang paper * still broken wang * d319 poisson works good * Poisson working pretty good now * initialize nernst-planck simulator; to be built subject to debugging * fix a few syntax bugs and build passed * Poisson solver; enable specifying initial values * update cell example * add GPU functions for d3q19 poisson * fix dumb bugs * fix bugs in initializing electric potential; the Psi on solid was accidentally overwritten before. * small change * fix bugs in importing ion model's dummy velocity * add membrane concentration init * remove bad warnings * remove print staetements * add barriers to poisson solver * update print * print membrane input concentration * read Membrane ion concentration list * fix bad ref to D3Q7 * update error analysis for Poisson solver * fix typo * update hip poisson solver * deprecate old error methods * a bunch of summit debug things to roll back later * fix poisson typo * update hip * debug crusher * debug charge density problems on crusher * fix charge density (i think) * remove Stokes solver from cell simulator; need to test build * update cpu ion valence * added membrane properties to input db * update cell db * update executable list and NP_cell simulator * correct use_membrane functionality * add functionality for user to choose either D3Q7 or D3Q19 lattice for Poisson;to be built and tested * build passed * make further corrections * correct D3Q7 Poisson LB algorithm * correct ion LB collison * udpate output precision * add more tweaks for cell simulator * update print-out * this makes mpi hang error explicit;to be debugged * cleanup with help from valgrind * update to cell vis routine * add hip for ion update * fix missing bracket * add new ion code for cuda * add barrier to membrane transport * debug gpu launch issue for ion * debug gpu * add functions to copy send / recv list from ScaLBL * updating membrane communication structure * membrane test works with new comm * communication seems to work * add sample files for plane membrane * update gpu routines first try * update hip * multiple nvidia gpu working with membrane * added membrane analysis capability * added support for swc file * support for SWC input format * swc reader works with MPI * shift swc data * SWC reader update * SWC reader update 2 * add offset to Domain for swc * add input files for simple bacteria * add performance counters to ion / poisson solvers * fix bug with SWC * add BC to poisson solver * fix compiler warnings * fix memory leaks * fix zlib download path * Fixing memory leak * Fixing memory leaks * restart for Poisson model * fix bug in ion model restart * trying to fix yaml * fix workflow indentation * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm Co-authored-by: James E McClure <mcclurej@vt.edu> Co-authored-by: Mark Berrill <berrillma@ornl.gov> Co-authored-by: Zhe Rex Li <zhe.rex.li@gmail.com> Co-authored-by: Zhe Li <zzl109@gadi-login-01.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-04.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-02.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-06.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-05.gadi.nci.org.au>
2022-09-07 15:44:16 -04:00
tolerance(0), time_conv_max(0), time_conv_MainLoop(0), comm(COMM) {}
ScaLBL_Multiphys_Controller::~ScaLBL_Multiphys_Controller() {}
2020-08-14 14:23:22 -04:00
void ScaLBL_Multiphys_Controller::ReadParams(string filename) {
2020-08-14 14:23:22 -04:00
// read the input database
db = std::make_shared<Database>(filename);
study_db = db->getDatabase("MultiphysController");
2020-08-14 14:23:22 -04:00
// Default parameters
timestepMax = 10000;
Restart = false;
Rc 08.15.2022 (#70) * summit configure * add spock scripts to FOM * get new models to build with hip * add hip slipping bc * testing communication on spock * update spock build based on olcf docs * update configure & test scripts for spock * Fixing potential bugs with communication * Adding simple test of GPU aware MPI * some changes to configure for spock * Modifying GPU aware MPI test to send multiple messages * playing with spock Gpu test * added gpu wrapper test * Cleaning up some compiler warnings * add barrier between pack / MPI send * Updating build to support HIP as a language * fixing gpu mpi sync * Adding script * local spock changes * add membrane class * update membrane structure * membrane communications * working on new comm data structures * add membrane unit test * membrane compiles * membrane test * Updating hip port to match cuda * update summit config * update summit config * add configure script for crusher * update membrane test * update membrane test * convention for inside / outside membrane link direction * working on membrane comm * try to fix time conversion factor for Poisson solver; to be built and tested * fix dumb typo * update summit config * tune launch for crusher * tune launch for mrt on crusher * update color * summit script with specific module versions * update crusher config * add crusher examples * add dense case for crusher * Fixing some quick annoying compile warnings * fix binding in example * working on fix * Adding simple crusher test * Adding new crusher MPI test * disable MPI thread multiple for crusher * updates to crusher configure * cpu test for crusher * Working on standalone reproducer for MPI bug * More work on creating standalone test * More work on creating standalone test * More work on creating standalone test * Reverting TestCrusher2, standalone version passes (TestCrusher3.cpp), need to figure out why * Working on standalone MPI test on crusher * Working on standalone MPI test on crusher * Getting closer to stand alone test * Still trying to create standalone reproducer * hang fix / workaround * Created standalone MPI failure test * Removing TestCrusher tests, the bug deals with the StackTrace which we disable the multistack trace for now. Moving the test out of LBPM * fix sendcount / recvcount * Testing persistent communication * Updating calculation of bandwidth * crusher hackathon final version * working on membrane communication structures * add cell simulator * added cell simulator * make sure halo is filled when measuring object * add membrane transport function for d3q7 * add membrane unpack function * poking at MF issue * update crusher build * membrane data structures compiling * update to membrane capability * update comments in ion model * fix dumb print bug * clean up relabel * adding membrane functions * move membrane to common folder * membrane structure in IonModel * membrane structure in IonModel * try at membrane simulator * add python script to generate bubble * add python script to generate bubble * cell simulator runs * read input files * add single cell example * refining cell example * start on cuda function * werkin * start on cuda function * start on hip function * updates and fix for user input reader * update cell example * add sigmoid to ion equilibrium dist * cuda build succeeds * update crusher script * getting ready to merge gpu * refactor compact AA routines for testing * add testing functions to ScaLBL * testing membrane ion transport * membrane transport test passing * membrane starts working ok... * original wang poisson solver (broken) * rex d3q19 (broken) * tau from wang paper * still broken wang * d319 poisson works good * Poisson working pretty good now * initialize nernst-planck simulator; to be built subject to debugging * fix a few syntax bugs and build passed * Poisson solver; enable specifying initial values * update cell example * add GPU functions for d3q19 poisson * fix dumb bugs * fix bugs in initializing electric potential; the Psi on solid was accidentally overwritten before. * small change * fix bugs in importing ion model's dummy velocity * add membrane concentration init * remove bad warnings * remove print staetements * add barriers to poisson solver * update print * print membrane input concentration * read Membrane ion concentration list * fix bad ref to D3Q7 * update error analysis for Poisson solver * fix typo * update hip poisson solver * deprecate old error methods * a bunch of summit debug things to roll back later * fix poisson typo * update hip * debug crusher * debug charge density problems on crusher * fix charge density (i think) * remove Stokes solver from cell simulator; need to test build * update cpu ion valence * added membrane properties to input db * update cell db * update executable list and NP_cell simulator * correct use_membrane functionality * add functionality for user to choose either D3Q7 or D3Q19 lattice for Poisson;to be built and tested * build passed * make further corrections * correct D3Q7 Poisson LB algorithm * correct ion LB collison * udpate output precision * add more tweaks for cell simulator * update print-out * this makes mpi hang error explicit;to be debugged * cleanup with help from valgrind * update to cell vis routine * add hip for ion update * fix missing bracket * add new ion code for cuda * add barrier to membrane transport * debug gpu launch issue for ion * debug gpu * add functions to copy send / recv list from ScaLBL * updating membrane communication structure * membrane test works with new comm * communication seems to work * add sample files for plane membrane * update gpu routines first try * update hip * multiple nvidia gpu working with membrane * added membrane analysis capability * added support for swc file * support for SWC input format * swc reader works with MPI * shift swc data * SWC reader update * SWC reader update 2 * add offset to Domain for swc * add input files for simple bacteria * add performance counters to ion / poisson solvers * fix bug with SWC * add BC to poisson solver * fix compiler warnings * fix memory leaks * fix zlib download path * Fixing memory leak * Fixing memory leaks * restart for Poisson model * fix bug in ion model restart * trying to fix yaml * fix workflow indentation * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm Co-authored-by: James E McClure <mcclurej@vt.edu> Co-authored-by: Mark Berrill <berrillma@ornl.gov> Co-authored-by: Zhe Rex Li <zhe.rex.li@gmail.com> Co-authored-by: Zhe Li <zzl109@gadi-login-01.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-04.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-02.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-06.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-05.gadi.nci.org.au>
2022-09-07 15:44:16 -04:00
restart_interval = 100000;
num_iter_Stokes = 1;
num_iter_Ion.push_back(1);
analysis_interval = 500;
2020-09-20 11:00:36 -04:00
visualization_interval = 10000;
tolerance = 1.0e-6;
time_conv_max = 0.0;
Rc 08.15.2022 (#70) * summit configure * add spock scripts to FOM * get new models to build with hip * add hip slipping bc * testing communication on spock * update spock build based on olcf docs * update configure & test scripts for spock * Fixing potential bugs with communication * Adding simple test of GPU aware MPI * some changes to configure for spock * Modifying GPU aware MPI test to send multiple messages * playing with spock Gpu test * added gpu wrapper test * Cleaning up some compiler warnings * add barrier between pack / MPI send * Updating build to support HIP as a language * fixing gpu mpi sync * Adding script * local spock changes * add membrane class * update membrane structure * membrane communications * working on new comm data structures * add membrane unit test * membrane compiles * membrane test * Updating hip port to match cuda * update summit config * update summit config * add configure script for crusher * update membrane test * update membrane test * convention for inside / outside membrane link direction * working on membrane comm * try to fix time conversion factor for Poisson solver; to be built and tested * fix dumb typo * update summit config * tune launch for crusher * tune launch for mrt on crusher * update color * summit script with specific module versions * update crusher config * add crusher examples * add dense case for crusher * Fixing some quick annoying compile warnings * fix binding in example * working on fix * Adding simple crusher test * Adding new crusher MPI test * disable MPI thread multiple for crusher * updates to crusher configure * cpu test for crusher * Working on standalone reproducer for MPI bug * More work on creating standalone test * More work on creating standalone test * More work on creating standalone test * Reverting TestCrusher2, standalone version passes (TestCrusher3.cpp), need to figure out why * Working on standalone MPI test on crusher * Working on standalone MPI test on crusher * Getting closer to stand alone test * Still trying to create standalone reproducer * hang fix / workaround * Created standalone MPI failure test * Removing TestCrusher tests, the bug deals with the StackTrace which we disable the multistack trace for now. Moving the test out of LBPM * fix sendcount / recvcount * Testing persistent communication * Updating calculation of bandwidth * crusher hackathon final version * working on membrane communication structures * add cell simulator * added cell simulator * make sure halo is filled when measuring object * add membrane transport function for d3q7 * add membrane unpack function * poking at MF issue * update crusher build * membrane data structures compiling * update to membrane capability * update comments in ion model * fix dumb print bug * clean up relabel * adding membrane functions * move membrane to common folder * membrane structure in IonModel * membrane structure in IonModel * try at membrane simulator * add python script to generate bubble * add python script to generate bubble * cell simulator runs * read input files * add single cell example * refining cell example * start on cuda function * werkin * start on cuda function * start on hip function * updates and fix for user input reader * update cell example * add sigmoid to ion equilibrium dist * cuda build succeeds * update crusher script * getting ready to merge gpu * refactor compact AA routines for testing * add testing functions to ScaLBL * testing membrane ion transport * membrane transport test passing * membrane starts working ok... * original wang poisson solver (broken) * rex d3q19 (broken) * tau from wang paper * still broken wang * d319 poisson works good * Poisson working pretty good now * initialize nernst-planck simulator; to be built subject to debugging * fix a few syntax bugs and build passed * Poisson solver; enable specifying initial values * update cell example * add GPU functions for d3q19 poisson * fix dumb bugs * fix bugs in initializing electric potential; the Psi on solid was accidentally overwritten before. * small change * fix bugs in importing ion model's dummy velocity * add membrane concentration init * remove bad warnings * remove print staetements * add barriers to poisson solver * update print * print membrane input concentration * read Membrane ion concentration list * fix bad ref to D3Q7 * update error analysis for Poisson solver * fix typo * update hip poisson solver * deprecate old error methods * a bunch of summit debug things to roll back later * fix poisson typo * update hip * debug crusher * debug charge density problems on crusher * fix charge density (i think) * remove Stokes solver from cell simulator; need to test build * update cpu ion valence * added membrane properties to input db * update cell db * update executable list and NP_cell simulator * correct use_membrane functionality * add functionality for user to choose either D3Q7 or D3Q19 lattice for Poisson;to be built and tested * build passed * make further corrections * correct D3Q7 Poisson LB algorithm * correct ion LB collison * udpate output precision * add more tweaks for cell simulator * update print-out * this makes mpi hang error explicit;to be debugged * cleanup with help from valgrind * update to cell vis routine * add hip for ion update * fix missing bracket * add new ion code for cuda * add barrier to membrane transport * debug gpu launch issue for ion * debug gpu * add functions to copy send / recv list from ScaLBL * updating membrane communication structure * membrane test works with new comm * communication seems to work * add sample files for plane membrane * update gpu routines first try * update hip * multiple nvidia gpu working with membrane * added membrane analysis capability * added support for swc file * support for SWC input format * swc reader works with MPI * shift swc data * SWC reader update * SWC reader update 2 * add offset to Domain for swc * add input files for simple bacteria * add performance counters to ion / poisson solvers * fix bug with SWC * add BC to poisson solver * fix compiler warnings * fix memory leaks * fix zlib download path * Fixing memory leak * Fixing memory leaks * restart for Poisson model * fix bug in ion model restart * trying to fix yaml * fix workflow indentation * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm Co-authored-by: James E McClure <mcclurej@vt.edu> Co-authored-by: Mark Berrill <berrillma@ornl.gov> Co-authored-by: Zhe Rex Li <zhe.rex.li@gmail.com> Co-authored-by: Zhe Li <zzl109@gadi-login-01.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-04.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-02.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-06.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-05.gadi.nci.org.au>
2022-09-07 15:44:16 -04:00
time_conv_MainLoop = 0.0;
2020-08-14 14:23:22 -04:00
// load input parameters
if (study_db->keyExists("timestepMax")) {
timestepMax = study_db->getScalar<int>("timestepMax");
}
if (study_db->keyExists("analysis_interval")) {
analysis_interval = study_db->getScalar<int>("analysis_interval");
}
if (study_db->keyExists("visualization_interval")) {
visualization_interval =
study_db->getScalar<int>("visualization_interval");
}
Rc 08.15.2022 (#70) * summit configure * add spock scripts to FOM * get new models to build with hip * add hip slipping bc * testing communication on spock * update spock build based on olcf docs * update configure & test scripts for spock * Fixing potential bugs with communication * Adding simple test of GPU aware MPI * some changes to configure for spock * Modifying GPU aware MPI test to send multiple messages * playing with spock Gpu test * added gpu wrapper test * Cleaning up some compiler warnings * add barrier between pack / MPI send * Updating build to support HIP as a language * fixing gpu mpi sync * Adding script * local spock changes * add membrane class * update membrane structure * membrane communications * working on new comm data structures * add membrane unit test * membrane compiles * membrane test * Updating hip port to match cuda * update summit config * update summit config * add configure script for crusher * update membrane test * update membrane test * convention for inside / outside membrane link direction * working on membrane comm * try to fix time conversion factor for Poisson solver; to be built and tested * fix dumb typo * update summit config * tune launch for crusher * tune launch for mrt on crusher * update color * summit script with specific module versions * update crusher config * add crusher examples * add dense case for crusher * Fixing some quick annoying compile warnings * fix binding in example * working on fix * Adding simple crusher test * Adding new crusher MPI test * disable MPI thread multiple for crusher * updates to crusher configure * cpu test for crusher * Working on standalone reproducer for MPI bug * More work on creating standalone test * More work on creating standalone test * More work on creating standalone test * Reverting TestCrusher2, standalone version passes (TestCrusher3.cpp), need to figure out why * Working on standalone MPI test on crusher * Working on standalone MPI test on crusher * Getting closer to stand alone test * Still trying to create standalone reproducer * hang fix / workaround * Created standalone MPI failure test * Removing TestCrusher tests, the bug deals with the StackTrace which we disable the multistack trace for now. Moving the test out of LBPM * fix sendcount / recvcount * Testing persistent communication * Updating calculation of bandwidth * crusher hackathon final version * working on membrane communication structures * add cell simulator * added cell simulator * make sure halo is filled when measuring object * add membrane transport function for d3q7 * add membrane unpack function * poking at MF issue * update crusher build * membrane data structures compiling * update to membrane capability * update comments in ion model * fix dumb print bug * clean up relabel * adding membrane functions * move membrane to common folder * membrane structure in IonModel * membrane structure in IonModel * try at membrane simulator * add python script to generate bubble * add python script to generate bubble * cell simulator runs * read input files * add single cell example * refining cell example * start on cuda function * werkin * start on cuda function * start on hip function * updates and fix for user input reader * update cell example * add sigmoid to ion equilibrium dist * cuda build succeeds * update crusher script * getting ready to merge gpu * refactor compact AA routines for testing * add testing functions to ScaLBL * testing membrane ion transport * membrane transport test passing * membrane starts working ok... * original wang poisson solver (broken) * rex d3q19 (broken) * tau from wang paper * still broken wang * d319 poisson works good * Poisson working pretty good now * initialize nernst-planck simulator; to be built subject to debugging * fix a few syntax bugs and build passed * Poisson solver; enable specifying initial values * update cell example * add GPU functions for d3q19 poisson * fix dumb bugs * fix bugs in initializing electric potential; the Psi on solid was accidentally overwritten before. * small change * fix bugs in importing ion model's dummy velocity * add membrane concentration init * remove bad warnings * remove print staetements * add barriers to poisson solver * update print * print membrane input concentration * read Membrane ion concentration list * fix bad ref to D3Q7 * update error analysis for Poisson solver * fix typo * update hip poisson solver * deprecate old error methods * a bunch of summit debug things to roll back later * fix poisson typo * update hip * debug crusher * debug charge density problems on crusher * fix charge density (i think) * remove Stokes solver from cell simulator; need to test build * update cpu ion valence * added membrane properties to input db * update cell db * update executable list and NP_cell simulator * correct use_membrane functionality * add functionality for user to choose either D3Q7 or D3Q19 lattice for Poisson;to be built and tested * build passed * make further corrections * correct D3Q7 Poisson LB algorithm * correct ion LB collison * udpate output precision * add more tweaks for cell simulator * update print-out * this makes mpi hang error explicit;to be debugged * cleanup with help from valgrind * update to cell vis routine * add hip for ion update * fix missing bracket * add new ion code for cuda * add barrier to membrane transport * debug gpu launch issue for ion * debug gpu * add functions to copy send / recv list from ScaLBL * updating membrane communication structure * membrane test works with new comm * communication seems to work * add sample files for plane membrane * update gpu routines first try * update hip * multiple nvidia gpu working with membrane * added membrane analysis capability * added support for swc file * support for SWC input format * swc reader works with MPI * shift swc data * SWC reader update * SWC reader update 2 * add offset to Domain for swc * add input files for simple bacteria * add performance counters to ion / poisson solvers * fix bug with SWC * add BC to poisson solver * fix compiler warnings * fix memory leaks * fix zlib download path * Fixing memory leak * Fixing memory leaks * restart for Poisson model * fix bug in ion model restart * trying to fix yaml * fix workflow indentation * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm Co-authored-by: James E McClure <mcclurej@vt.edu> Co-authored-by: Mark Berrill <berrillma@ornl.gov> Co-authored-by: Zhe Rex Li <zhe.rex.li@gmail.com> Co-authored-by: Zhe Li <zzl109@gadi-login-01.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-04.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-02.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-06.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-05.gadi.nci.org.au>
2022-09-07 15:44:16 -04:00
if (study_db->keyExists("restart_interval")) {
restart_interval = study_db->getScalar<int>("restart_interval");
}
if (study_db->keyExists("tolerance")) {
tolerance = study_db->getScalar<double>("tolerance");
}
//if (study_db->keyExists( "time_conv" )){
// time_conv = study_db->getScalar<double>( "time_conv" );
//}
//if (study_db->keyExists( "Schmidt_Number" )){
// SchmidtNum = study_db->getScalar<double>( "Schmidt_Number" );
//}
2020-08-14 14:23:22 -04:00
// recalculate relevant parameters
//if (SchmidtNum>1){
// num_iter_Stokes = int(round(SchmidtNum/2)*2);
// num_iter_Ion = 1;
//}
//else if (SchmidtNum>0 && SchmidtNum<1){
// num_iter_Ion = int(round((1.0/SchmidtNum)/2)*2);
// num_iter_Stokes = 1;
//}
//else if (SchmidtNum==1){
// num_iter_Stokes = 1;
// num_iter_Ion = 1;
//}
//else{
// ERROR("Error: SchmidtNum (Schmidt number) must be a positive number! \n");
//}
2020-08-14 14:23:22 -04:00
// load input parameters
// in case user wants to have an absolute control over the iternal iteration
if (study_db->keyExists("num_iter_Ion_List")) {
num_iter_Ion.clear();
num_iter_Ion = study_db->getVector<int>("num_iter_Ion_List");
2020-08-14 14:23:22 -04:00
}
if (study_db->keyExists("num_iter_Stokes")) {
num_iter_Stokes = study_db->getScalar<int>("num_iter_Stokes");
2020-08-14 14:23:22 -04:00
}
}
int ScaLBL_Multiphys_Controller::getStokesNumIter_PNP_coupling(
double StokesTimeConv, const vector<double> &IonTimeConv) {
//Return number of internal iterations for the Stokes solver
int num_iter_stokes;
vector<double> TimeConv;
TimeConv.assign(IonTimeConv.begin(), IonTimeConv.end());
TimeConv.insert(TimeConv.begin(), StokesTimeConv);
vector<double>::iterator it_max =
max_element(TimeConv.begin(), TimeConv.end());
int idx_max = distance(TimeConv.begin(), it_max);
if (idx_max == 0) {
num_iter_stokes = 2;
} else {
double temp =
2 * TimeConv[idx_max] /
StokesTimeConv; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_stokes = int(round(temp / 2) * 2);
}
return num_iter_stokes;
}
2020-08-14 14:23:22 -04:00
vector<int> ScaLBL_Multiphys_Controller::getIonNumIter_PNP_coupling(
double StokesTimeConv, const vector<double> &IonTimeConv) {
//Return number of internal iterations for the Ion transport solver
vector<int> num_iter_ion;
vector<double> TimeConv;
TimeConv.assign(IonTimeConv.begin(), IonTimeConv.end());
TimeConv.insert(TimeConv.begin(), StokesTimeConv);
vector<double>::iterator it_max =
max_element(TimeConv.begin(), TimeConv.end());
unsigned int idx_max = distance(TimeConv.begin(), it_max);
if (idx_max == 0) {
for (unsigned int idx = 1; idx < TimeConv.size(); idx++) {
double temp =
2 * StokesTimeConv /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
} else if (idx_max == 1) {
num_iter_ion.push_back(2);
for (unsigned int idx = 2; idx < TimeConv.size(); idx++) {
double temp =
2 * TimeConv[idx_max] /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
} else if (idx_max == TimeConv.size() - 1) {
for (unsigned int idx = 1; idx < TimeConv.size() - 1; idx++) {
double temp =
2 * TimeConv[idx_max] /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
num_iter_ion.push_back(2);
} else {
for (unsigned int idx = 1; idx < idx_max; idx++) {
double temp =
2 * TimeConv[idx_max] /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
num_iter_ion.push_back(2);
for (unsigned int idx = idx_max + 1; idx < TimeConv.size(); idx++) {
double temp =
2 * TimeConv[idx_max] /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
}
return num_iter_ion;
}
Rc 08.15.2022 (#70) * summit configure * add spock scripts to FOM * get new models to build with hip * add hip slipping bc * testing communication on spock * update spock build based on olcf docs * update configure & test scripts for spock * Fixing potential bugs with communication * Adding simple test of GPU aware MPI * some changes to configure for spock * Modifying GPU aware MPI test to send multiple messages * playing with spock Gpu test * added gpu wrapper test * Cleaning up some compiler warnings * add barrier between pack / MPI send * Updating build to support HIP as a language * fixing gpu mpi sync * Adding script * local spock changes * add membrane class * update membrane structure * membrane communications * working on new comm data structures * add membrane unit test * membrane compiles * membrane test * Updating hip port to match cuda * update summit config * update summit config * add configure script for crusher * update membrane test * update membrane test * convention for inside / outside membrane link direction * working on membrane comm * try to fix time conversion factor for Poisson solver; to be built and tested * fix dumb typo * update summit config * tune launch for crusher * tune launch for mrt on crusher * update color * summit script with specific module versions * update crusher config * add crusher examples * add dense case for crusher * Fixing some quick annoying compile warnings * fix binding in example * working on fix * Adding simple crusher test * Adding new crusher MPI test * disable MPI thread multiple for crusher * updates to crusher configure * cpu test for crusher * Working on standalone reproducer for MPI bug * More work on creating standalone test * More work on creating standalone test * More work on creating standalone test * Reverting TestCrusher2, standalone version passes (TestCrusher3.cpp), need to figure out why * Working on standalone MPI test on crusher * Working on standalone MPI test on crusher * Getting closer to stand alone test * Still trying to create standalone reproducer * hang fix / workaround * Created standalone MPI failure test * Removing TestCrusher tests, the bug deals with the StackTrace which we disable the multistack trace for now. Moving the test out of LBPM * fix sendcount / recvcount * Testing persistent communication * Updating calculation of bandwidth * crusher hackathon final version * working on membrane communication structures * add cell simulator * added cell simulator * make sure halo is filled when measuring object * add membrane transport function for d3q7 * add membrane unpack function * poking at MF issue * update crusher build * membrane data structures compiling * update to membrane capability * update comments in ion model * fix dumb print bug * clean up relabel * adding membrane functions * move membrane to common folder * membrane structure in IonModel * membrane structure in IonModel * try at membrane simulator * add python script to generate bubble * add python script to generate bubble * cell simulator runs * read input files * add single cell example * refining cell example * start on cuda function * werkin * start on cuda function * start on hip function * updates and fix for user input reader * update cell example * add sigmoid to ion equilibrium dist * cuda build succeeds * update crusher script * getting ready to merge gpu * refactor compact AA routines for testing * add testing functions to ScaLBL * testing membrane ion transport * membrane transport test passing * membrane starts working ok... * original wang poisson solver (broken) * rex d3q19 (broken) * tau from wang paper * still broken wang * d319 poisson works good * Poisson working pretty good now * initialize nernst-planck simulator; to be built subject to debugging * fix a few syntax bugs and build passed * Poisson solver; enable specifying initial values * update cell example * add GPU functions for d3q19 poisson * fix dumb bugs * fix bugs in initializing electric potential; the Psi on solid was accidentally overwritten before. * small change * fix bugs in importing ion model's dummy velocity * add membrane concentration init * remove bad warnings * remove print staetements * add barriers to poisson solver * update print * print membrane input concentration * read Membrane ion concentration list * fix bad ref to D3Q7 * update error analysis for Poisson solver * fix typo * update hip poisson solver * deprecate old error methods * a bunch of summit debug things to roll back later * fix poisson typo * update hip * debug crusher * debug charge density problems on crusher * fix charge density (i think) * remove Stokes solver from cell simulator; need to test build * update cpu ion valence * added membrane properties to input db * update cell db * update executable list and NP_cell simulator * correct use_membrane functionality * add functionality for user to choose either D3Q7 or D3Q19 lattice for Poisson;to be built and tested * build passed * make further corrections * correct D3Q7 Poisson LB algorithm * correct ion LB collison * udpate output precision * add more tweaks for cell simulator * update print-out * this makes mpi hang error explicit;to be debugged * cleanup with help from valgrind * update to cell vis routine * add hip for ion update * fix missing bracket * add new ion code for cuda * add barrier to membrane transport * debug gpu launch issue for ion * debug gpu * add functions to copy send / recv list from ScaLBL * updating membrane communication structure * membrane test works with new comm * communication seems to work * add sample files for plane membrane * update gpu routines first try * update hip * multiple nvidia gpu working with membrane * added membrane analysis capability * added support for swc file * support for SWC input format * swc reader works with MPI * shift swc data * SWC reader update * SWC reader update 2 * add offset to Domain for swc * add input files for simple bacteria * add performance counters to ion / poisson solvers * fix bug with SWC * add BC to poisson solver * fix compiler warnings * fix memory leaks * fix zlib download path * Fixing memory leak * Fixing memory leaks * restart for Poisson model * fix bug in ion model restart * trying to fix yaml * fix workflow indentation * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm * porosity factor in effperm Co-authored-by: James E McClure <mcclurej@vt.edu> Co-authored-by: Mark Berrill <berrillma@ornl.gov> Co-authored-by: Zhe Rex Li <zhe.rex.li@gmail.com> Co-authored-by: Zhe Li <zzl109@gadi-login-01.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-04.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-02.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-06.gadi.nci.org.au> Co-authored-by: Zhe Li <zzl109@gadi-login-05.gadi.nci.org.au>
2022-09-07 15:44:16 -04:00
vector<int> ScaLBL_Multiphys_Controller::getIonNumIter_NernstPlanck_coupling(
const vector<double> &IonTimeConv) {
//Return number of internal iterations for the Ion transport solver
vector<double> TimeConv;
TimeConv.assign(IonTimeConv.begin(), IonTimeConv.end());
vector<int> num_iter_ion;
vector<double>::iterator it_max = max_element(TimeConv.begin(), TimeConv.end());
unsigned int idx_max = distance(TimeConv.begin(), it_max);
if (idx_max == 0) {
num_iter_ion.push_back(2);
for (unsigned int idx = 1; idx < TimeConv.size(); idx++) {
double temp =
2 * TimeConv[idx_max] /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
} else if (idx_max == TimeConv.size() - 1) {
for (unsigned int idx = 0; idx < TimeConv.size() - 1; idx++) {
double temp =
2 * TimeConv[idx_max] /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
num_iter_ion.push_back(2);
} else {
for (unsigned int idx = 0; idx < idx_max; idx++) {
double temp =
2 * TimeConv[idx_max] /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
num_iter_ion.push_back(2);
for (unsigned int idx = idx_max + 1; idx < TimeConv.size(); idx++) {
double temp =
2 * TimeConv[idx_max] /
TimeConv
[idx]; //the factor 2 is the number of iterations for the element has max time_conv
num_iter_ion.push_back(int(round(temp / 2) * 2));
}
}
return num_iter_ion;
}
void ScaLBL_Multiphys_Controller::getTimeConvMax_PNP_coupling(
double StokesTimeConv, const vector<double> &IonTimeConv) {
//Return maximum of the time converting factor from Stokes and ion solvers
vector<double> TimeConv;
TimeConv.assign(IonTimeConv.begin(), IonTimeConv.end());
TimeConv.insert(TimeConv.begin(), StokesTimeConv);
time_conv_max = *max_element(TimeConv.begin(), TimeConv.end());
}