Rc 08.15.2022 (#70)

* summit configure

* add spock scripts to FOM

* get new models to build with hip

* add hip slipping bc

* testing communication on spock

* update spock build based on olcf docs

* update configure & test scripts for spock

* Fixing potential bugs with communication

* Adding simple test of GPU aware MPI

* some changes to configure for spock

* Modifying GPU aware MPI test to send multiple messages

* playing with spock Gpu test

* added gpu wrapper test

* Cleaning up some compiler warnings

* add barrier between pack / MPI send

* Updating build to support HIP as a language

* fixing gpu mpi sync

* Adding script

* local spock changes

* add membrane class

* update membrane structure

* membrane communications

* working on new comm data structures

* add membrane unit test

* membrane compiles

* membrane test

* Updating hip port to match cuda

* update summit config

* update summit config

* add configure script for crusher

* update membrane test

* update membrane test

* convention for inside / outside membrane link direction

* working on membrane comm

* try to fix time conversion factor for Poisson solver; to be built and tested

* fix dumb typo

* update summit config

* tune launch for crusher

* tune launch for mrt on crusher

* update color

* summit script with specific module versions

* update crusher config

* add crusher examples

* add dense case for crusher

* Fixing some quick annoying compile warnings

* fix binding in example

* working on fix

* Adding simple crusher test

* Adding new crusher MPI test

* disable MPI thread multiple for crusher

* updates to crusher configure

* cpu test for crusher

* Working on standalone reproducer for MPI bug

* More work on creating standalone test

* More work on creating standalone test

* More work on creating standalone test

* Reverting TestCrusher2, standalone version passes (TestCrusher3.cpp), need to figure out why

* Working on standalone MPI test on crusher

* Working on standalone MPI test on crusher

* Getting closer to stand alone test

* Still trying to create standalone reproducer

* hang fix / workaround

* Created standalone MPI failure test

* Removing TestCrusher tests, the bug deals with the StackTrace which we disable the multistack trace for now.  Moving the test out of LBPM

* fix sendcount / recvcount

* Testing persistent communication

* Updating calculation of bandwidth

* crusher hackathon final version

* working on membrane communication structures

* add cell simulator

* added cell simulator

* make sure halo is filled when measuring object

* add membrane transport function for d3q7

* add membrane unpack function

* poking at MF issue

* update crusher build

* membrane data structures compiling

* update to membrane capability

* update comments in ion model

* fix dumb print bug

* clean up relabel

* adding membrane functions

* move membrane to common folder

* membrane structure in IonModel

* membrane structure in IonModel

* try at membrane simulator

* add python script to generate bubble

* add python script to generate bubble

* cell simulator runs

* read input files

* add single cell example

* refining cell example

* start on cuda function

* werkin

* start on cuda function

* start on hip function

* updates and fix for user input reader

* update cell example

* add sigmoid to ion equilibrium dist

* cuda build succeeds

* update crusher script

* getting ready to merge gpu

* refactor compact AA routines for testing

* add testing functions to ScaLBL

* testing membrane ion transport

* membrane transport test passing

* membrane starts working ok...

* original wang poisson solver (broken)

* rex d3q19 (broken)

* tau from wang paper

* still broken wang

* d319 poisson works good

* Poisson working pretty good now

* initialize nernst-planck simulator; to be built subject to debugging

* fix a few syntax bugs and build passed

* Poisson solver; enable specifying initial values

* update cell example

* add GPU functions for d3q19 poisson

* fix dumb bugs

* fix bugs in initializing electric potential; the Psi on solid was accidentally overwritten before.

* small change

* fix bugs in importing ion model's dummy velocity

* add membrane concentration init

* remove bad warnings

* remove print staetements

* add barriers to poisson solver

* update print

* print membrane input concentration

* read Membrane ion concentration list

* fix bad ref to D3Q7

* update error analysis for Poisson solver

* fix typo

* update hip poisson solver

* deprecate old error methods

* a bunch of summit debug things to roll back later

* fix poisson typo

* update hip

* debug crusher

* debug charge density problems on crusher

* fix charge density (i think)

* remove Stokes solver from cell simulator; need to test build

* update cpu ion valence

* added membrane properties to input db

* update cell db

* update executable list and NP_cell simulator

* correct use_membrane functionality

* add functionality for user to choose either D3Q7 or D3Q19 lattice for Poisson;to be built and tested

* build passed

* make further corrections

* correct D3Q7 Poisson LB algorithm

* correct ion LB collison

* udpate output precision

* add more tweaks for cell simulator

* update print-out

* this makes mpi hang error explicit;to be debugged

* cleanup with help from valgrind

* update to cell vis routine

* add hip for ion update

* fix missing bracket

* add new ion code for cuda

* add barrier to membrane transport

* debug gpu launch issue for ion

* debug gpu

* add functions to copy send / recv list from ScaLBL

* updating membrane communication structure

* membrane test works with new comm

* communication seems to work

* add sample files for plane membrane

* update gpu routines first try

* update hip

* multiple nvidia gpu working with membrane

* added membrane analysis capability

* added support for swc file

* support for SWC input format

* swc reader works with MPI

* shift swc data

* SWC reader update

* SWC reader update 2

* add offset to Domain for swc

* add input files for simple bacteria

* add performance counters to ion / poisson solvers

* fix bug with SWC

* add BC to poisson solver

* fix compiler warnings

* fix memory leaks

* fix zlib download path

* Fixing memory leak

* Fixing memory leaks

* restart for Poisson model

* fix bug in ion model restart

* trying to fix yaml

* fix workflow indentation

* porosity factor in effperm

* porosity factor in effperm

* porosity factor in effperm

* porosity factor in effperm

Co-authored-by: James E McClure <mcclurej@vt.edu>
Co-authored-by: Mark Berrill <berrillma@ornl.gov>
Co-authored-by: Zhe Rex Li <zhe.rex.li@gmail.com>
Co-authored-by: Zhe Li <zzl109@gadi-login-01.gadi.nci.org.au>
Co-authored-by: Zhe Li <zzl109@gadi-login-04.gadi.nci.org.au>
Co-authored-by: Zhe Li <zzl109@gadi-login-02.gadi.nci.org.au>
Co-authored-by: Zhe Li <zzl109@gadi-login-06.gadi.nci.org.au>
Co-authored-by: Zhe Li <zzl109@gadi-login-05.gadi.nci.org.au>

models/MultiPhysController.cpp

@@ -4,7 +4,7 @@ ScaLBL_Multiphys_Controller::ScaLBL_Multiphys_Controller(
     int RANK, int NP, const Utilities::MPI &COMM)
     : rank(RANK), nprocs(NP), Restart(0), timestepMax(0), num_iter_Stokes(0),
       num_iter_Ion(0), analysis_interval(0), visualization_interval(0),
-      tolerance(0), time_conv_max(0), comm(COMM) {}
+      tolerance(0), time_conv_max(0), time_conv_MainLoop(0), comm(COMM) {}
 ScaLBL_Multiphys_Controller::~ScaLBL_Multiphys_Controller() {}
 
 void ScaLBL_Multiphys_Controller::ReadParams(string filename) {
@@ -16,12 +16,14 @@ void ScaLBL_Multiphys_Controller::ReadParams(string filename) {
     // Default parameters
     timestepMax = 10000;
     Restart = false;
+    restart_interval = 100000;
     num_iter_Stokes = 1;
     num_iter_Ion.push_back(1);
     analysis_interval = 500;
     visualization_interval = 10000;
     tolerance = 1.0e-6;
     time_conv_max = 0.0;
+    time_conv_MainLoop = 0.0;
 
     // load input parameters
     if (study_db->keyExists("timestepMax")) {
@@ -34,6 +36,9 @@ void ScaLBL_Multiphys_Controller::ReadParams(string filename) {
         visualization_interval =
             study_db->getScalar<int>("visualization_interval");
     }
+    if (study_db->keyExists("restart_interval")) {
+    	restart_interval = study_db->getScalar<int>("restart_interval");
+    }
     if (study_db->keyExists("tolerance")) {
         tolerance = study_db->getScalar<double>("tolerance");
     }
@@ -150,6 +155,53 @@ vector<int> ScaLBL_Multiphys_Controller::getIonNumIter_PNP_coupling(
     return num_iter_ion;
 }
 
+vector<int> ScaLBL_Multiphys_Controller::getIonNumIter_NernstPlanck_coupling(
+    const vector<double> &IonTimeConv) {
+    //Return number of internal iterations for the Ion transport solver
+    vector<double> TimeConv;
+    TimeConv.assign(IonTimeConv.begin(), IonTimeConv.end());
+    vector<int> num_iter_ion;
+    vector<double>::iterator it_max = max_element(TimeConv.begin(), TimeConv.end());
+    unsigned int idx_max = distance(TimeConv.begin(), it_max);
+    if (idx_max == 0) {
+        num_iter_ion.push_back(2);
+        for (unsigned int idx = 1; idx < TimeConv.size(); idx++) {
+            double temp =
+                2 * TimeConv[idx_max] /
+                TimeConv
+                    [idx]; //the factor 2 is the number of iterations for the element has max time_conv
+            num_iter_ion.push_back(int(round(temp / 2) * 2));
+        }
+    } else if (idx_max == TimeConv.size() - 1) {
+        for (unsigned int idx = 0; idx < TimeConv.size() - 1; idx++) {
+            double temp =
+                2 * TimeConv[idx_max] /
+                TimeConv
+                    [idx]; //the factor 2 is the number of iterations for the element has max time_conv
+            num_iter_ion.push_back(int(round(temp / 2) * 2));
+        }
+        num_iter_ion.push_back(2);
+    } else {
+        for (unsigned int idx = 0; idx < idx_max; idx++) {
+            double temp =
+                2 * TimeConv[idx_max] /
+                TimeConv
+                    [idx]; //the factor 2 is the number of iterations for the element has max time_conv
+            num_iter_ion.push_back(int(round(temp / 2) * 2));
+        }
+        num_iter_ion.push_back(2);
+        for (unsigned int idx = idx_max + 1; idx < TimeConv.size(); idx++) {
+            double temp =
+                2 * TimeConv[idx_max] /
+                TimeConv
+                    [idx]; //the factor 2 is the number of iterations for the element has max time_conv
+            num_iter_ion.push_back(int(round(temp / 2) * 2));
+        }
+    }
+    return num_iter_ion;
+}
+
+
 void ScaLBL_Multiphys_Controller::getTimeConvMax_PNP_coupling(
     double StokesTimeConv, const vector<double> &IonTimeConv) {
     //Return maximum of the time converting factor from Stokes and ion solvers