GitLab

Created by: jmgate

Hey @bartlettroscoe, I'm having trouble getting a build of EMPIRE up and running on chama. It'll build, but I'm seeing all of EMPIRE-PIC's tests fail (see EMPIRE_EM-Plasma-Trilinos-chama-test-all-of-EMPIRE-intel-opt-openmp). I think what's happening is on chama the ATDM_CONFIG_MPI_EXEC variable gets set to srun in cmake/std/atdm/common/toss3/environment.sh. When we test EMPIRE, we wind up submitting the ctest command via srun. Unfortunately each EMPIRE-PIC regression test winds up running

$Trilinos_MPI_EXEC -n $NUM_PROCS $EXE --i=$INPUT

where $TRILINOS_MPI_EXEC is $ATDM_CONFIG_MPI_EXEC, which is srun, so we're effectively invoking srun within something that's been submitted via srun. I don't know slurm inside and out, but that sounds like a problem.

The question, then, is what should we do about it? In the CMake file that's used for all of EMPIRE-PIC's regression tests, do we need to replace $Trilinos_MPI_EXEC with $(which mpiexec) or something to that effect? Or do we need to do something more clever with the $ATDM_CONFIG_MPI_EXEC variable in cmake/std/atdm/common/toss3/environment.sh such that it works for whatever you need it to do, but then it doesn't also mess up EMPIRE's testing on toss3 systems?

@bathmatt, did you set up our CMake stuff to use $Trilinos_MPI_EXEC? Do you know the rationale behind it?