Randomly failing Sacado tests on ATDM KNL mutrino build
Created by: fryeguy52
CC: @trilinos/sacado, @rppawlo (Trilinos Nonlinear Solvers Product Lead), @bartlettroscoe, @fryeguy52
??: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>Next Action Status
Waiting for https://sems-atlassian-son.sandia.gov/jira/browse/TRIL-219 to be addressed. This seems to be an environment issue with slurm.
Description
As shown in this query the tests:
- Sacado_FadKokkosTests_Serial_MPI_1
- Sacado_FadSerializationTests_MPI_1
- Sacado_TaySerializationTests_MPI_1
are failing in the build:
- Trilinos-atdm-mutrino-intel-opt-openmp-KNL
each of these tests have timed out more than once in the last month and this appears to be random. Here are links to each test's history going back to 2018-10-01:
Sacado_FadKokkosTests_Serial_MPI_1 Sacado_FadSerializationTests_MPI_1 Sacado_TaySerializationTests_MPI_1
Current Status on CDash
The current status of the Sacado tests for this build for the current testing day can be found here
Steps to Reproduce
Because these failures are appear to be random it may be difficult to reproduce the failures. One should be able to reproduce the identical build as described in:
More specifically, the commands given for mutrino are provided at:
The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-mutrino-intel-opt-openmp-KNL
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Sacado=ON \
$TRILINOS_DIR
$ make NP=16
$ salloc -N 1 -p standard -J $ATDM_CONFIG_JOB_NAME ctest -j16