Stratimikos and Rythmos tests failing on many ATDM builds

Created by: fryeguy52

CC: @trilinos/stratimikos, @srajama1 (Trilinos Linear Solvers Product Lead), @rppawlo (Trilinos Nonlinear Solvers Product Lead), @bartlettroscoe, @fryeguy52

??: Add label "ATDM"> ??: Add label "bug"?> ??: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)> ??: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean"> ??: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files> ??: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>

Next Action Status

Description

As shown in this query the tests:

Stratimikos_test_single_stratimikos_solver_driver_belos_np_MPI_1
Stratimikos_test_single_stratimikos_solver_driver_belos_ml_MPI_1
Stratimikos_test_single_stratimikos_solver_driver_belos_ifpack_MPI_1
Rythmos_timeDiscretizedBackwardEuler_amesos_MPI_1

are failing in many ATDM builds.

new commits when these started failing

Current Status on CDash

currently failing tests in ATDM builds can be seen here

Steps to Reproduce

One should be able to reproduce this failure on with a sems rhel6 environment as described in:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

More specifically, the commands given for with a sems rhel6 environment are provided at:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment

The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel6-gnu-7.2.0-openmp-release
$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON \
 -DTrilinos_ENABLE_Stratimikos=ON \
 -DTrilinos_ENABLE_Rythmos=ON \
 $TRILINOS_DIR
$ make NP=16
$ ctest -j8