Teko_testdriver_tpetra_MPI_1 randomly failing in ATDM waterman build
Created by: fryeguy52
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
Next Action Status
Description
As shown in this query the test:
- Teko_testdriver_tpetra_MPI_1
looks to be randomly failing in the build:
- Trilinos-atdm-waterman-cuda-9.2-release-debug
It has failed 5 times in the last month each time with:
terminate called after throwing an instance of 'std::runtime_error'
what(): cudaGetLastError() error( cudaErrorAssert): device-side assert triggered /home/jenkins/waterman/workspace/Trilinos-atdm-waterman-cuda-9.2-release-debug/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaExec.hpp:401
Traceback functionality not available
full output from a failed run can be found here
Current Status on CDash
current 4 week history can be found here
Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
More specifically, the commands given for waterman are provided at:
The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20