MueLu_UnitTestsTpetra_MPI_ tests timing out on ATDM cuda 9.2 builds on waterman, ride, and white
Created by: fryeguy52
CC: @trilinos/muelu , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe
Next Action Status
PR #3498 merged on 9/25/2018 which reduces cost of the expensive BlockCrs unit tests and PR #3517 merged on 9/26/2018 split up MueLu_UnitTests into multiple executables. On 2/27/2018 all MueLu tests (including new split up MueLu_UnitTests*
tests) passe on all promoted "ATDM" builds and all 'waterman' builds.
Description
As shown in this query the tests:
- MueLu_UnitTestsTpetra_MPI_1
- MueLu_UnitTestsTpetra_MPI_4
are failing often in the builds:
- Trilinos-atdm-white-ride-cuda-9.2-opt
- Trilinos-atdm-white-ride-cuda-9.2-debug
the test:
- MueLu_UnitTestsTpetra_MPI_4
is also failing every night on waterman in the builds:
- Trilinos-atdm-waterman-cuda-9.2-opt
- Trilinos-atdm-waterman-cuda-9.2-debug
All of the failures are from timeouts
Steps to Reproduce on white
One should be able to reproduce this failure on the machine white as described in:
- https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md More specifically, the commands given for the system white are provided at:
- https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-9.2-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
Steps to Reproduce on waterman
One should be able to reproduce this failure on the machine waterman as described in:
- https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md More specifically, the commands given for the system white are provided at:
- https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=20
$ bsub -x -Is -n 20 ctest -j20