Zoltan2 test build errors in all ATDM Trilinos CUDA builds starting 1/22/2019
Created by: bartlettroscoe
CC: @trilinos/zoltan2, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
Next Action Status
PR #4252 merged to 'develop' on 1/23/2019 resulted in 100% passing Zoltan2 builds and tests on all ATDM Trilinos platforms on 1/23/2019.
Description
The build of several Zoltan2 test object files and executables are broken in the ATDM Trilinos CUDA builds starting 1/22/2018 as shown in the builds:
- Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt
- Trilinos-atdm-white-ride-cuda-10.0-gnu-7.4.0-debug-exp
(NOTE: It is likely broken in all of the other CUDA builds as well but we can't tell because of a configure failure in all of that ATDM Trilinos CUDA builds on 1/22/2019 as described in #4242 (closed). The only reason that the build Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt
did not fail at configure time was that it has Fortran enabled and the only reason that the build Trilinos-atdm-white-ride-cuda-10.0-gnu-7.4.0-debug-exp
which disables Fortran did not have configure failures is that reverting PR #4238 was merged before this build was run.)
There were also 2 new failing Zoltan2 tests as shown here for the tests:
- Zoltan2_directoryTest_Kokkos_MPI_4
- Zoltan2_directoryTest_KokkosSimple_MPI_4
and 33 not-run tests (as you would expect given the build failures).
The new failing tests showed CUDA errors and segfaults like shown for Zoltan2_directoryTest_Kokkos_MPI_4 showing:
...
Passed: Vector_User_Multiple_GID contiguous int mode: Aggregate
Testing totalIds: 1
Passed: Single_User_Single_GID contiguous int mode: Replace
Passed: Single_User_Single_GID non-contiguous int mode: Replace
Passed: Single_User_Single_GID long long mode: Replace
terminate called after throwing an instance of 'std::runtime_error'
what(): cudaGetLastError() error( cudaErrorLaunchFailure): unspecified launch failure /home/jenkins/white/workspace/Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaExec.hpp:401
Traceback functionality not available
[white23:35877] *** Process received signal ***
[white23:35877] Signal: Aborted (6)
[white23:35877] Signal code: (-6)
...
The new commits that were pulled the day that these failures started are show, for example, here which suggest that the breaking commits were from the merged PR #1533.
Current Status on CDash
The current status of these tests/builds for the current testing day can be found here.
Steps to Reproduce
One should be able to reproduce this failure on any of the machines with a CUDA build as described in:
More specifically, the commands given for the system white
or ride
are provided at:
The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-9.2-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Zoltan2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16