Skip to content

GitLab

Closed
Created by James Willenbring@jmwilleMaintainer

Zoltan2 test build errors in all ATDM Trilinos CUDA builds starting 1/22/2019

Created by: bartlettroscoe

CC: @trilinos/zoltan2, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52

Next Action Status

PR #4252 merged to 'develop' on 1/23/2019 resulted in 100% passing Zoltan2 builds and tests on all ATDM Trilinos platforms on 1/23/2019.

Description

The build of several Zoltan2 test object files and executables are broken in the ATDM Trilinos CUDA builds starting 1/22/2018 as shown in the builds:

(NOTE: It is likely broken in all of the other CUDA builds as well but we can't tell because of a configure failure in all of that ATDM Trilinos CUDA builds on 1/22/2019 as described in #4242 (closed). The only reason that the build Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt did not fail at configure time was that it has Fortran enabled and the only reason that the build Trilinos-atdm-white-ride-cuda-10.0-gnu-7.4.0-debug-exp which disables Fortran did not have configure failures is that reverting PR #4238 was merged before this build was run.)

There were also 2 new failing Zoltan2 tests as shown here for the tests:

  • Zoltan2_directoryTest_Kokkos_MPI_4
  • Zoltan2_directoryTest_KokkosSimple_MPI_4

and 33 not-run tests (as you would expect given the build failures).

The new failing tests showed CUDA errors and segfaults like shown for Zoltan2_directoryTest_Kokkos_MPI_4 showing:

...
 Passed: Vector_User_Multiple_GID contiguous int mode: Aggregate
Testing totalIds: 1
  Passed: Single_User_Single_GID contiguous int mode: Replace
  Passed: Single_User_Single_GID non-contiguous int mode: Replace
  Passed: Single_User_Single_GID long long mode: Replace
terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaGetLastError() error( cudaErrorLaunchFailure): unspecified launch failure /home/jenkins/white/workspace/Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaExec.hpp:401
Traceback functionality not available

[white23:35877] *** Process received signal ***
[white23:35877] Signal: Aborted (6)
[white23:35877] Signal code:  (-6)
...

The new commits that were pulled the day that these failures started are show, for example, here which suggest that the breaking commits were from the merged PR #1533.

Current Status on CDash

The current status of these tests/builds for the current testing day can be found here.

Steps to Reproduce

One should be able to reproduce this failure on any of the machines with a CUDA build as described in:

More specifically, the commands given for the system white or ride are provided at:

The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-9.2-gnu-7.2.0-release-debug

$ cmake \
  -GNinja \
  -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
  -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Zoltan2=ON \
  $TRILINOS_DIR

$ make NP=16

$ bsub -x -Is -q rhel7F -n 16 ctest -j16