Sacado test Fad_KokkosTests_NoViewSpec_XXX.cpp CUDA build failures on all CUDA builds starting 11/10/2018
Created by: bartlettroscoe
CC: @trilinos/sacado, @rppawlo (Trilinos Nonlinear Solvers Product Lead)
Next Action Status
Enable of Sacado_ENABLE_HIERARCHICAL_DFAD=ON
in PR #3800 merged on 10/9/2018 triggered this build failure. PR #3845 (closed) (once merged) should fix the failures for all builds ...
Description
As shown in this query the Sacado test suite has two build errors in every Sacado build that builds the Sacado test suite.
The ATDM builds involved (that have submitted to as of 9:20 AM MT on 11/10/2018) include:
- hansen | Trilinos-atdm-hansen-shiller-cuda-8.0-debug
- hansen | Trilinos-atdm-hansen-shiller-cuda-8.0-opt
- hansen | Trilinos-atdm-hansen-shiller-cuda-9.0-debug
- hansen | Trilinos-atdm-hansen-shiller-cuda-9.0-opt
- waterman | Trilinos-atdm-waterman-cuda-9.2-debug
- waterman | Trilinos-atdm-waterman-cuda-9.2-opt
- white | Trilinos-atdm-white-ride-cuda-9.2-debug
- white | Trilinos-atdm-white-ride-cuda-9.2-opt
- waterman | Trilinos-atdm-waterman-cuda-9.2-release-debug
- ride | Trilinos-atdm-white-ride-cuda-9.2-debug
- ride | Trilinos-atdm-white-ride-cuda-9.2-debug-pt
- white | Trilinos-atdm-white-ride-cuda-9.2-debug-pt
- ride | Trilinos-atdm-white-ride-cuda-9.2-opt
Some of the non-ATDM Trilinos builds showing this build error include:
- ascic141 | Linux-gcc-7.3.0-Sierra_MPI_release_DEV_ETI_SERIAL-ON_OPENMP-OFF_PTHREAD-OFF_CUDA-OFF_COMPLEX-ON
- ascic141 | Linux-gcc-7.3.0-Sierra_MPI_release_DEV_ETI_SERIAL-ON_OPENMP-ON_PTHREAD-OFF_CUDA-OFF_COMPLEX-ON
This results in the Not Run tests:
- Sacado_FadKokkosTests_NoViewSpec_Cuda_MPI_1
- Sacado_FadKokkosTests_NoViewSpec_Serial_MPI_1
in all of these builds.
The build error that occurs is in the files sacado/test/UnitTests/Fad_KokkosTests_NoViewSpec_Cuda.cpp
and sacado/test/UnitTests/Fad_KokkosTests_NoViewSpec_Serial.cpp
shown at, for example, here which shows:
home/jenkins/waterman/workspace/Trilinos-atdm-waterman-cuda-9.2-release-debug/SRC_AND_BUILD/Trilinos/packages/sacado/test/UnitTests/Fad_KokkosTests.hpp(223): error: namespace "Kokkos" has no member "is_dynrankview_fad_contiguous"
detected during instantiation of "void Kokkos_View_Fad_ScalarAssign_UnitTest<FadType, Layout,
...
15 errors detected in the compilation of "/tmp/tmpxft_000167b4_00000000-6_Fad_KokkosTests_NoViewSpec_Serial.cpp1.ii".
The new commits that were pulled the day that these failures started are show, for example, here. From looking over that set of commits, there are no changes to the Sacado code or tests. Therefore, one might assume that somehow the PR #3800 merged on 10/9/2018 must have triggered these failures?
Current Status on CDash
To see the current status of all of these Sacado CUDA builds, see:
NOTE: To see the full status, one will need to look at the previous testing day (i.e. click above link and then click "Previous" at the top.
Steps to Reproduce
One should be able to reproduce this failure on any of the ATDM Trilinos platforms that has a CUDA build as described in:
For example, on 'white' or 'ride' as described at:
the exact commands to reproduce this build error are:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-9.2-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Sacado=ON \
$TRILINOS_DIR
$ make NP=16
But, again, any of the CUDA builds on any of the ATDM Trilinos testing platforms should be able to reproduce this build failure.