GitLab

Created by: mhoemmen

@trilinos/framework

I'm not sure if this is a Kokkos (@crtrott, @dsunder, @ibaned ) issue, a Trilinos issue, or a SEMS issue (@jgfouca), but something recently (as of last week) totally messed up CUDA builds for me. This is blocking #1569, #1606, #1367 (closed), #1623 (closed), and a bunch of other issues. It is delaying testing and acceptance of @tjfulle 's and @brian-kelley 's hard work.

When I run make, I get this (replace ... here and elsewhere with the path to Trilinos' source directory):

[  0%] Building CXX object packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_hwloc.cpp.o
cd .../../Trilinos/CHECKIN-CUDA-8.0/MPI_DEBUG_REAL/packages/kokkos/core/src && /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/bin/g++   -Dkokkoscore_EXPORTS -pedantic -Wall -Wno-long-long -Wwrite-strings   -Wall -std=c++11 -expt-extended-lambda -std=c++11 -fopenmp -g -O0 -fPIC -I/scratch/prj/Trilinos/CHECKIN-CUDA-8.0/MPI_DEBUG_REAL -I.../../Trilinos/CHECKIN-CUDA-8.0/MPI_DEBUG_REAL/packages/kokkos/core/src -I.../Trilinos/packages/kokkos/core/src -I/projects/sems/install/rhel6-x86_64/kokkos/compiler/cuda/8.0.44/include    -o CMakeFiles/kokkoscore.dir/impl/Kokkos_hwloc.cpp.o -c .../Trilinos/packages/kokkos/core/src/impl/Kokkos_hwloc.cpp
In file included from .../Trilinos/packages/kokkos/core/src/impl/Kokkos_hwloc.cpp:50:0:
.../Trilinos/packages/kokkos/core/src/Kokkos_Macros.hpp:170:40: error: ‘__device__’ does not name a type
   #define KOKKOS_INLINE_FUNCTION       __device__  __host__  inline
                                        ^
.../Trilinos/packages/kokkos/core/src/impl/Kokkos_Error.hpp:73:1: note: in expansion of macro ‘KOKKOS_INLINE_FUNCTION’
 KOKKOS_INLINE_FUNCTION
 ^

Notice that it's running g++ directly, not nvcc or mpicxx or anything like that. I am using the SEMS modules on my workstation, artemis. Here is the list of loaded modules:

$ module list
Currently Loaded Modulefiles:
  1) sems-env                    3) sems-cmake/3.3.2            5) kokkos-cuda/8.0.44
  2) kokkos-env                  4) sems-gcc/5.3.0              6) kokkos-openmpi/2.0.1/cuda

Here is the output of nvcc --version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Sep__4_22:14:01_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44

Here is the output of echo $OMPI_CXX:

.../Trilinos/packages/kokkos/config/nvcc_wrapper

Here is the output of mpicxx --version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Sep__4_22:14:01_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44

Here are my relevant CMake options:

-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-D BUILD_SHARED_LIBS:BOOL=ON
-D Trilinos_ENABLE_OpenMP:BOOL=ON
-D Trilinos_SHOW_DEPRECATED_WARNINGS:BOOL=ON
-D Trilinos_ENABLE_Fortran:BOOL=OFF
-D CMAKE_VERBOSE_MAKEFILE:BOOL=ON
-D CMAKE_Fortran_LINK_EXECUTABLE:FILEPATH=""
-D TPL_ENABLE_CUDA:BOOL=ON
  -D Kokkos_ENABLE_Cuda_UVM:BOOL=ON
  -D Tpetra_INST_CUDA:BOOL=ON
  -D Kokkos_ENABLE_Cuda_Lambda:BOOL=ON
  -D Kokkos_ENABLE_CXX11_DISPATCH_LAMBDA:BOOL=ON
-D CMAKE_CXX_FLAGS:STRING="-Wall -std=c++11 -expt-extended-lambda"
-D CMAKE_BUILD_TYPE:STRING=DEBUG
-D Kokkos_ENABLE_DEBUG:BOOL=ON
-D Teuchos_ENABLE_DEBUG:BOOL=ON
-D CMAKE_CXX_FLAGS_DEBUG:STRING="-Os"
-D Tpetra_INST_SERIAL:BOOL=OFF
-D Teuchos_ENABLE_COMPLEX:BOOL=OFF
-D Trilinos_ENABLE_COMPLEX_FLOAT:BOOL=OFF
-D Trilinos_ENABLE_COMPLEX_DOUBLE:BOOL=OFF

I have the same trouble with my CUDA 7.5 build. Thus, this issue also hinders #1278 (closed), because we are supposed to keep supporting CUDA 7.5 until at least the next minor release.