CUDA config & build are broken
Created by: mhoemmen
@trilinos/framework
I'm not sure if this is a Kokkos (@crtrott, @dsunder, @ibaned ) issue, a Trilinos issue, or a SEMS issue (@jgfouca), but something recently (as of last week) totally messed up CUDA builds for me. This is blocking #1569, #1606, #1367 (closed), #1623 (closed), and a bunch of other issues. It is delaying testing and acceptance of @tjfulle 's and @brian-kelley 's hard work.
When I run make
, I get this (replace ...
here and elsewhere with the path to Trilinos' source directory):
[ 0%] Building CXX object packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_hwloc.cpp.o
cd .../../Trilinos/CHECKIN-CUDA-8.0/MPI_DEBUG_REAL/packages/kokkos/core/src && /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/bin/g++ -Dkokkoscore_EXPORTS -pedantic -Wall -Wno-long-long -Wwrite-strings -Wall -std=c++11 -expt-extended-lambda -std=c++11 -fopenmp -g -O0 -fPIC -I/scratch/prj/Trilinos/CHECKIN-CUDA-8.0/MPI_DEBUG_REAL -I.../../Trilinos/CHECKIN-CUDA-8.0/MPI_DEBUG_REAL/packages/kokkos/core/src -I.../Trilinos/packages/kokkos/core/src -I/projects/sems/install/rhel6-x86_64/kokkos/compiler/cuda/8.0.44/include -o CMakeFiles/kokkoscore.dir/impl/Kokkos_hwloc.cpp.o -c .../Trilinos/packages/kokkos/core/src/impl/Kokkos_hwloc.cpp
In file included from .../Trilinos/packages/kokkos/core/src/impl/Kokkos_hwloc.cpp:50:0:
.../Trilinos/packages/kokkos/core/src/Kokkos_Macros.hpp:170:40: error: ‘__device__’ does not name a type
#define KOKKOS_INLINE_FUNCTION __device__ __host__ inline
^
.../Trilinos/packages/kokkos/core/src/impl/Kokkos_Error.hpp:73:1: note: in expansion of macro ‘KOKKOS_INLINE_FUNCTION’
KOKKOS_INLINE_FUNCTION
^
Notice that it's running g++
directly, not nvcc
or mpicxx
or anything like that. I am using the SEMS modules on my workstation, artemis
. Here is the list of loaded modules:
$ module list
Currently Loaded Modulefiles:
1) sems-env 3) sems-cmake/3.3.2 5) kokkos-cuda/8.0.44
2) kokkos-env 4) sems-gcc/5.3.0 6) kokkos-openmpi/2.0.1/cuda
Here is the output of nvcc --version
:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Sep__4_22:14:01_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44
Here is the output of echo $OMPI_CXX
:
.../Trilinos/packages/kokkos/config/nvcc_wrapper
Here is the output of mpicxx --version
:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Sep__4_22:14:01_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44
Here are my relevant CMake options:
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-D BUILD_SHARED_LIBS:BOOL=ON
-D Trilinos_ENABLE_OpenMP:BOOL=ON
-D Trilinos_SHOW_DEPRECATED_WARNINGS:BOOL=ON
-D Trilinos_ENABLE_Fortran:BOOL=OFF
-D CMAKE_VERBOSE_MAKEFILE:BOOL=ON
-D CMAKE_Fortran_LINK_EXECUTABLE:FILEPATH=""
-D TPL_ENABLE_CUDA:BOOL=ON
-D Kokkos_ENABLE_Cuda_UVM:BOOL=ON
-D Tpetra_INST_CUDA:BOOL=ON
-D Kokkos_ENABLE_Cuda_Lambda:BOOL=ON
-D Kokkos_ENABLE_CXX11_DISPATCH_LAMBDA:BOOL=ON
-D CMAKE_CXX_FLAGS:STRING="-Wall -std=c++11 -expt-extended-lambda"
-D CMAKE_BUILD_TYPE:STRING=DEBUG
-D Kokkos_ENABLE_DEBUG:BOOL=ON
-D Teuchos_ENABLE_DEBUG:BOOL=ON
-D CMAKE_CXX_FLAGS_DEBUG:STRING="-Os"
-D Tpetra_INST_SERIAL:BOOL=OFF
-D Teuchos_ENABLE_COMPLEX:BOOL=OFF
-D Trilinos_ENABLE_COMPLEX_FLOAT:BOOL=OFF
-D Trilinos_ENABLE_COMPLEX_DOUBLE:BOOL=OFF
I have the same trouble with my CUDA 7.5 build. Thus, this issue also hinders #1278 (closed), because we are supposed to keep supporting CUDA 7.5 until at least the next minor release.