Build Trilinos on white with CUDA BLAS
Created by: CamelliaDPG
I'd like to build Trilinos on Sandia's white machine with Tpetra, MueLu, and Belos, using CUDA BLAS.
@ibaned suggested I post an issue for this, pinging @nmhamster, @mhoemmen, and @crtrott.
So far, my configure script is as follows:
source white-modules-cublas
DAN_HOME=/home/daibane
cmake $DAN_HOME/src/Trilinos \
-DCMAKE_INSTALL_PREFIX:PATH=$HOME/trilinos-install/tpetra-muelu-belos-cublas \
-DCMAKE_BUILD_TYPE:STRING=NONE \
-DCMAKE_CXX_COMPILER:FILEPATH=$DAN_HOME/src/alexa/doc/dashboards/nvcc_wrapper_60 \
-DCMAKE_C_COMPILER:FILEPATH=`which mpicc` \
-DCMAKE_CXX_FLAGS:STRING="-O3 -g" \
-DCMAKE_C_FLAGS:STRING="-O3 -g" \
-DTrilinos_CXX11_FLAGS:STRING="-std=c++11 -expt-extended-lambda" \
-DTrilinos_ENABLE_Fortran:BOOL=OFF \
-DBUILD_SHARED_LIBS:BOOL=ON \
-DTPL_ENABLE_MPI:BOOL=ON \
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
-DTrilinos_ENABLE_OpenMP:BOOL=OFF \
-DTrilinos_ENABLE_Belos:BOOL=ON \
-DTrilinos_ENABLE_Ifpack2:BOOL=ON \
-DTrilinos_ENABLE_KokkosCore:BOOL=ON \
-DTrilinos_ENABLE_KokkosContainers:BOOL=ON \
-DTrilinos_ENABLE_MueLu:BOOL=ON \
-DTrilinos_ENABLE_Teuchos:BOOL=ON \
-DTrilinos_ENABLE_TeuchosParser:BOOL=ON \
-DTrilinos_ENABLE_Tpetra:BOOL=ON \
-DTpetra_INST_SERIAL:BOOL=ON \
-DKokkos_ENABLE_Serial:BOOL=ON \
-DKokkos_ENABLE_OpenMP:BOOL=OFF \
-DKokkos_ENABLE_Pthread:BOOL=OFF \
-DKokkos_ENABLE_Cuda:BOOL=ON \
-DKokkos_ENABLE_Cuda_UVM:BOOL=ON \
-DKokkos_ENABLE_Cuda_Lambda:BOOL=ON \
-DTPL_ENABLE_CUDA:BOOL=ON \
-DTPL_ENABLE_yaml-cpp:BOOL=ON \
-DBLAS_LIBRARY_DIRS:PATH=$CUDA_ROOT/lib64/ \
-DBLAS_LIBRARY_NAMES=cublas \
-DLAPACK_LIBRARY_DIRS:PATH=$CUDA_ROOT/lib64/ \
-DLAPACK_LIBRARY_NAMES=cusolver \
-Dyaml-cpp_INCLUDE_DIRS:PATH=$DAN_HOME/install/yaml-cpp/include \
-Dyaml-cpp_LIBRARY_DIRS:PATH=$DAN_HOME/install/yaml-cpp/lib \
2>&1 | tee config_log
where white-modules-cublas
is as follows:
module purge
module load cmake/3.6.2
module load openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
With these options, the Trilinos build and install succeed. The issue comes when I try to link my executable against Trilinos; then I get missing LAPACK symbol errors such as the following:
/ascldap/users/nvrober/trilinos-install/tpetra-muelu-belos-cublas/lib/libteuchosnumerics.so.12.11: undefined reference to `dgerfs_'
/ascldap/users/nvrober/trilinos-install/tpetra-muelu-belos-cublas/lib/libteuchosnumerics.so.12.11: undefined reference to `dorm2r_'
/ascldap/users/nvrober/trilinos-install/tpetra-muelu-belos-cublas/lib/libteuchosnumerics.so.12.11: undefined reference to `sormqr_'
/ascldap/users/nvrober/trilinos-install/tpetra-muelu-belos-cublas/lib/libteuchosnumerics.so.12.11: undefined reference to `sgebak_'
/ascldap/users/nvrober/trilinos-install/tpetra-muelu-belos-cublas/lib/libteuchosnumerics.so.12.11: undefined reference to `dgeequ_'
/ascldap/users/nvrober/trilinos-install/tpetra-muelu-belos-cublas/lib/libteuchosnumerics.so.12.11: undefined reference to `sgbequ_'
/ascldap/users/nvrober/trilinos-install/tpetra-muelu-belos-cublas/lib/libteuchosnumerics.so.12.11: undefined reference to `sormhr_'
I'd much appreciate any guidance or advice you can provide. Thanks!