Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2017-03-05T22:12:00Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/1099KokkosKernels pullout broke my configures2017-03-05T22:12:00ZJames WillenbringKokkosKernels pullout broke my configures*Created by: bathmatt*
Any idea on what I need to add to my scripts to get configure to work. It is a silent failure, by that I mean it isn't obvious what the problem is, and my CMakeErrors.log looks pretty much the same before and aft...*Created by: bathmatt*
Any idea on what I need to add to my scripts to get configure to work. It is a silent failure, by that I mean it isn't obvious what the problem is, and my CMakeErrors.log looks pretty much the same before and after the commit.
I added Trilinos_ENABLE_KokkosKernels=ON
Set up for creating a distribution ...
tail of the configure.
`
Finished configuring Trilinos!
-- Configuring incomplete, errors occurred!
See also "/jenkins/slave/workspace/EMPIRE-debug/build_drekar/CMakeFiles/CMakeOutput.log".
See also "/jenkins/slave/workspace/EMPIRE-debug/build_drekar/CMakeFiles/CMakeError.log".
Error in configuration of drekar, trying to continue`
Here is my configure script
[config.txt](https://github.com/trilinos/Trilinos/files/811733/config.txt)
and all the log files. log has been renamed to txt
[configure.txt](https://github.com/trilinos/Trilinos/files/811735/configure.txt)
[CMakeError.txt](https://github.com/trilinos/Trilinos/files/811737/CMakeError.txt)
[CMakeOutput.txt](https://github.com/trilinos/Trilinos/files/811738/CMakeOutput.txt)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/980KokkosKernels: Sparse matrix-matrix multiply performance test build error2017-04-13T18:43:39ZJames WillenbringKokkosKernels: Sparse matrix-matrix multiply performance test build error*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/tpetra
Lots of errors like this:
```
.../Trilinos/packages/tpetra/kernels/perf_test/graph/KokkosKernels_SPMM_KK.cpp: In function ‘int main(int, char**)’:
.../Trilinos/packages/tpet...*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/tpetra
Lots of errors like this:
```
.../Trilinos/packages/tpetra/kernels/perf_test/graph/KokkosKernels_SPMM_KK.cpp: In function ‘int main(int, char**)’:
.../Trilinos/packages/tpetra/kernels/perf_test/graph/KokkosKernels_SPMM_KK.cpp:688:13: error: ‘Kokkos::OpenMP’ has not been declared
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/978KokkosKernels: Sparse matrix-matrix multiply test build error2017-04-13T18:42:58ZJames WillenbringKokkosKernels: Sparse matrix-matrix multiply test build error*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
KokkosKernels' sparse matrix-matrix multiply test has a build error:
```
.../Trilinos/packages/tpetra/kernels/src/stage/graph/utils/KokkosKernels_SimpleUtils.hpp:50:41: error:...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
KokkosKernels' sparse matrix-matrix multiply test has a build error:
```
.../Trilinos/packages/tpetra/kernels/src/stage/graph/utils/KokkosKernels_SimpleUtils.hpp:50:41: error: no match for ‘operator>’ (operand types are ‘Kokkos::ViewTraits<Kokkos::complex<double>*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> >::value_type {aka Kokkos::complex<double>}’ and ‘int’)
#define KOKKOSKERNELS_MACRO_ABS(x) ((x) > (0) ? (x): (-x))
^
.../Trilinos/packages/tpetra/kernels/src/stage/graph/utils/KokkosKernels_SimpleUtils.hpp:133:9: note: in expansion of macro ‘KOKKOSKERNELS_MACRO_ABS’
if (KOKKOSKERNELS_MACRO_ABS (val_diff) > eps) {
...
```
This may just be an issue with the test not accounting for complex numbers.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3438KokkosKernels_sparse_* tests are failing on ATDM cuda builds2018-11-30T03:09:58ZJames WillenbringKokkosKernels_sparse_* tests are failing on ATDM cuda builds*Created by: fryeguy52*
CC: @trilinos/kokkos-kernels , @kddevin (Trilinos Data Services Product Lead)
## Next Action Status
The test `KokkosKernels_sparse_serial_MPI_1` on 'waterman' has been passing without timing out in each '...*Created by: fryeguy52*
CC: @trilinos/kokkos-kernels , @kddevin (Trilinos Data Services Product Lead)
## Next Action Status
The test `KokkosKernels_sparse_serial_MPI_1` on 'waterman' has been passing without timing out in each 'debug' build on 'waterman' since 10/9/2018 as shown [here](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&date=2018-10-19&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-waterman-&field2=buildname&compare2=63&value2=debug&field3=testname&compare3=61&value3=KokkosKernels_sparse_serial_MPI_1&field4=buildstarttime&compare4=83&value4=2018-10-01).
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-09-12&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=Trilinos-atdm-&field2=testname&compare2=63&value2=KokkosKernels_sparse_&field3=status&compare3=62&value3=Passed) the tests:
* KokkosKernels_sparse_serial_MPI_1
* KokkosKernels_sparse_cuda_MPI_1
are failing in all the cuda builds on white, ride, hansen, and waterman:
* Trilinos-atdm-waterman-gnu-debug-openmp
* Trilinos-atdm-waterman-cuda-9.2-debug
* Trilinos-atdm-waterman-cuda-9.2-opt
* Trilinos-atdm-white-ride-cuda-9.2-opt
* Trilinos-atdm-white-ride-cuda-9.2-debug-pt
* Trilinos-atdm-white-ride-cuda-9.2-debug
* Trilinos-atdm-hansen-shiller-cuda-8.0-opt
* Trilinos-atdm-hansen-shiller-cuda-9.0-debug
* Trilinos-atdm-hansen-shiller-cuda-9.0-opt
[Here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-09-04&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=7&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=cuda&field3=buildname&compare3=64&value3=hansen&field4=buildname&compare4=64&value4=waterman&field5=testname&compare5=63&value5=KokkosKernels_sparse_cuda&field6=buildstarttime&compare6=84&value6=2018-09-13&field7=buildstarttime&compare7=83&value7=2018-09-05) you can see that these tests started failing on 9/7/2018
at the bottom of [this page](https://testing.sandia.gov/cdash/viewNotes.php?buildid=3912160) is a list of commits that were new on that day.
## Steps to Reproduce
One should be able to reproduce this failure as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_KokkosKernels=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/948KokkosKernels: SPMV performance comparison test2017-01-06T00:12:10ZJames WillenbringKokkosKernels: SPMV performance comparison test*Created by: crtrott*
This adds a performance comparison test without MPI for just SPMV kernel. Can compare the KokkosKernel function, a custom Kokkos implementation, CuSparse and MKL.
This can read MatrixMarket format, and also gener...*Created by: crtrott*
This adds a performance comparison test without MPI for just SPMV kernel. Can compare the KokkosKernel function, a custom Kokkos implementation, CuSparse and MKL.
This can read MatrixMarket format, and also generate a binary matrix storage file, which it can read again. Depending on your system reading the binary format back in can be 10x faster than the text file.
I had that written quite a while ago I just cleaned it up now. https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3864KokkosKernels tests randomly timing out on ATDM mutrino KNL build2019-04-02T19:15:07ZJames WillenbringKokkosKernels tests randomly timing out on ATDM mutrino KNL build*Created by: fryeguy52*
CC: @trilinos/<package-name>, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.ph...*Created by: fryeguy52*
CC: @trilinos/<package-name>, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=65&value2=KokkosKernels_&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00&field6=details&compare6=61&value6=Completed%20%28Timeout%29) the tests:
* [KokkosKernels_sparse_serial_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_sparse_serial_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00:00:00&field5=buildstarttime&compare5=83&value5=2018-10-14T00:00:00)
* [KokkosKernels_sparse_openmp_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_sparse_openmp_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
* [KokkosKernels_graph_serial_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_graph_serial_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
* [KokkosKernels_graph_openmp_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_graph_openmp_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00:00:00&field5=buildstarttime&compare5=83&value5=2018-10-14T00:00:00)
* [KokkosKernels_common_serial_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_common_serial_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
* [KokkosKernels_common_openmp_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_common_openmp_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00:00:00&field5=buildstarttime&compare5=83&value5=2018-10-14T00:00:00)
* [KokkosKernels_blas_serial_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_blas_serial_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
* [KokkosKernels_blas_openmp_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_blas_openmp_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
are randomly timing out in the build:
* Trilinos-atdm-mutrino-intel-opt-openmp-KNL
Links above are to a 30 day history of each test. The `KokkosKernels_*_serial_MPI_1` tests are nearly 3 times as likely to timeout over the last 30 days vs the `KokkosKernels_*_openmp_MPI_1` tests. On average 3 of the 8 tests fail and there has been only one day in that last 30 when all passed.
## Current Status on CDash
The current status of these tests/builds for the current testing day can be found at:
[Current test status](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=65&value2=KokkosKernels_&field3=buildstarttime&compare3=83&value3=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on the machine mutrino as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system mutrino are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#<mutrino
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-mutrino-intel-opt-openmp-KNL
$ cmake \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_KokkosKernels=ON \
$TRILINOS_DIR
$ make -j16
$ salloc -N 1 -p standard -J Trilinos-atdm-mutrino-intel-opt-openmp-KNL ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/1729kokkos-kernels: Turn off unneeded instantiations by default2017-10-26T21:10:54ZJames Willenbringkokkos-kernels: Turn off unneeded instantiations by default*Created by: mhoemmen*
@trilinos/tpetra
Trilinos doesn't need the following instantiations:
- `CudaSpace` (not used, Trilinos assumes UVM and uses `CudaUVMSpace`)
- Unused offset types
See `${PACKAGE_NAME}_INST_MEMSPACE_C...*Created by: mhoemmen*
@trilinos/tpetra
Trilinos doesn't need the following instantiations:
- `CudaSpace` (not used, Trilinos assumes UVM and uses `CudaUVMSpace`)
- Unused offset types
See `${PACKAGE_NAME}_INST_MEMSPACE_CUDASPACE` in `kokkos-kernels/CMakeLists.txt`. I'm not sure if Tpetra can control these directly; it may be necessary to change defaults in kokkos-kernels.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/759KokkosKernels unit tests' dependency to kokkos gtest build2017-04-13T18:45:04ZJames WillenbringKokkosKernels unit tests' dependency to kokkos gtest build*Created by: mndevec*
Kokkos is using gtest for the unit tests. I think we talked briefly that we will use the same unit test harness for KokkosKernels, rather than introducing a dependency on Teuchos.
I am trying to add unit tests ...*Created by: mndevec*
Kokkos is using gtest for the unit tests. I think we talked briefly that we will use the same unit test harness for KokkosKernels, rather than introducing a dependency on Teuchos.
I am trying to add unit tests that use gtest, assuming that kokkos builds gtest and I can access the prebuild kokkos_gtest library. However, it is only compiled when Kokkos test is enabled.
I don't know an elegant way to handle this, I was wondering how this should be handled.
@srajama1 @mhoemmen @crtrott @ambrad @kyungjoo-kim
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/667KokkosKernels: Use existing macros for TPLs like MKL2017-09-01T19:25:43ZJames WillenbringKokkosKernels: Use existing macros for TPLs like MKL*Created by: mhoemmen*
@trilinos/tpetra
I'm looking in tpetra/kernels/src/stage/graph/impl/KokkosKernels_SPGEMM_mkl_impl.hpp. I notice a macro KERNELS_HAVE_MKL. This is some macro that gets defined somewhere by hand in the code. It...*Created by: mhoemmen*
@trilinos/tpetra
I'm looking in tpetra/kernels/src/stage/graph/impl/KokkosKernels_SPGEMM_mkl_impl.hpp. I notice a macro KERNELS_HAVE_MKL. This is some macro that gets defined somewhere by hand in the code. It is not plugged into the CMake build system at all. KokkosKernels already has a macro, HAVE_TPETRAKERNELS_MKL. Please use that instead. "KERNELS_HAVE_MKL" is too general of a name; it is likely to collide with other software library's macro.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1818kokkos-kernels vs kokkoskernels2017-11-01T15:34:48ZJames Willenbringkokkos-kernels vs kokkoskernels*Created by: nschloe*
There is a discrepancy in files names for the new Kokkos kernels:
* The package directory is called `kokkos-kernels`.
* The library file name is `kokkoskernels.so`.
This creates some problems on the Debi...*Created by: nschloe*
There is a discrepancy in files names for the new Kokkos kernels:
* The package directory is called `kokkos-kernels`.
* The library file name is `kokkoskernels.so`.
This creates some problems on the Debian side. I'd be great if this could be made consistent one way or the other. For comparison, check `trilinoscouplings`.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1170KokkosKernels: warnings on CUDA2017-10-27T04:19:03ZJames WillenbringKokkosKernels: warnings on CUDA*Created by: rppawlo*
When compiling with Tpetra enabled for cuda, we get a ton of warnings from kokkos kernels. It seems to stem from the three lines below. Can someone take a look at this?
```
[ 69%] Built target locathyra
/ascldap...*Created by: rppawlo*
When compiling with Tpetra enabled for cuda, we get a ton of warnings from kokkos kernels. It seems to stem from the three lines below. Can someone take a look at this?
```
[ 69%] Built target locathyra
/ascldap/users/rppawlo/Trilinos/packages/kokkos-kernels/src/Kokkos_ArithTraits.hpp(1480): warning: pointless comparison of unsigned integer with zero
/ascldap/users/rppawlo/Trilinos/packages/kokkos-kernels/src/Kokkos_ArithTraits.hpp(157): warning: pointless comparison of unsigned integer with zero
detected during instantiation of "IntType <unnamed>::intPowSigned(IntType, IntType) [with IntType=char]"
(1506): here
/ascldap/users/rppawlo/Trilinos/packages/kokkos-kernels/src/Kokkos_ArithTraits.hpp(161): warning: pointless comparison of unsigned integer with a negative constant
detected during instantiation of "IntType <unnamed>::intPowSigned(IntType, IntType) [with IntType=char]"
(1506): here
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2735Kokkos Promotion: 2.6 -> 2.72018-05-25T17:09:16ZJames WillenbringKokkos Promotion: 2.6 -> 2.7*Created by: crtrott*
This is to track progress on the Kokkos promotion. Part of this promotion is to remove utilization of deprecated features from Trilinos. See https://github.com/kokkos/kokkos/wiki/DeprecationPage for a list of featu...*Created by: crtrott*
This is to track progress on the Kokkos promotion. Part of this promotion is to remove utilization of deprecated features from Trilinos. See https://github.com/kokkos/kokkos/wiki/DeprecationPage for a list of features (may not yet be complete). The branch for this is kokkos-promotion. Add a symlink into the root Trilinos directory to a separate git clone of kokkos and Kokkos-kernels with their develop branches checked out. Then add
```
-D Kokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \
-D KokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels \
-D KOKKOS_ENABLE_DEPRECATED_CODE:BOOL=OFF \
```
To your configure script. https://gitlab.osti.gov/jmwille/Trilinos/-/issues/964Kokkos::StaticCrsGraph does not use memory space template parameter passed to...2017-01-10T20:54:12ZJames WillenbringKokkos::StaticCrsGraph does not use memory space template parameter passed to KokkosSparse::CrsMatrix*Created by: mndevec*
I am providing a device to CrsMatrix in the constructor. This is different than default, where device is either Kokkos::Cuda with hostpinned space, or Kokkos::OpenMP with Kokkos::Hostspace when HBM is enabled.
...*Created by: mndevec*
I am providing a device to CrsMatrix in the constructor. This is different than default, where device is either Kokkos::Cuda with hostpinned space, or Kokkos::OpenMP with Kokkos::Hostspace when HBM is enabled.
I was expecting CrsMatrix memory to be allocated at the memory space I provide. However, this holds only for the values view, while row pointers and entries are still allocated at the default memory space of the provided execution space.
It seems that StaticCrsGraph do not take the device as template argument, instead it is provided the execution space. It creates a default device, as a result allocated memories diverge for values and entries views.
Shouldn't StaticCrsGraph take the device as template argument instead of execution space?
@srajama1 @crtrott @mhoemmen
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4571Long file names in KokkosKernels have broken git clone of Trilinos on Windows2019-03-07T19:52:38ZJames WillenbringLong file names in KokkosKernels have broken git clone of Trilinos on Windows*Created by: bartlettroscoe*
CC: @trilinos/framework, @trilinos/kokkos-kernels
## Description
It appears that the Trilinos git repo has gotten into a state where it can't even be cloned on Windows systems with git.
The CASL PH...*Created by: bartlettroscoe*
CC: @trilinos/framework, @trilinos/kokkos-kernels
## Description
It appears that the Trilinos git repo has gotten into a state where it can't even be cloned on Windows systems with git.
The CASL PHI INF lead @lefebvrera reported the following error when trying to clone the current Trilinos git repo just a short time ago:
```
G:\raq\scale_dev\dev>git clone https://github.com/trilinos/Trilinos.git Trilinos
Cloning into 'Trilinos'...
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 957120 (delta 17), reused 14 (delta 7), pack-reused 957073
Receiving objects: 100% (957120/957120), 562.05 MiB | 26.56 MiB/s, done.
Resolving deltas: 100% (771153/771153), done.
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_numeric/KokkosSparse_gauss_seidel_numeric_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
Checking out files: 100% (56578/56578), done.
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2683MueLu failing to compile on cuda-92018-11-30T03:09:57ZJames WillenbringMueLu failing to compile on cuda-9*Created by: bathmatt*
Not sure how long it's been failing
```
muelu/src/Utils/ExplicitInstantiation/MueLu_Utilities_kokkos.cpp
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(369): warning...*Created by: bathmatt*
Not sure how long it's been failing
```
muelu/src/Utils/ExplicitInstantiation/MueLu_Utilities_kokkos.cpp
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(369): warning: calling a __host__ function("__builtin_cabs") from a __host__ __device__ function(" const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(387): warning: calling a __host__ function("__builtin_cabs") from a __host__ __device__ function(" const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(369): warning: calling a __host__ function("__builtin_cabs") from a __host__ __device__ function(" const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(387): warning: calling a __host__ function("__builtin_cabs") from a __host__ __device__ function(" const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(385): error: calling a __host__ function("__builtin_cabs") from a __device__ function("Kokkos::Impl::ParallelFor< ::, ::Kokkos::RangePolicy<int, ::Kokkos::Cuda > , ::Kokkos::Cuda> ::operator () const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(385): error: calling a __host__ function("__builtin_cabs") from a __device__ function("Kokkos::Impl::ParallelFor< ::, ::Kokkos::RangePolicy<int, ::Kokkos::Cuda > , ::Kokkos::Cuda> ::operator () const") is not allowed
2 errors detected in the compilation of "/tmp/tmpxft_00008577_00000000-6_MueLu_Utilities_kokkos.cpp1.ii".
ninja: build stopped: subcommand failed.
```Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3912MueLu: spgemm OpenMP bug, now fixed2018-11-21T16:51:55ZJames WillenbringMueLu: spgemm OpenMP bug, now fixed*Created by: mhoemmen*
@trilinos/muelu @trilinos/tpetra
Mirror of the following kokkos-kernels issue, now fixed: https://github.com/kokkos/kokkos-kernels/issues/345*Created by: mhoemmen*
@trilinos/muelu @trilinos/tpetra
Mirror of the following kokkos-kernels issue, now fixed: https://github.com/kokkos/kokkos-kernels/issues/345https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2894MueLu: types mismatch in Driver.cpp equilibration2018-06-06T17:13:43ZJames WillenbringMueLu: types mismatch in Driver.cpp equilibration*Created by: lucbv*
@trilinos/muelu
## Expectations
The vector and matrix Scalar types forming a linear system should be consistent.
## Current Behavior
It seems that a call to KokkosBlas::abs is done on a `Scalar` type vector ...*Created by: lucbv*
@trilinos/muelu
## Expectations
The vector and matrix Scalar types forming a linear system should be consistent.
## Current Behavior
It seems that a call to KokkosBlas::abs is done on a `Scalar` type vector and a `magnitude` type vector.
## Motivation and Context
The code is not compiling properly when `std::complex<>` is used
## Definition of Done
- [ ] MueLu Driver compiles
## Possible Solution
My guess is that in MueLu_Driver.cpp on line 167 where the call to KokkosBlas::abs() is made, the two vectors should use the same `Scalar` type. Most likely the magnitude type needs to be replaced by a Scalar type even if this means that in the case of complex numbers only the real part is non zero.
## Steps to Reproduce
Build MueLu with tests and examples on and with `Trilinos_ENABLE_Complex=ON`.
## Your Environment
See builds on cdashhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2471New failing tests in ATDM debug builds of Trilinos due to KOKKOS_ENABLE_DEBUG...2018-11-30T03:09:20ZJames WillenbringNew failing tests in ATDM debug builds of Trilinos due to KOKKOS_ENABLE_DEBUG=ON being set*Created by: bartlettroscoe*
**CC:** @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/amesos2 , @trilinos/anasazi, @trilinos/panzer
## Next Action Status
The PR #2476 fixed two of the tests on 3/30/2018 and PR #2494 disabled...*Created by: bartlettroscoe*
**CC:** @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/amesos2 , @trilinos/anasazi, @trilinos/panzer
## Next Action Status
The PR #2476 fixed two of the tests on 3/30/2018 and PR #2494 disabled one single unit test on 4/3/2018 not appropriate to run on GPUs.
## Description
As shown in the query:
* https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-28&limit=0&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=-debug&field3=status&compare3=62&value3=passed
several tests are timing out today and failing in the ATDM `-debug` builds of Trilinos:
* `Amesos2_KLU2_UnitTests_MPI_2`
* `Anasazi_Epetra_ModalSolversTester_MPI_4`
* `KokkosCore_UnitTest_Cuda_MPI_1`
* `KokkosKernels_sparse_cuda_MPI_1`
* `PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4`
The set of tests that are failing and which platforms they are failing shown in the above query are shown in the below table:
<details>
<summary>
<b>Table of failing tests</b> (click to expend)
</summary>
| Site | Build Name | Test Name | Status | Time | Details |
| :-- | :-- | :-- | :-- | --: | :-- |
| hansen | Trilinos-atdm-hansen-shiller-cuda-debug | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.09 | Completed (Timeout) |
| hansen | Trilinos-atdm-hansen-shiller-gnu-debug-openmp | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 8.51 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-gnu-debug-serial | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.51 | Completed (Timeout) |
| hansen | Trilinos-atdm-hansen-shiller-intel-debug-openmp | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 2.39 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-intel-debug-serial | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.1 | Completed (Timeout) |
| ride | Trilinos-atdm-white-ride-cuda-debug | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.05 | Completed (Timeout) |
| white | Trilinos-atdm-white-ride-cuda-debug | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.04 | Completed (Timeout) |
| white | Trilinos-atdm-white-ride-gnu-debug-openmp | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 1.31 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | Anasazi_Epetra_ModalSolversTester_MPI_4 | Failed | 0.84 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4 | Failed | 0.71 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-cuda-debug | KokkosCore_UnitTest_Cuda_MPI_1 | Failed | 103.19 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | KokkosCore_UnitTest_Cuda_MPI_1 | Failed | 213.56 | Completed (Failed) |
| white | Trilinos-atdm-white-ride-cuda-debug | KokkosCore_UnitTest_Cuda_MPI_1 | Failed | 213.74 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-cuda-debug | KokkosKernels_sparse_cuda_MPI_1 | Failed | 16.49 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | KokkosKernels_sparse_cuda_MPI_1 | Failed | 2.43 | Completed (Failed) |
| white | Trilinos-atdm-white-ride-cuda-debug | KokkosKernels_sparse_cuda_MPI_1 | Failed | 2.39 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-cuda-debug | PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4 | Failed | 21.95 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4 | Failed | 24.31 | Completed (Failed) |
| white | Trilinos-atdm-white-ride-cuda-debug | PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4 | Failed | 24.9 | Completed (Failed) |
</details>
*
Except for the failing Anasazi tests in the build `Trilinos-atdm-white-ride-cuda-debug` (which I will write another GitHub issue for), all of these tests (even the timeouts) seem to be failing due to debug-mode checks from `KOKKOS_ENABLE_DEBUG=ON` being set (see #2439) failing and throwing exceptions. In the case of the failing tests `Amesos2_KLU2_UnitTests_MPI_2`, for example, it shows:
```
5. KLU2_double_int_int_NonContgGID_UnitTest ...
p=0: *** Caught standard std::exception of type 'std::runtime_error' :
View bounds error of view MV::DualView ( -1 < 6 , 0 < 1 )
Traceback functionality not available
[FAILED] (0.0885 sec) KLU2_double_int_int_NonContgGID_UnitTest
Location: /home/rabartl/WHITE/ATDM_Driver/Trilinos-atdm-white-ride-cuda-debug/SRC_AND_BUILD/Trilinos/packages/amesos2/test/solvers/KLU2_UnitTests.cpp:383
```
This exception causes a hang and a timeout in some cases and fails quickly and aborts in other cases. (So much for assuming that one MPI process throwing an excpetion will bring down an MPI job in all cases.)
Many of these builds have been promoted to the "ATDM" CDash group/track and therefore triggered CDash error emails today. Therefore, this must get fixed quickly if possible (or we will need to demote these builds again).
## Steps to Reproduce
One can log onto `white` (SON) or `ride` (SRN) and then reproduce the build and tests as described at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
I just reproduced many of these failures on 'white' using
```
$ ssh white
$ cd ~/rilinos.base/BUILD/WHITE/CHECKIN/
$ bsub -x -I -q rhel7F -n 16 \
./checkin-test-atdm.sh cuda-debug --enable-packages=Kokkos,KokkosKernels,Amesos2,Panzer --local-do-all
...
FAILED (NOT READY TO PUSH): Trilinos: white22
Wed Mar 28 11:03:05 MDT 2018
Enabled Packages: Kokkos, KokkosKernels, Amesos2, Panzer
Build test results:
-------------------
0) MPI_RELEASE_DEBUG_SHARED_PT => Test case MPI_RELEASE_DEBUG_SHARED_PT was not run! => Does not affect push readiness! (-1.00 min)
1) cuda-debug => FAILED: passed=189,notpassed=5 => Not ready to push! (120.13 min)
REQUESTED ACTIONS: FAILED
```
This showed the test results:
```
97% tests passed, 5 tests failed out of 194
Subproject Time Summary:
Amesos2 = 1232.97 sec*proc (8 tests)
Kokkos = 954.94 sec*proc (26 tests)
KokkosKernels = 870.46 sec*proc (8 tests)
Panzer = 7490.79 sec*proc (152 tests)
Total Test time (real) = 1518.22 sec
The following tests FAILED:
2 - KokkosCore_UnitTest_Cuda_MPI_1 (Failed)
28 - KokkosKernels_sparse_cuda_MPI_1 (Failed)
35 - Amesos2_KLU2_UnitTests_MPI_2 (Timeout)
174 - PanzerAdaptersSTK_PoissonInterfaceExample_2d_diffsideids_MPI_1 (Timeout)
192 - PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4 (Failed)
Errors while running CTest
Total time for cuda-debug = 120.13 min
```
The test failure timeout `PanzerAdaptersSTK_PoissonInterfaceExample_2d_diffsideids_MPI_1` was also seen in #2446 as well. Not sure why that test timed out when run locally but not in the driver jobs. But otherwise, this one build reproduced all of the failing tests shown on CDash except for the test `Anasazi_Epetra_ModalSolversTester_MPI_4` (which does not look to be related to `KOKKOS_ENABLE_DEBUG=ON`).
## Related Issues
* Related to: #2439, #2464
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2827New Kokkos, KokkosKernels, and Panzer test failures on CUDA 8.0 and CUDA 9.0 ...2018-11-30T03:09:20ZJames WillenbringNew Kokkos, KokkosKernels, and Panzer test failures on CUDA 8.0 and CUDA 9.0 builds after Kokkos and KokkosKernels update*Created by: bartlettroscoe*
CC: @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/panzer, @ndellingwood
## Next Action Status
Kokkos, KokkosKernels, and Panzer failing and timing-out tests have been fixed by PRs #2863, #2874,...*Created by: bartlettroscoe*
CC: @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/panzer, @ndellingwood
## Next Action Status
Kokkos, KokkosKernels, and Panzer failing and timing-out tests have been fixed by PRs #2863, #2874, #2927, and #2964 . No Panzer, Kokkos or KokkosKernels failures observed 6/19 or 6/20/2018.
## Description
The Kokkos and KokkosKernels updates in the recent commits 51cb7c5 and 816e703:
```
51cb7c5: Merge branch 'develop' into kokkos-promotion
Author: ndellingwood <ndellin@sandia.gov>
Date: Thu May 24 23:55:26 2018 -0600
816e703: Snapshot of kokkos-kernels.git from commit 1a7b524ba38fdfab6c1058065af06cbcb4a2ce6f
Author: Nathan Ellingwood <ndellin@sandia.gov>
Date: Thu May 24 23:30:27 2018 -0600
```
seem to have triggered several new test failures and timeouts in the packages in Kokkos, KokkosKernels, and Panzer as shown in:
* https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&date=2018-05-26&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=Trilinos-atdm-&field2=status&compare2=62&value2=passed&field3=status&compare3=62&value3=notrun&field4=buildname&compare4=63&value4=cuda&field5=buildname&compare5=62&value5=Trilinos-atdm-white-ride-cuda-debug-pt-all-at-once
The new failing and timing-out tests are:
| Test | Status | Details |
| :-- | :-- | :-- |
| KokkosContainers_UnitTest_Serial_MPI_1 | Failed | Completed (Timeout) |
| KokkosCore_UnitTest_Cuda_MPI_1 | Failed | Completed (Failed) |
| KokkosKernels_sparse_serial_MPI_1 | Failed | Completed (Timeout) |
| PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-2 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_PoissonExample-ConvTest-Quad-Order-3 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_PoissonExample-ConvTest-Quad-Order-4 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_PoissonExample-ConvTest-Tri-Order-3 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_PoissonExample-ConvTest-Tri-Order-4 | Failed | Completed (Failed) |
which failed in one or more of the unique builds:
* Trilinos-atdm-hansen-shiller-cuda-8.0-debug
* Trilinos-atdm-hansen-shiller-cuda-8.0-opt
* Trilinos-atdm-white-ride-cuda-debug
* Trilinos-atdm-white-ride-cuda-opt
These are all basically CUDA 8.0 builds.
These commits were shown pulled in this testing day at:
* https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3547898#!#note0
## Steps to Reproduce
The most failures are produced on the `Trilinos-atdm-white-ride-cuda-debug` build on 'white' and 'ride' so that is likely the bet bet to use to reproduce these failures. Therefore, as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
after logging into 'white' or 'ride' and cloning the Trilinos Git repo (pointed to by `TRILINOS_DIR`) and getting on the 'develop' branch, one would do:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Kokkos=ON \
-DTrilinos_ENABLE_KokkosKernels=ON \
-DTrilinos_ENABLE_Panzer=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2221Pamgen,kokkos-kernels: NVCC build error with -D CMAKE_CXX_USE_RESPONSE_FILE_F...2019-03-05T22:14:10ZJames WillenbringPamgen,kokkos-kernels: NVCC build error with -D CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON*Created by: mhoemmen*
kokkos-kernels requires `-D CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON` for CUDA builds, even with complex arithmetic disabled.
https://github.com/trilinos/Trilinos/issues/2115#issuecomment-357750048
However...*Created by: mhoemmen*
kokkos-kernels requires `-D CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON` for CUDA builds, even with complex arithmetic disabled.
https://github.com/trilinos/Trilinos/issues/2115#issuecomment-357750048
However, Pamgen does not build, given those settings.
@trilinos/kokkos-kernels @trilinos/pamgen @trilinos/framework
CC: @rrdrake @prwolfe
## Expectations
1. Whatever flags kokkos-kernels needs to build on CUDA, they need not to block building other essential packages.
2. kokkos-kernels needs to document its CMake requirements.
3. kokkos-kernels needs to fail at configure time, with an informative message, if the CMake variables that it needs are not set.
## Current Behavior
```
$ make
[ 0%] Linking CXX shared library libpamgen.so
nvcc fatal : No input files specified; use option --help for more information
make[2]: *** [packages/pamgen/src/libpamgen.so.12.13] Error 1
make[1]: *** [packages/pamgen/src/CMakeFiles/pamgen.dir/all] Error 2
make: *** [all] Error 2
```
## Motivation and Context
Many downstream tests, including Belos and Ifpack2, depend on Pamgen. This blocks adequate Trilinos testing on CUDA.
## Steps to Reproduce
```
$ module list
Currently Loaded Modulefiles:
1) sems-env 3) sems-cmake/3.3.2 5) kokkos-cuda/8.0.44
2) kokkos-env 4) sems-gcc/5.3.0 6) kokkos-openmpi/2.0.1/cuda
```
CMake configuration:
```
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-D BUILD_SHARED_LIBS:BOOL=ON
-D Trilinos_ENABLE_OpenMP:BOOL=ON
-D Kokkos_ENABLE_OpenMP:BOOL=ON
-D Tpetra_INST_OPENMP:BOOL=ON
-D Trilinos_SHOW_DEPRECATED_WARNINGS:BOOL=ON
-D Trilinos_ENABLE_Fortran:BOOL=OFF
-D TPL_ENABLE_CUDA:BOOL=ON
-D KOKKOS_ARCH="SNB;Kepler35"
-D Kokkos_ENABLE_Cuda:BOOL=ON
-D Kokkos_ENABLE_Cuda_UVM:BOOL=ON
-D Tpetra_INST_CUDA:BOOL=ON
-D Kokkos_ENABLE_Cuda_Lambda:BOOL=ON
-D CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS:BOOL=ON
-D CMAKE_CXX_FLAGS:STRING="-Wall"
-D Trilinos_CXX11_FLAGS:STRING="-std=c++11 --expt-extended-lambda"
-D TPL_ENABLE_MKL:BOOL=OFF
-D TPL_ENABLE_Matio:BOOL=OFF
-D TPL_ENABLE_SuperLU:BOOL=OFF
-D TPL_ENABLE_Zlib:BOOL=OFF
-D TPL_ENABLE_Netcdf:BOOL=OFF
-D TPL_ENABLE_HDF5:BOOL=OFF
-D TPL_ENABLE_ParMETIS:BOOL=OFF
-D TPL_ENABLE_Boost:BOOL=OFF
-D TPL_ENABLE_BoostLib:BOOL=OFF
-D TPL_ENABLE_yaml-cpp:BOOL=OFF
-D TPL_ENABLE_MPI:BOOL=ON
```
## Your Environment
- develop branch, commit 400765e21e17dfb995e0f4a2759ce9c5f961b685 (likely not related)
## Related Issues
* Blocks #2115