Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2016-11-04T05:30:13Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/700KokkosKernels: Add simplified kernels for integer Scalar types2016-11-04T05:30:13ZJames WillenbringKokkosKernels: Add simplified kernels for integer Scalar types*Created by: mhoemmen*
@trilinos/zoltan2 @trilinos/tpetra
Key words: build size, build time
Many Trilinos users have complained about long build times and large library and executable sizes. One of the biggest sources of this in...*Created by: mhoemmen*
@trilinos/zoltan2 @trilinos/tpetra
Key words: build size, build time
Many Trilinos users have complained about long build times and large library and executable sizes. One of the biggest sources of this in the Tpetra solver stack, is the large number of template parameter combinations for which Tpetra classes get instantiated. For example, we build all of Tpetra for Scalar = int and Scalar = GlobalOrdinal for EVERY enabled GlobalOrdinal type, as well as for the usual Scalar types like double and std::complex<double>.
Use of integer Scalar types seems a little weird. In most cases where Tpetra or downstream Trilinos packages use integer Scalar types, they use them for communication (as the source or target of an Export or Import), not for computation. This could justify refactoring Tpetra's class hierarchy into integer and non-integer "branches." However, I had a conversation with Michael Wolf about Zoltan2's needs. He explained that for some computations of metrics, Zoltan2 does sparse matrix-vector multiplies with integer Scalar types. This means that we really do need to compute with integer Scalar types. However, we don't need highly optimized kernels for integer Scalar types, as far as I know.
This suggests that we could address the problem at the KokkosKernels level, by falling back to simple kernels for integer Scalar types. This issue proposes to do just that. The kernels still need to be thread parallel, and must use CUDA appropriately. However, they don't need such heavy optimization. We can write simple one-level parallelism, for example.
Here are some build directory size statistics, for the Trilinos/packages/tpetra build directory after `make clean` and `make`, with no examples or tests enabled. I used GCC 4.7.2 on Linux, and enabled Scalar = std::complex<double>. Otherwise, I only use default settings for enabled types. (Default enabled LocalOrdinal type is int. Default enabled GlobalOrdinal types are int and long long.) __STATIC builds indicate static libraries; otherwise, I use dynamic libraries. *_DEBUG_ builds have Kokkos and Teuchos debugging features (e.g., bounds checking) turned on; __RELEASE_ builds have these debugging features turned off. I enabled only the Kokkos::OpenMP version of Tpetra (this should generate more code than the Kokkos::Serial version).
- MPI_DEBUG: 2.1 G
- MPI_DEBUG_STATIC: 11 G
- MPI_RELEASE: 187 M
- MPI_RELEASE_STATIC: 2.3 G
Do you see why we recommend dynamic libraries? ;-)
Correction: My MPI_RELEASE build is Kokkos::Serial only.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/908Small dense matrix inversion2016-12-14T20:33:53ZJames WillenbringSmall dense matrix inversion*Created by: bathmatt*
@mhoemmen I can see this being somewhere for your block crs matrix inversion.
Is there a class where I can do a small dense matrix inversion? Example, I can grab a sub-view out of a kokkos 2D view and call it ...*Created by: bathmatt*
@mhoemmen I can see this being somewhere for your block crs matrix inversion.
Is there a class where I can do a small dense matrix inversion? Example, I can grab a sub-view out of a kokkos 2D view and call it a matrix and compute its inverse?
Thanks
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2390Ifpack2 - SIMD/Vectorization Intrinsics Error with Type Conversion on HSW/Kep...2018-05-31T20:55:53ZJames WillenbringIfpack2 - SIMD/Vectorization Intrinsics Error with Type Conversion on HSW/Kepler37 Optimized Build*Created by: nmhamster*
@trilinos/ifpack2 :
Running an optimized build for HSW/Kepler37 with GCC 4.9.3 and CUDA 9.0. The following error is created. If this code is using STK::SIMD or SIMD Types, its possible there is a invalid type ...*Created by: nmhamster*
@trilinos/ifpack2 :
Running an optimized build for HSW/Kepler37 with GCC 4.9.3 and CUDA 9.0. The following error is created. If this code is using STK::SIMD or SIMD Types, its possible there is a invalid type conversion present in this library. This particular file generates all kinds of warnings.
Compile line is (lots of stuff) and:
```
-pedantic -Wall -Wno-long-long -Wwrite-strings -fPIC -lineinfo --std=c++11 -expt-extended-lambda -march=core-avx2 -mtune=core-avx2 -arch=sm_37 -g -O0 -o CMakeFiles/ifpack2.dir/Ifpack2_BlockTriDiContainer_Serial.cpp.o -c /home/sdhammo/git/trilinos-sparc-gitlab-repo/build-sparc-493-90176-arch/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_Serial.cpp
```
Produces the following error when using `KOKKOS_ARCH=HSW;Kepler37`
```
...
/home/sdhammo/git/trilinos-sparc-gitlab-repo/packages/kokkos-kernels/src/batched/KokkosBatched_InnerTrsm_Serial_Impl.hpp(1367): warning: calling a __host__ function from a __host__ __device__ function is not allowed
/home/sdhammo/git/trilinos-sparc-gitlab-repo/packages/kokkos-kernels/src/batched/KokkosBatched_InnerTrsm_Serial_Impl.hpp(1380): warning: calling a __host__ function from a __host__ __device__ function is not allowed
/home/projects/x86-64/gcc/4.9.3/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/include/avxintrin.h(54): error: Internal Compiler Error (codegen): "unsupported type during translation!"
make[2]: *** [packages/ifpack2/src/CMakeFiles/ifpack2.dir/build.make:927: packages/ifpack2/src/CMakeFiles/ifpack2.dir/Ifpack2_BlockTriDiContainer_Serial.cpp.o] Error 2
make[1]: *** [CMakeFiles/Makefile2:11785: packages/ifpack2/src/CMakeFiles/ifpack2.dir/all] Error 2
make: *** [Makefile:161: all] Error 2
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/288Ifpack2: Build and test new threaded coloring Gauss-Seidel by default2017-01-06T20:29:55ZJames WillenbringIfpack2: Build and test new threaded coloring Gauss-Seidel by default*Created by: mhoemmen*
@trilinos/ifpack2 @kddevin @jdbooth
Right now, enabling Ifpack2's new threaded coloring Gauss-Seidel requires enabling "experimental" CMake options (both off by default) in both KokkosKernels and Ifpack2. It wo...*Created by: mhoemmen*
@trilinos/ifpack2 @kddevin @jdbooth
Right now, enabling Ifpack2's new threaded coloring Gauss-Seidel requires enabling "experimental" CMake options (both off by default) in both KokkosKernels and Ifpack2. It would make sense to build and make this new capability available by default, even if it is not the default Gauss-Seidel implementation.
Tpetra-FY17-Q2https://gitlab.osti.gov/jmwille/Trilinos/-/issues/178Tpetra BCRS: Thread-parallelize sparse matrix-vector multiply2016-06-02T15:57:50ZJames WillenbringTpetra BCRS: Thread-parallelize sparse matrix-vector multiply*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2 @crtrott @kyungjoo-kim @amklinv
Thread-parallelize the sparse matrix-vector multiply in the apply() method of Tpetra::Experimental::BlockCrsMatrix. Please interact with Ryan E...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2 @crtrott @kyungjoo-kim @amklinv
Thread-parallelize the sparse matrix-vector multiply in the apply() method of Tpetra::Experimental::BlockCrsMatrix. Please interact with Ryan Eberhardt, who has an excellent CUDA implementation for column-major blocks.
It would be wise to do this in two passes. First, add a simple host execution space parallelization using a lambda. Then, implement an optimized kernel, using Ryan's as a start.
This affects Ifpack2 as well as Tpetra, because for Jacobi with > 1 sweep, Ifpack2 uses sparse mat-vec.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/208build of Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o fails if you use nvcc and ha...2016-03-19T08:19:49ZJames Willenbringbuild of Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o fails if you use nvcc and have cuda disabled*Created by: bathmatt*
If I don't configure with cuda but still have OMPI_CXX=nvcc_wrapper I get the following. This is on hansen
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV...*Created by: bathmatt*
If I don't configure with cuda but still have OMPI_CXX=nvcc_wrapper I get the following. This is on hansen
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o
/home/mbetten/Trilinos/Trilinos/packages/tpetra/kernels/src/impl/Kokkos_Sparse_impl_spmv.hpp(885): error: namespace "Kokkos" has no member "shfl_down"
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/226cuda build of Kokkos_Sparse_MV_impl_spmv takes 25 minutes for 1 file.2016-05-18T19:44:14ZJames Willenbringcuda build of Kokkos_Sparse_MV_impl_spmv takes 25 minutes for 1 file.*Created by: bathmatt*
Can we split up the build of this file? This is all on hansen with
-bash-4.1$ module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18
The build of
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeF...*Created by: bathmatt*
Can we split up the build of this file? This is all on hansen with
-bash-4.1$ module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18
The build of
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Cuda.cpp.o
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o
is very slow under cuda/debug (not -G though). By very slow, 25 minutes.
-bash-4.1$ time make -j
[ 0%] Built target kokkoscore
[ 0%] Built target kokkosalgorithms
[ 0%] Built target kokkoscontainers
[ 33%] Built target teuchoscore
[ 66%] Built target teuchosparameterlist
[ 66%] Built target teuchoscomm
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Cuda.cpp.o
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o
[ 66%] Linking CXX static library libtpetrakernels.a
[100%] Built target tpetrakernels
real 24m16.402s
user 29m35.614s
sys 0m58.005s
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/340KokkosKernels: Fix ETI2017-07-31T23:45:39ZJames WillenbringKokkosKernels: Fix ETI*Created by: mhoemmen*
@trilinos/tpetra
We need to fix ETI for KokkosKernels, in order to keep build and link times down for Tpetra and downstream packages. This is especially important for link-time optimization (LTO).
A fix would b...*Created by: mhoemmen*
@trilinos/tpetra
We need to fix ETI for KokkosKernels, in order to keep build and link times down for Tpetra and downstream packages. This is especially important for link-time optimization (LTO).
A fix would be to use the same .cpp file generation approach used in TpetraCore and Ifpack2.
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/416KokkosKernels: Optimize BCRS mat-vec for GPUs and CPU SIMD2017-10-26T20:00:31ZJames WillenbringKokkosKernels: Optimize BCRS mat-vec for GPUs and CPU SIMD*Created by: mhoemmen*
@trilinos/tpetra
Our fix for #178 includes only a one-level thread parallelization of BlockCrsMatrix (BCRS) matrix-vector multiply. It runs on all supported platforms (including GPUs), but needs further optimiza...*Created by: mhoemmen*
@trilinos/tpetra
Our fix for #178 includes only a one-level thread parallelization of BlockCrsMatrix (BCRS) matrix-vector multiply. It runs on all supported platforms (including GPUs), but needs further optimization, both for GPUs and for non-GPU vectorization. At least, we need to plug in vendor libraries, if they are well optimized (esp. for block sizes that are not powers of two).
What does "well optimized" mean? The lower bar should be "matches performance of mat-vec with the same sparse matrix, stored in (non-block) CRS format." That is, we compare against Christian's hand-optimized (non-block) CRS kernel, by taking the block matrix and storing it in non-block format. When performance matches, we have achieved the lower bar.
This is a lower bar because it inflates total per-MPI-process memory usage of the matrix by `M_crs / M_bcrs`, where
`M_crs = Z * (sizeof(LO)*b^2 + sizeof(Scalar)*b^2) + (m-1)*sizeof(offset_type)`
and
`M_bcrs = Z * (sizeof(LO) + sizeof(Scalar)*b^2) + (m-1)*sizeof(offset_type)`.
In these formulae, `Z` is the number of matrix entries on that process (counting each entry in a block as separate), `m` is the number of rows on that process, and `LO` ("local ordinal") is the type of each column index. Matrix-vector multiply must read all the matrix's data once, so this is a bandwidth-based lower bound (for a kernel that should be bandwidth bound when the problem does not fit in cache).
This is a _reasonable_ lower bar because, for typical `Scalar = double` and `LO = int`, `M_crs / M_bcrs < 2`. This also suggests a stopping criterion for optimization. It may be possible for BCRS mat-vec to get a higher fraction of memory bandwidth peak than CRS mat-vec, because it involves more contiguous, direct memory loads. Thus, I am open to suggestions for a better stopping criterion.
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/447KokkosKernels: CrsMatrix sumIntoValuesSorted minor questions2016-06-19T20:49:20ZJames WillenbringKokkosKernels: CrsMatrix sumIntoValuesSorted minor questions*Created by: mhoemmen*
@bathmatt 's commit https://github.com/trilinos/Trilinos/commit/1e65cffac05ae95bbc4e6f0d6d4d428886704834 added a sumIntoValuesSorted method to KokkosSparse::CrsMatrix. I have a few comments and questions:
1. We s...*Created by: mhoemmen*
@bathmatt 's commit https://github.com/trilinos/Trilinos/commit/1e65cffac05ae95bbc4e6f0d6d4d428886704834 added a sumIntoValuesSorted method to KokkosSparse::CrsMatrix. I have a few comments and questions:
1. We should introduce the "hint" that Epetra and Tpetra use for optimizing search for multiple column indices. It introduces an extra branch per input index, but avoids search for common cases. @etphipp first implemented it in Tpetra and found it to be useful, and `findRelOffset` (in tpetra/core/src/Tpetra_Util.hpp) does it too.
2. It's legit to use `ordinal_type` (32-bit) instead of `size_type` (64-bit on everything but CUDA) for the difference between two consecutive row offsets, as long as the row doesn't have too many duplicate entries. SparseRowView(Const) already uses `ordinal_type` for the row length, for this reason.
3. Was there a particular reason for the `hi - low > 10` cut-off, or is that just a good guess?
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/583Ifpack2: Plug in HTS for thread-parallel sparse triangular solve 2017-09-28T01:43:23ZJames WillenbringIfpack2: Plug in HTS for thread-parallel sparse triangular solve *Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/tpetra
*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/tpetra
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/568Tpetra: proposed changes to handle processes with zero rows2016-09-14T17:19:30ZJames WillenbringTpetra: proposed changes to handle processes with zero rows*Created by: allaffa*
@allaffa @jhux2 @trilinos/tpetra @rstumin
I am currently working on a project at Sandia about multigrid preconditioners where some tasks have no matrix rows. This has revealed two places where KokkosKernels and T...*Created by: allaffa*
@allaffa @jhux2 @trilinos/tpetra @rstumin
I am currently working on a project at Sandia about multigrid preconditioners where some tasks have no matrix rows. This has revealed two places where KokkosKernels and Tpetra throw an exception.
I would like to propose a change of the conditions to throw the exception, checking also that the number of rows be different from zero.
See attached .txt files.
[0005-Tpetra-added-check-on-rows-in-exception.txt](https://github.com/trilinos/Trilinos/files/425981/0005-Tpetra-added-check-on-rows-in-exception.txt)
[0007-Kokkos-added-check-on-rows-in-exception.txt](https://github.com/trilinos/Trilinos/files/425980/0007-Kokkos-added-check-on-rows-in-exception.txt)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/662KokkosKernels: Add segmented sort / sort-and-merge2016-10-26T04:10:08ZJames WillenbringKokkosKernels: Add segmented sort / sort-and-merge*Created by: mhoemmen*
See #660 for a use case. Tpetra::Crs{Graph,Matrix}::fillComplete currently needs segmented sort-and-merge, though a fix for #119 would remove the "-and-merge" requirement.
Thrust doesn't have anything like this....*Created by: mhoemmen*
See #660 for a use case. Tpetra::Crs{Graph,Matrix}::fillComplete currently needs segmented sort-and-merge, though a fix for #119 would remove the "-and-merge" requirement.
Thrust doesn't have anything like this. stable_sort_by_key() just does what Tpetra::sort2 currently does, namely apply the implicit permutation resulting from sorting keys, to a corresponding array of values.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/666KokkosKernels: Fix sparse matrix-matrix multiply error handling2016-11-02T17:34:54ZJames WillenbringKokkosKernels: Fix sparse matrix-matrix multiply error handling*Created by: mhoemmen*
@trilinos/tpetra
For example, if the user requests a TPL (such as CUSP), but that TPL is not installed, the implementation should throw an exception or return an error code, not just print some error message to ...*Created by: mhoemmen*
@trilinos/tpetra
For example, if the user requests a TPL (such as CUSP), but that TPL is not installed, the implementation should throw an exception or return an error code, not just print some error message to stderr. I see a similar approach to error handling when the memory space is wrong (e.g., lines 67-80 of KokkosKernels_SPGEMM_mkl_impl.hpp).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/667KokkosKernels: Use existing macros for TPLs like MKL2017-09-01T19:25:43ZJames WillenbringKokkosKernels: Use existing macros for TPLs like MKL*Created by: mhoemmen*
@trilinos/tpetra
I'm looking in tpetra/kernels/src/stage/graph/impl/KokkosKernels_SPGEMM_mkl_impl.hpp. I notice a macro KERNELS_HAVE_MKL. This is some macro that gets defined somewhere by hand in the code. It...*Created by: mhoemmen*
@trilinos/tpetra
I'm looking in tpetra/kernels/src/stage/graph/impl/KokkosKernels_SPGEMM_mkl_impl.hpp. I notice a macro KERNELS_HAVE_MKL. This is some macro that gets defined somewhere by hand in the code. It is not plugged into the CMake build system at all. KokkosKernels already has a macro, HAVE_TPETRAKERNELS_MKL. Please use that instead. "KERNELS_HAVE_MKL" is too general of a name; it is likely to collide with other software library's macro.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/691KokkosKernels: One kernel instantiation for both CudaSpace and CudaUVMSpace2016-10-05T19:11:59ZJames WillenbringKokkosKernels: One kernel instantiation for both CudaSpace and CudaUVMSpace*Created by: mhoemmen*
This depends on https://github.com/kokkos/kokkos/issues/290. (See also https://github.com/kokkos/kokkos/issues/290 , marked as redundant.)
Once the above Kokkos issue is fixed, we'll be able to assign from a Cud...*Created by: mhoemmen*
This depends on https://github.com/kokkos/kokkos/issues/290. (See also https://github.com/kokkos/kokkos/issues/290 , marked as redundant.)
Once the above Kokkos issue is fixed, we'll be able to assign from a CudaUVMSpace View to a CudaSpace View. This will let us just instantiate kernels for CudaSpace.
@trilinos/tpetra
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/690AllowPadding and WithoutInitializing moving to Kokkos namespace change downst...2017-10-27T04:10:00ZJames WillenbringAllowPadding and WithoutInitializing moving to Kokkos namespace change downstream Trilinos code accordingly*Created by: mhoemmen*
Kokkos/develop has moved AllowPadding and WithoutInitializing out of the Kokkos::Experimental namespace, into the Kokkos namespace. Once this gets moved into Kokkos/master and snapshotted into Trilinos, change Tr...*Created by: mhoemmen*
Kokkos/develop has moved AllowPadding and WithoutInitializing out of the Kokkos::Experimental namespace, into the Kokkos namespace. Once this gets moved into Kokkos/master and snapshotted into Trilinos, change Trilinos downstream code accordingly.
https://github.com/kokkos/kokkos/issues/325
@trilinos/tpetra
@trilinos/stokhos
@trilinos/sacado
@trilinos/shylu
@trilinos/stk
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/701Why does TriBITS' ETI add Nodes to list of manglings / typedefs, even when I ...2016-10-12T17:32:39ZJames WillenbringWhy does TriBITS' ETI add Nodes to list of manglings / typedefs, even when I don't want them there?*Created by: mhoemmen*
I'm using the ETI system for KokkosKernels. I don't want the manglings and typedefs to include Node types, because those don't exist in KokkosKernels. I just want to strip those out. How do I do that? I even c...*Created by: mhoemmen*
I'm using the ETI system for KokkosKernels. I don't want the manglings and typedefs to include Node types, because those don't exist in KokkosKernels. I just want to strip those out. How do I do that? I even cleared out list_of_manglings and eti_typedefs at the top of tpetra/kernels/cmake/ExplicitInstantiationSupport.cmake, and the Node types came back!!!
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/759KokkosKernels unit tests' dependency to kokkos gtest build2017-04-13T18:45:04ZJames WillenbringKokkosKernels unit tests' dependency to kokkos gtest build*Created by: mndevec*
Kokkos is using gtest for the unit tests. I think we talked briefly that we will use the same unit test harness for KokkosKernels, rather than introducing a dependency on Teuchos.
I am trying to add unit tests ...*Created by: mndevec*
Kokkos is using gtest for the unit tests. I think we talked briefly that we will use the same unit test harness for KokkosKernels, rather than introducing a dependency on Teuchos.
I am trying to add unit tests that use gtest, assuming that kokkos builds gtest and I can access the prebuild kokkos_gtest library. However, it is only compiled when Kokkos test is enabled.
I don't know an elegant way to handle this, I was wondering how this should be handled.
@srajama1 @mhoemmen @crtrott @ambrad @kyungjoo-kim
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/786Ifpack2: Linking error, duplicate symbols, with TpetraKernels_ENABLE_Experime...2016-12-09T05:00:29ZJames WillenbringIfpack2: Linking error, duplicate symbols, with TpetraKernels_ENABLE_Experimental=ON*Created by: bathmatt*
Is anyone else getting this??
4062/5246] Linking CXX executable packages/stratimikos/test/Stratimikos_issue_535.exe
FAILED: packages/stratimikos/test/Stratimikos_issue_535.exe
: && /projects/sems/install...*Created by: bathmatt*
Is anyone else getting this??
4062/5246] Linking CXX executable packages/stratimikos/test/Stratimikos_issue_535.exe
FAILED: packages/stratimikos/test/Stratimikos_issue_535.exe
: && /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/bin/mpicxx -std=c++11 -fopenmp -g -O0 packages/stratimikos/test/CMakeFiles/Stratimikos_issue_535.dir/test_issue_535.cpp.o -o packages/stratimikos/test/Stratimikos_issue_535.exe -rdynamic packages/stratimikos/src/libstratimikos.a packages/stratimikos/adapters/belos/src/libstratimikosbelos.a packages/stratimikos/adapters/aztecoo/src/libstratimikosaztecoo.a packages/stratimikos/adapters/amesos/src/libstratimikosamesos.a packages/stratimikos/adapters/ml/src/libstratimikosml.a packages/stratimikos/adapters/ifpack/src/libstratimikosifpack.a packages/ifpack2/adapters/libifpack2-adapters.a packages/ifpack2/src/libifpack2.a packages/thyra/adapters/tpetra/src/libthyratpetra.a packages/triutils/src/libtriutils.a packages/ml/src/libml.a packages/zoltan/src/libzoltan.a -lm packages/ifpack/src/libifpack.a packages/amesos/src/libamesos.a packages/belos/tpetra/src/libbelostpetra.a packages/belos/epetra/src/libbelosepetra.a packages/belos/src/libbelos.a packages/thyra/adapters/tpetra/src/libthyratpetra.a packages/aztecoo/src/libaztecoo.a packages/thyra/adapters/epetraext/src/libthyraepetraext.a packages/thyra/adapters/epetra/src/libthyraepetra.a packages/epetraext/src/libepetraext.a packages/triutils/src/libtriutils.a packages/thyra/core/src/libthyracore.a packages/rtop/src/librtop.a packages/tpetra/core/ext/libtpetraext.a packages/tpetra/core/inout/libtpetrainout.a packages/tpetra/core/src/libtpetra.a packages/epetra/src/libepetra.a packages/tpetra/kernels/src/libtpetrakernels.a packages/kokkos/algorithms/src/libkokkosalgorithms.a packages/kokkos/containers/src/libkokkoscontainers.a packages/tpetra/classic/LinAlg/libtpetraclassiclinalg.a packages/tpetra/classic/NodeAPI/libtpetraclassicnodeapi.a packages/tpetra/classic/src/libtpetraclassic.a packages/teuchos/kokkoscomm/src/libteuchoskokkoscomm.a packages/teuchos/kokkoscompat/src/libteuchoskokkoscompat.a packages/teuchos/remainder/src/libteuchosremainder.a packages/teuchos/numerics/src/libteuchosnumerics.a /usr/lib64/liblapack.so.3 /usr/lib64/libblas.so.3 packages/teuchos/comm/src/libteuchoscomm.a packages/teuchos/parameterlist/src/libteuchosparameterlist.a packages/teuchos/core/src/libteuchoscore.a /projects/sems/install/rhel6-x86_64/sems/tpl/boost/1.59.0/gcc/5.3.0/base/lib/libboost_program_options.so /projects/sems/install/rhel6-x86_64/sems/tpl/boost/1.59.0/gcc/5.3.0/base/lib/libboost_system.so packages/kokkos/core/src/libkokkoscore.a /usr/lib64/libdl.so -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -lquadmath -Wl,-rpath,/projects/sems/install/rhel6-x86_64/sems/tpl/boost/1.59.0/gcc/5.3.0/base/lib && :
packages/ifpack2/src/libifpack2.a(Ifpack2_Relaxation_OpenMP.cpp.o): In function `KokkosKernels::Experimental::Util::endswith(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
/net/fs02eppic/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/include/c++/5.3.0/bits/char_traits.h:258: multiple definition of `KokkosKernels::Experimental::Util::endswith(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
packages/ifpack2/src/libifpack2.a(Ifpack2_Details_OneLevelFactory_OpenMP.cpp.o):/net/fs02eppic/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/5.3.0/base/include/c++/5.3.0/bits/char_traits.h:258: first defined here
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/922Tpetra: New thread-parallel sparse matrix-matrix multiply test fails ("experi...2017-01-25T17:51:39ZJames WillenbringTpetra: New thread-parallel sparse matrix-matrix multiply test fails ("experimental")*Created by: mhoemmen*
@trilinos/tpetra
http://testing.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=2651259*Created by: mhoemmen*
@trilinos/tpetra
http://testing.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=2651259Tpetra-FY17-Q2https://gitlab.osti.gov/jmwille/Trilinos/-/issues/948KokkosKernels: SPMV performance comparison test2017-01-06T00:12:10ZJames WillenbringKokkosKernels: SPMV performance comparison test*Created by: crtrott*
This adds a performance comparison test without MPI for just SPMV kernel. Can compare the KokkosKernel function, a custom Kokkos implementation, CuSparse and MKL.
This can read MatrixMarket format, and also gener...*Created by: crtrott*
This adds a performance comparison test without MPI for just SPMV kernel. Can compare the KokkosKernel function, a custom Kokkos implementation, CuSparse and MKL.
This can read MatrixMarket format, and also generate a binary matrix storage file, which it can read again. Depending on your system reading the binary format back in can be 10x faster than the text file.
I had that written quite a while ago I just cleaned it up now. https://gitlab.osti.gov/jmwille/Trilinos/-/issues/964Kokkos::StaticCrsGraph does not use memory space template parameter passed to...2017-01-10T20:54:12ZJames WillenbringKokkos::StaticCrsGraph does not use memory space template parameter passed to KokkosSparse::CrsMatrix*Created by: mndevec*
I am providing a device to CrsMatrix in the constructor. This is different than default, where device is either Kokkos::Cuda with hostpinned space, or Kokkos::OpenMP with Kokkos::Hostspace when HBM is enabled.
...*Created by: mndevec*
I am providing a device to CrsMatrix in the constructor. This is different than default, where device is either Kokkos::Cuda with hostpinned space, or Kokkos::OpenMP with Kokkos::Hostspace when HBM is enabled.
I was expecting CrsMatrix memory to be allocated at the memory space I provide. However, this holds only for the values view, while row pointers and entries are still allocated at the default memory space of the provided execution space.
It seems that StaticCrsGraph do not take the device as template argument, instead it is provided the execution space. It creates a default device, as a result allocated memories diverge for values and entries views.
Shouldn't StaticCrsGraph take the device as template argument instead of execution space?
@srajama1 @crtrott @mhoemmen
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/977KokkosKernels: Gauss-Seidel test build error2017-01-06T21:13:24ZJames WillenbringKokkosKernels: Gauss-Seidel test build error*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
KokkosKernels' Gauss-Seidel test has a build error:
```
.../Trilinos/packages/tpetra/kernels/unit_test/graph/KokkosKernels_Test_GaussSeidel.cpp: In instantiation of ‘scalar_v...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
KokkosKernels' Gauss-Seidel test has a build error:
```
.../Trilinos/packages/tpetra/kernels/unit_test/graph/KokkosKernels_Test_GaussSeidel.cpp: In instantiation of ‘scalar_view_t create_x_vector(size_t, typename scalar_view_t::value_type) [with scalar_view_t = Kokkos::View<Kokkos::complex<double>*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> >; size_t = long unsigned int; typename scalar_view_t::value_type = Kokkos::complex<double>]’:
.../Trilinos/packages/tpetra/kernels/unit_test/graph/KokkosKernels_Test_GaussSeidel.cpp:178:66: required from ‘void test_gauss_seidel(KokkosKernels::Experimental::Graph::GSAlgorithm) [with scalar_t = Kokkos::complex<double>; lno_t = int; device = Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>]’
.../Trilinos/packages/tpetra/kernels/unit_test/graph/KokkosKernels_Test_GaussSeidel.cpp:232:3: required from here
.../Trilinos/packages/tpetra/kernels/unit_test/graph/KokkosKernels_Test_GaussSeidel.cpp:140:68: error: no match for ‘operator/’ (operand types are ‘int’ and ‘Kokkos::ViewTraits<Kokkos::complex<double>*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> >::value_type {aka Kokkos::complex<double>}’)
static_cast <typename scalar_view_t::value_type> (RAND_MAX / max_value);
^
```
I think the code itself is fine; at least it builds and runs correctly in Ifpack2. This looks like an issue with the test not accounting for complex numbers.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/978KokkosKernels: Sparse matrix-matrix multiply test build error2017-04-13T18:42:58ZJames WillenbringKokkosKernels: Sparse matrix-matrix multiply test build error*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
KokkosKernels' sparse matrix-matrix multiply test has a build error:
```
.../Trilinos/packages/tpetra/kernels/src/stage/graph/utils/KokkosKernels_SimpleUtils.hpp:50:41: error:...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
KokkosKernels' sparse matrix-matrix multiply test has a build error:
```
.../Trilinos/packages/tpetra/kernels/src/stage/graph/utils/KokkosKernels_SimpleUtils.hpp:50:41: error: no match for ‘operator>’ (operand types are ‘Kokkos::ViewTraits<Kokkos::complex<double>*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> >::value_type {aka Kokkos::complex<double>}’ and ‘int’)
#define KOKKOSKERNELS_MACRO_ABS(x) ((x) > (0) ? (x): (-x))
^
.../Trilinos/packages/tpetra/kernels/src/stage/graph/utils/KokkosKernels_SimpleUtils.hpp:133:9: note: in expansion of macro ‘KOKKOSKERNELS_MACRO_ABS’
if (KOKKOSKERNELS_MACRO_ABS (val_diff) > eps) {
...
```
This may just be an issue with the test not accounting for complex numbers.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/979KokkosKernels: Graph coloring test fails2017-04-13T18:43:15ZJames WillenbringKokkosKernels: Graph coloring test fails*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
The KokkosKernels graph coloring test fails with the following run-time error:
```
terminate called after throwing an instance of 'std::runtime_error'
11: what(): Kokkos:...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
The KokkosKernels graph coloring test fails with the following run-time error:
```
terminate called after throwing an instance of 'std::runtime_error'
11: what(): Kokkos::OpenMP parallel_for ERROR: not initialized
11: Traceback functionality not available
```
This looks pretty easy to fix. Somewhere, something is not initializing Kokkos::OpenMP.
I am disabling the test for now to ensure that I can proceed with #288.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/980KokkosKernels: Sparse matrix-matrix multiply performance test build error2017-04-13T18:43:39ZJames WillenbringKokkosKernels: Sparse matrix-matrix multiply performance test build error*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/tpetra
Lots of errors like this:
```
.../Trilinos/packages/tpetra/kernels/perf_test/graph/KokkosKernels_SPMM_KK.cpp: In function ‘int main(int, char**)’:
.../Trilinos/packages/tpet...*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/tpetra
Lots of errors like this:
```
.../Trilinos/packages/tpetra/kernels/perf_test/graph/KokkosKernels_SPMM_KK.cpp: In function ‘int main(int, char**)’:
.../Trilinos/packages/tpetra/kernels/perf_test/graph/KokkosKernels_SPMM_KK.cpp:688:13: error: ‘Kokkos::OpenMP’ has not been declared
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/981Remove Tribits dependencies in KokkosKernels2017-01-06T03:27:05ZJames WillenbringRemove Tribits dependencies in KokkosKernels*Created by: srajama1*
There are bunch of tribits related dependencies that has come into KokkosKernels. Eventually we would like a Kokkos model where we could build without Tribits and test with gtest. This needs a cleanup.
@mndevec...*Created by: srajama1*
There are bunch of tribits related dependencies that has come into KokkosKernels. Eventually we would like a Kokkos model where we could build without Tribits and test with gtest. This needs a cleanup.
@mndevec @crtrott @mhoemmen
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/988KokkosKernels: Gauss-Seidel test fails with CUDA2017-04-13T18:44:14ZJames WillenbringKokkosKernels: Gauss-Seidel test fails with CUDA*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
The Gauss-Seidel test that I reenabled as part of #977 fails in a CUDA debug build for me, with CUDA 7.5 and GCC 4.8.4. It passes in a non-CUDA Clang 3.9 build with Kokkos::...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2
The Gauss-Seidel test that I reenabled as part of #977 fails in a CUDA debug build for me, with CUDA 7.5 and GCC 4.8.4. It passes in a non-CUDA Clang 3.9 build with Kokkos::Serial. The error messages look like this:
```
result_norm_res:3.05254e+36 initial_norm_res:160.873
3: .../Trilinos/packages/tpetra/kernels/unit_test/graph/KokkosKernels_Test_GaussSeidel.cpp:219: Failure
3: Value of: (result_norm_res < initial_norm_res)
3: Actual: false
3: Expected: true
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1084Graph coloring build warnings2017-02-22T17:26:25ZJames WillenbringGraph coloring build warnings*Created by: mhoemmen*
This is a duplicate of https://github.com/kokkos/kokkos-kernels/issues/3 (not yet externally visible, but will be soon) for the sake of external tracking.*Created by: mhoemmen*
This is a duplicate of https://github.com/kokkos/kokkos-kernels/issues/3 (not yet externally visible, but will be soon) for the sake of external tracking.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1099KokkosKernels pullout broke my configures2017-03-05T22:12:00ZJames WillenbringKokkosKernels pullout broke my configures*Created by: bathmatt*
Any idea on what I need to add to my scripts to get configure to work. It is a silent failure, by that I mean it isn't obvious what the problem is, and my CMakeErrors.log looks pretty much the same before and aft...*Created by: bathmatt*
Any idea on what I need to add to my scripts to get configure to work. It is a silent failure, by that I mean it isn't obvious what the problem is, and my CMakeErrors.log looks pretty much the same before and after the commit.
I added Trilinos_ENABLE_KokkosKernels=ON
Set up for creating a distribution ...
tail of the configure.
`
Finished configuring Trilinos!
-- Configuring incomplete, errors occurred!
See also "/jenkins/slave/workspace/EMPIRE-debug/build_drekar/CMakeFiles/CMakeOutput.log".
See also "/jenkins/slave/workspace/EMPIRE-debug/build_drekar/CMakeFiles/CMakeError.log".
Error in configuration of drekar, trying to continue`
Here is my configure script
[config.txt](https://github.com/trilinos/Trilinos/files/811733/config.txt)
and all the log files. log has been renamed to txt
[configure.txt](https://github.com/trilinos/Trilinos/files/811735/configure.txt)
[CMakeError.txt](https://github.com/trilinos/Trilinos/files/811737/CMakeError.txt)
[CMakeOutput.txt](https://github.com/trilinos/Trilinos/files/811738/CMakeOutput.txt)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1170KokkosKernels: warnings on CUDA2017-10-27T04:19:03ZJames WillenbringKokkosKernels: warnings on CUDA*Created by: rppawlo*
When compiling with Tpetra enabled for cuda, we get a ton of warnings from kokkos kernels. It seems to stem from the three lines below. Can someone take a look at this?
```
[ 69%] Built target locathyra
/ascldap...*Created by: rppawlo*
When compiling with Tpetra enabled for cuda, we get a ton of warnings from kokkos kernels. It seems to stem from the three lines below. Can someone take a look at this?
```
[ 69%] Built target locathyra
/ascldap/users/rppawlo/Trilinos/packages/kokkos-kernels/src/Kokkos_ArithTraits.hpp(1480): warning: pointless comparison of unsigned integer with zero
/ascldap/users/rppawlo/Trilinos/packages/kokkos-kernels/src/Kokkos_ArithTraits.hpp(157): warning: pointless comparison of unsigned integer with zero
detected during instantiation of "IntType <unnamed>::intPowSigned(IntType, IntType) [with IntType=char]"
(1506): here
/ascldap/users/rppawlo/Trilinos/packages/kokkos-kernels/src/Kokkos_ArithTraits.hpp(161): warning: pointless comparison of unsigned integer with a negative constant
detected during instantiation of "IntType <unnamed>::intPowSigned(IntType, IntType) [with IntType=char]"
(1506): here
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1176Analysis of configure and build failures due to KokkosKernels pushes on March...2017-03-23T17:54:16ZJames WillenbringAnalysis of configure and build failures due to KokkosKernels pushes on March 1-2 reported in #1099*Created by: bartlettroscoe*
**Description:**
This story is to analyze the KokkosKernels commits pushed on March 1-2, 2017 that broke the configure and build of Trilinos that was reported in #1099 and see if usage of the [checkin-tes...*Created by: bartlettroscoe*
**Description:**
This story is to analyze the KokkosKernels commits pushed on March 1-2, 2017 that broke the configure and build of Trilinos that was reported in #1099 and see if usage of the [checkin-test-sems.sh](https://github.com/trilinos/Trilinos/wiki/Policies-%7C-Safe-Checkin-Testing) script could have avoided the failures (and resulting consequences) if it had been used for the test and push (which would have stopped the push).
This started when a set of commits were pushed to the Trilinos 'develop' branch on March 1 with the top commit 97ed757 being:
```
97ed757 [Wed Mar 1 08:24:01 2017 -0700] <crtrott@sandia.gov>
Kokkos-Kernels: Adding Kokkos-Kernels as a stand-alone package
```
as shown by the CI build:
* http://testing.sandia.gov/cdash/viewConfigure.php?buildid=2766164
That version of Trilinos failed to configure as shown on that CI build iteration.
Later that day, issue #1099 was created by an important Trilinos customer and it resulted in 35 comments that involved 9 people in that issue before it was resolved.
An attempt to fix this problem was pushed later that day with the top commit de7ac5a being:
```
de7ac5a [Wed Mar 1 13:17:16 2017 -0700] <mhoemme@sandia.gov>
KokkosKernels: Fix #1099
```
as shown by the CI build:
* http://testing.sandia.gov/cdash/viewConfigure.php?buildid=2766462
That version passed the configure but resulted in many build failures.
The build was not finally fixed until March 2 as shown at:
* http://testing.sandia.gov/cdash/viewConfigure.php?buildid=2767860
Could the usage of the checkin-test-sems.sh script have caught these problems and stop the pushes that broke the configure and build of Trilinos over these two days?
**CC:** @trilinos/framework, @bathmatt, @crtrott, @mhoemmen
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1180Consider taking advantage of new CUDA language support in CMake 3.8.02017-09-13T22:41:43ZJames WillenbringConsider taking advantage of new CUDA language support in CMake 3.8.0*Created by: bartlettroscoe*
**Description:**
I was looking over the release notes for CMake 3.8 at:
* https://cmake.org/cmake/help/v3.8/release/3.8.html
and it says that CUDA is now a first-class CMake language and it supports...*Created by: bartlettroscoe*
**Description:**
I was looking over the release notes for CMake 3.8 at:
* https://cmake.org/cmake/help/v3.8/release/3.8.html
and it says that CUDA is now a first-class CMake language and it supports the nvcc compiler.
Should this be something that we look into for Trilinos?
**CC:** @trilinos/framework, @trilinos/tpetra, @crtrott, @nmhamster
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1237KokkosKernels CI build failure on 4/12/20172017-04-12T16:26:22ZJames WillenbringKokkosKernels CI build failure on 4/12/2017*Created by: bartlettroscoe*
A KokkosKernels build failure was caught by the standard CI build:
* http://testing.sandia.gov/cdash/index.php?project=Trilinos&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=61&v...*Created by: bartlettroscoe*
A KokkosKernels build failure was caught by the standard CI build:
* http://testing.sandia.gov/cdash/index.php?project=Trilinos&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Linux-GCC-4.7.2-MPI_RELEASE_DEBUG_SHARED_PT_CI&field2=groupname&compare2=61&value2=Continuous&field3=buildstarttime&compare3=84&value3=now
in the first CI iteration this morning:
* http://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=2834087&filtercount=3&showfilters=1&field1=groupname&compare1=61&value1=Continuous&field2=buildstarttime&compare2=84&value2=now&filtercombine=and
(which suggests the checkin-test-sems.sh script should have caught this build failure before the push). But this was only a test/example failure which does not disable KokkosKernels in downstream packages. (Note that we will lose this behavior once we do the single configure, build, and tests https://github.com/TriBITSPub/TriBITS/issues/183 ).
But it looks like @crtrott reacted to this quickly fixed the KokkosKernels in the latest CI build iteration:
* http://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=2834513&filtercount=3&showfilters=1&field1=groupname&compare1=61&value1=Continuous&field2=buildstarttime&compare2=84&value2=now&filtercombine=and
Note that since this was only a test/example failure for KokkosKernels, this would not have stopped anyone else's pushes using the checkin-test-sems.sh script where were only changing a package downstream from KokkosKernels.
Just added this issue to document this failure since it was interesting for a few reasons:
* The build failure should have likely have been caused by the checkin-test-sems.sh script
* The failure was seen and fixed in the very next CI iteration
* This was only a build failure for examples/tests so this would not have blocked anyone else's push to changes in packages downstream from KokkosKernels
Since Trilinos CDash info is forgotten pretty quickly (just a month or so).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1423KokkosKernels: Problems with Kokkos_Sparse_CrsMatrix2017-07-03T19:41:58ZJames WillenbringKokkosKernels: Problems with Kokkos_Sparse_CrsMatrix*Created by: brian-kelley*
I'm trying to use the Kokkos::CrsMatrix to write a local threaded matrix addition. I've run into a couple of issues with it (one I've figured out, and one I haven't - maybe I should make 2 issues?):
-The do...*Created by: brian-kelley*
I'm trying to use the Kokkos::CrsMatrix to write a local threaded matrix addition. I've run into a couple of issues with it (one I've figured out, and one I haven't - maybe I should make 2 issues?):
-The documentation says that the [raw host array constructor](https://trilinos.org/docs/dev/packages/kokkos/doc/html/classKokkos_1_1CrsMatrix.html#a0c2a800a9ffc10f658038bb64ac47972) expects coordinates format (rows, cols, values) but it really expects CRS (rowptrs, colinds, values). I've changed my usage code to give CRS and it now works.
-I should be able to get a [Kokkos::SparseRowView](https://trilinos.org/docs/dev/packages/kokkos/doc/html/structKokkos_1_1SparseRowView.html) of row i by calling mat.row(i). This should let me directly get at the colinds and values in row i. But, every view I get back is empty (rowView.length == 0). I tried calling .row(i) for every row:
> C has 380473 nonzeros, 10000 rows, 10000 cols.
> Number of nz in each row: 0 0 0 0 0 0 0 0 0 0 (...and so on for 10,000 zeros)
C->nnz(), C->numRows() and C->numCols() all print what I expect (the first line of output), so there are entries in there. This looks like a simple bug, but is there anything special I need to do to get these row views?https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1401Experimental RBILUK 2017-09-21T16:05:25ZJames WillenbringExperimental RBILUK *Created by: mndevec*
This issue will track the development of new RBILUK.
Goals:
1- Clear separation of the symbolic, numeric and solve phases.
2- Replacing old blas calls such as GEMM, GETF2, with the register blocking little bl...*Created by: mndevec*
This issue will track the development of new RBILUK.
Goals:
1- Clear separation of the symbolic, numeric and solve phases.
2- Replacing old blas calls such as GEMM, GETF2, with the register blocking little block implementations in KokkosKernels.
The issue will be developed in the fork:
https://github.com/mndevec/Trilinos/tree/develop
@trilinos/ifpack2 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1622KokkosKernels: Gauss-Seidel threaded setup performance issues with Ifpack2 an...2017-09-12T19:15:52ZJames WillenbringKokkosKernels: Gauss-Seidel threaded setup performance issues with Ifpack2 and MueLu*Created by: pwxy*
I am trying to use the KokkosKernels threaded Gauss-Seidel.
I'm calling it through ifpack2 ("MT Gauss-Seidel") as a smoother for MueLu,
so the problem could be a bad interaction between KokkosKernels and ifpack2 or ...*Created by: pwxy*
I am trying to use the KokkosKernels threaded Gauss-Seidel.
I'm calling it through ifpack2 ("MT Gauss-Seidel") as a smoother for MueLu,
so the problem could be a bad interaction between KokkosKernels and ifpack2 or MueLu.
I'm running drekar on a single KNL of mutrino, with 1 MPI process, and I increase the OMP threads from 1 to 64 (1 OMP thread per core):
setup smoother (ifpack2 "MT Gauss-Seidel")
| t | solve time(s) |GS setup time(s)|
| -:| -:|-------:|
|1|33.27|493.10|
|2|24.67|286.50|
|4|12.26|157.80|
|8|6.97|79.82|
|16|3.97|36.61|
|32|3.50|24.06|
|64|3.16|16.01|
For reference, here are the times if I use the standard, non-threaded Gauss-Seidel
(but if it really is non-threaded, why is the setup time going down as the number of OMP threads is increased?)
setup smoother (ifpack2 "Gauss-Seidel")
|t|solve time(s)|GS setup time(s)|
| -:| -:|-------:|
|1|27.04|0.36|
|2|25.13|0.21|
|4|24.09|0.13|
|8|23.58|0.09|
|16|23.38|0.06|
|32|23.32|0.05|
|64|23.33|0.05|
drekar/Trilinos was built with intel 17.0.2 and gnu 6.1.0 (Trilinos repo as of August 16, 2017)
I ran vtune on ellis for the 1 OMP case (the 493.1s case above).
According to vtune, all the time is the two Kokkos::parallel_for calls in
KokkosKernels::Experimental::Util::symmetrize_graph_symbolic_hashmap (lines 1097 and 1139 of KokkosKernels_Utils.hpp)
the time is pretty much equally split between the two Kokkos::parallel_for calls
The following is the stack trace from Ifpack2:
```
Ifpack2::Relaxation::initialize()
KokkosKernels::Experimental::Graph::gauss_seidel_symbolic
KokkosKernels::Experimental::Graph::Impl::GaussSeidel
KokkosKernels::Experimental::Util::symmetrize_graph_symbolic_hashmap (Kokkos::parallel_for on line 1097 and 1139)
Kokkos::parallel_for
Kokkos::parallel_for
```
Edit (@aprokop): formattinghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/1729kokkos-kernels: Turn off unneeded instantiations by default2017-10-26T21:10:54ZJames Willenbringkokkos-kernels: Turn off unneeded instantiations by default*Created by: mhoemmen*
@trilinos/tpetra
Trilinos doesn't need the following instantiations:
- `CudaSpace` (not used, Trilinos assumes UVM and uses `CudaUVMSpace`)
- Unused offset types
See `${PACKAGE_NAME}_INST_MEMSPACE_C...*Created by: mhoemmen*
@trilinos/tpetra
Trilinos doesn't need the following instantiations:
- `CudaSpace` (not used, Trilinos assumes UVM and uses `CudaUVMSpace`)
- Unused offset types
See `${PACKAGE_NAME}_INST_MEMSPACE_CUDASPACE` in `kokkos-kernels/CMakeLists.txt`. I'm not sure if Tpetra can control these directly; it may be necessary to change defaults in kokkos-kernels.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1818kokkos-kernels vs kokkoskernels2017-11-01T15:34:48ZJames Willenbringkokkos-kernels vs kokkoskernels*Created by: nschloe*
There is a discrepancy in files names for the new Kokkos kernels:
* The package directory is called `kokkos-kernels`.
* The library file name is `kokkoskernels.so`.
This creates some problems on the Debi...*Created by: nschloe*
There is a discrepancy in files names for the new Kokkos kernels:
* The package directory is called `kokkos-kernels`.
* The library file name is `kokkoskernels.so`.
This creates some problems on the Debian side. I'd be great if this could be made consistent one way or the other. For comparison, check `trilinoscouplings`.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2057TpetraExt MMDetails / MM multiply compile error2017-12-06T20:40:33ZJames WillenbringTpetraExt MMDetails / MM multiply compile error*Created by: jjellio*
<!--- Provide a general summary of the issue in the Title above. -->
I've pulled develop, and when I try building on Cori with Intel 17, I am seeing the following error related to TpetraExt not finding the new mat...*Created by: jjellio*
<!--- Provide a general summary of the issue in the Title above. -->
I've pulled develop, and when I try building on Cori with Intel 17, I am seeing the following error related to TpetraExt not finding the new matrix matrix multiply routines.
## Compile error:
```
/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp(3366): error: class "Tpetra::MMdetails::KernelWrappers<Tpetra::Details::DefaultTypes::scalar_type={double}, Tpetra::Details::DefaultTypes::local_ordinal_type={int}, Tpetra::Details::DefaultTypes::global_ordinal_type={int}, Tpetra::Details::DefaultTypes::node_type>" has no member "jacobi_A_B_newmatrix_kernel_wrapper"
KernelWrappers<Scalar,LocalOrdinal,GlobalOrdinal,Node>::jacobi_A_B_newmatrix_kernel_wrapper(omega,Dinv,Aview,Bview,targetMapToOrigRow,targetMapToImportRow,Bcol2Ccol,Icol2Ccol,C,Cimport,label,params);
^
detected during:
instantiation of "void Tpetra::MMdetails::jacobi_A_B_newmatrix(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=double,
LocalOrdinal=Tpetra::Details::DefaultTypes::local_ordinal_type={int}, GlobalOrdinal=Tpetra::Details::DefaultTypes::global_ordinal_type={int}, Node=Tpetra::Details::DefaultTypes::node_type]" at line 386
instantiation of "void Tpetra::MatrixMatrix::Jacobi(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, bool, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=double,
LocalOrdinal=Tpetra::Details::DefaultTypes::local_ordinal_type={int}, GlobalOrdinal=Tpetra::Details::DefaultTypes::global_ordinal_type={int}, Node=Tpetra::Details::DefaultTypes::node_type]" at line 55 of "/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix.cpp"
/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp(3366): error: class "Tpetra::MMdetails::KernelWrappers<double, int, Tpetra::longlong={long long}, Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode>" has no member "jacobi_A_B_newmatrix_kernel_wrapper"
KernelWrappers<Scalar,LocalOrdinal,GlobalOrdinal,Node>::jacobi_A_B_newmatrix_kernel_wrapper(omega,Dinv,Aview,Bview,targetMapToOrigRow,targetMapToImportRow,Bcol2Ccol,Icol2Ccol,C,Cimport,label,params);
^
detected during:
instantiation of "void Tpetra::MMdetails::jacobi_A_B_newmatrix(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=double, LocalOrdinal=int,
GlobalOrdinal=Tpetra::longlong={long long}, Node=Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode]" at line 386
instantiation of "void Tpetra::MatrixMatrix::Jacobi(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, bool, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=double,
LocalOrdinal=int, GlobalOrdinal=Tpetra::longlong={long long}, Node=Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode]" at line 55 of "/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix.cpp"
/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp(3366): error: class "Tpetra::MMdetails::KernelWrappers<int, int, int, Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode>" has no member "jacobi_A_B_newmatrix_kernel_wrapper"
KernelWrappers<Scalar,LocalOrdinal,GlobalOrdinal,Node>::jacobi_A_B_newmatrix_kernel_wrapper(omega,Dinv,Aview,Bview,targetMapToOrigRow,targetMapToImportRow,Bcol2Ccol,Icol2Ccol,C,Cimport,label,params);
^
detected during:
instantiation of "void Tpetra::MMdetails::jacobi_A_B_newmatrix(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=int, LocalOrdinal=int,
GlobalOrdinal=int, Node=Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode]" at line 386
instantiation of "void Tpetra::MatrixMatrix::Jacobi(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, bool, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=int,
LocalOrdinal=int, GlobalOrdinal=int, Node=Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode]" at line 55 of "/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix.cpp"
/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp(3366): error: class "Tpetra::MMdetails::KernelWrappers<Tpetra::longlong={long long}, int, Tpetra::longlong={long long}, Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode>" has no member "jacobi_A_B_newmatrix_kernel_wrapper"
KernelWrappers<Scalar,LocalOrdinal,GlobalOrdinal,Node>::jacobi_A_B_newmatrix_kernel_wrapper(omega,Dinv,Aview,Bview,targetMapToOrigRow,targetMapToImportRow,Bcol2Ccol,Icol2Ccol,C,Cimport,label,params);
^
detected during:
instantiation of "void Tpetra::MMdetails::jacobi_A_B_newmatrix(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=Tpetra::longlong={long long},
LocalOrdinal=int, GlobalOrdinal=Tpetra::longlong={long long}, Node=Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode]" at line 386
instantiation of "void Tpetra::MatrixMatrix::Jacobi(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, bool, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with
Scalar=Tpetra::longlong={long long}, LocalOrdinal=int, GlobalOrdinal=Tpetra::longlong={long long}, Node=Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode]" at line 55 of "/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix.cpp"
/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp(3366): error: class "Tpetra::MMdetails::KernelWrappers<int, int, Tpetra::longlong={long long}, Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode>" has no member "jacobi_A_B_newmatrix_kernel_wrapper"
KernelWrappers<Scalar,LocalOrdinal,GlobalOrdinal,Node>::jacobi_A_B_newmatrix_kernel_wrapper(omega,Dinv,Aview,Bview,targetMapToOrigRow,targetMapToImportRow,Bcol2Ccol,Icol2Ccol,C,Cimport,label,params);
^
detected during:
instantiation of "void Tpetra::MMdetails::jacobi_A_B_newmatrix(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrixStruct<Scalar, LocalOrdinal, GlobalOrdinal, Node> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=int, LocalOrdinal=int,
GlobalOrdinal=Tpetra::longlong={long long}, Node=Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode]" at line 386
instantiation of "void Tpetra::MatrixMatrix::Jacobi(Scalar, const Tpetra::Vector<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, const Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, Tpetra::CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node, Node::classic> &, bool, const std::string &, const Teuchos::RCP<Teuchos::ParameterList> &) [with Scalar=int,
LocalOrdinal=int, GlobalOrdinal=Tpetra::longlong={long long}, Node=Tpetra::Kokkos_Compat_KokkosOpenMPWrapperNode]" at line 76 of "/global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix.cpp"
compilation aborted for /global/homes/j/jjellio/src/Trilinos-dev/packages/tpetra/core/ext/TpetraExt_MatrixMatrix.cpp (code 2)
packages/tpetra/core/ext/CMakeFiles/tpetraext.dir/build.make:62: recipe for target 'packages/tpetra/core/ext/CMakeFiles/tpetraext.dir/TpetraExt_MatrixMatrix.cpp.o' failed
make[2]: *** [packages/tpetra/core/ext/CMakeFiles/tpetraext.dir/TpetraExt_MatrixMatrix.cpp.o] Error 2
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:5417: recipe for target 'packages/tpetra/core/ext/CMakeFiles/tpetraext.dir/all' failed
make[1]: *** [packages/tpetra/core/ext/CMakeFiles/tpetraext.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
```
## CMake:
```
cmake -D Trilinos_LINK_SEARCH_START_STATIC:BOOL=ON
-D MPI_USE_COMPILER_WRAPPERS=OFF
-D CMAKE_CXX_COMPILER="CC"
-D CMAKE_C_COMPILER="cc"
-D CMAKE_CXX_FLAGS="-xMIC-AVX512 -mkl=parallel -g1"
-D CMAKE_C_FLAGS="-xMIC-AVX512 -mkl=parallel -g1"
-D CMAKE_Fortran_FLAGS="-xMIC-AVX512 -mkl=parallel -g1"
-D CMAKE_EXE_LINKER_FLAGS="-xMIC-AVX512 -mkl=parallel "
-D CMAKE_Fortran_COMPILER="ftn"
-D HAVE_GCC_ABI_DEMANGLE:BOOL=TRUE
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-D Trilinos_ENABLE_DEBUG:BOOL=OFF
-D CMAKE_INSTALL_PREFIX:PATH="/global/homes/j/jjellio/install/trilinos-12.13/intel-17.0.2.174_mpt-7.6.0_knl-openmp_hp-2-dev"
-D Trilinos_ENABLE_INSTALL_CMAKE_CONFIG_FILES:BOOL=ON
-D CMAKE_BUILD_TYPE:STRING=RELEASE
-D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF
-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON
-D Trilinos_ENABLE_TESTS:BOOL=OFF
-D Trilinos_ENABLE_EXAMPLES:BOOL=OFF
-D BUILD_SHARED_LIBS:BOOL=OFF
-D Trilinos_SET_INSTALL_RPATH=OFF
-D DART_TESTING_TIMEOUT:STRING=200
-D Trilinos_ENABLE_CXX11=ON
-D TPL_ENABLE_MPI=ON
-D MPI_EXEC:PATH="srun"
-D MPI_EXEC_NUMPROCS_FLAG:STRING="-c;8;--cpu_bind=cores;-n"
-D TPL_ENABLE_BLAS=ON
-D BLAS_LIBRARY_DIRS:PATH="/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64;/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/../compiler/lib/intel64_lin/;/opt/intel/compilers_and_libraries_2017.2.174/linux/lib/intel64_lin/"
-D BLAS_LIBRARY_NAMES:STRING="mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;m;dl"
-D TPL_ENABLE_LAPACK=ON
-D LAPACK_LIBRARY_DIRS:PATH="/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64;/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/../compiler/lib/intel64_lin/;/opt/intel/compilers_and_libraries_2017.2.174/linux/lib/intel64_lin/"
-D LAPACK_LIBRARY_NAMES:STRING="mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;m;dl"
-D TPL_ENABLE_Zlib=ON
-D TPL_ENABLE_DLlib=ON
-D TPL_ENABLE_yaml-cpp=ON
-D yaml-cpp_LIBRARY_DIRS:PATH=/global/homes/j/jjellio/install/yaml-cpp/intel-17.0.1.132_mpt-7.4.4_hsw/yaml-cpp/lib
-D yaml-cpp_INCLUDE_DIRS:PATH=/global/homes/j/jjellio/install/yaml-cpp/intel-17.0.1.132_mpt-7.4.4_hsw/yaml-cpp/include
-D TPL_ENABLE_Boost=ON
-D Boost_INCLUDE_DIRS:PATH="/usr/common/software/boost/1.62/intel/mic-knl/include"
-D Boost_LIBRARY_DIRS:PATH="/usr/common/software/boost/1.62/intel/mic-knl/lib"
-D TPL_ENABLE_BoostLib=ON
-D BoostLib_INCLUDE_DIRS:PATH="/usr/common/software/boost/1.62/intel/mic-knl/include"
-D BoostLib_LIBRARY_DIRS:PATH="/usr/common/software/boost/1.62/intel/mic-knl/lib"
-D TPL_ENABLE_Netcdf=ON
-D Netcdf_INCLUDE_DIRS:PATH="/opt/cray/pe/parallel-netcdf/1.8.0/intel/15.0/include;/global/homes/j/jjellio/install/netcdf-4.1.1.1/include"
-D Netcdf_LIBRARY_DIRS:PATH="/opt/cray/pe/parallel-netcdf/1.8.0/intel/15.0/lib;/global/homes/j/jjellio/install/netcdf-4.1.1.1/lib;/opt/cray/pe/hdf5-parallel/1.10.0.1/INTEL/15.0/lib"
-D Netcdf_LIBRARY_NAMES:STRING="netcdf;pnetcdf;hdf5hl_fortran;hdf5_fortran;hdf5_hl;hdf5;z"
-D TPL_Netcdf_Enables_Netcdf4:BOOL=ON
-D TPL_Netcdf_Enables_PNetcdf:BOOL=ON
-D TPL_Netcdf_PARALLEL:BOOL=ON
-D TPL_ENABLE_HDF5=ON
-D HDF5_INCLUDE_DIRS:PATH="/opt/cray/pe/hdf5-parallel/1.10.0.1/INTEL/15.0/include"
-D HDF5_LIBRARY_DIRS:PATH="/opt/cray/pe/hdf5-parallel/1.10.0.1/INTEL/15.0/lib"
-D HDF5_LIBRARY_NAMES:STRING="hdf5hl_fortran;hdf5_fortran;hdf5_hl;hdf5;z"
-D TPL_ENABLE_METIS=OFF
-D METIS_INCLUDE_DIRS:PATH="/usr/common/software/metis/5.1.0/include"
-D METIS_LIBRARY_DIRS:PATH="/usr/common/software/metis/5.1.0/lib"
-D TPL_ENABLE_ParMETIS=OFF
-D ParMETIS_INCLUDE_DIRS:PATH="/global/homes/j/jjellio/install/parmetis-4.0.3/include;/usr/common/software/metis/5.1.0/include"
-D ParMETIS_LIBRARY_DIRS:PATH="/global/homes/j/jjellio/install/parmetis-4.0.3/lib;/usr/common/software/metis/5.1.0/lib"
-D ParMETIS_LIBRARY_NAMES:STRING="parmetis;metis"
-D Trilinos_ENABLE_Kokkos=ON
-D Kokkos_ENABLE_TESTS:BOOL=ON
-D Trilinos_ENABLE_OpenMP=ON
-D Kokkos_ENABLE_Serial:BOOL=ON
-D Kokkos_ENABLE_OpenMP:BOOL=ON
-D Kokkos_ENABLE_Pthread:BOOL=OFF
-D Kokkos_ENABLE_Cuda:BOOL=OFF
-D TPL_ENABLE_Pthread=OFF
-D TPL_ENABLE_CUDA=OFF
-D Kokkos_ENABLE_Cuda_UVM:BOOL=ON
-D Teuchos_ENABLE_COMPLEX=OFF
-D Trilinos_ENABLE_Tpetra=ON
-D Tpetra_INST_SERIAL:BOOL=ON
-D Tpetra_INST_OPENMP:BOOL=ON
-D Tpetra_INST_PTHREAD:BOOL=OFF
-D Tpetra_INST_CUDA:BOOL=OFF
-D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF
-D Tpetra_ENABLE_TESTS:BOOL=OFF
-D Tpetra_ENABLE_EXAMPLES:BOOL=OFF
-D Trilinos_ENABLE_Belos=ON
-D Tpetra_ENABLE_MMM_Timings:BOOL=ON
-D Trilinos_ENABLE_MueLu=ON
-D MueLu_ENABLE_TESTS=OFF
-D MueLu_ENABLE_EXAMPLES=OFF
-D KokkosKernels_ENABLE_Experimental:BOOL=ON
-D TpetraKernels_ENABLE_Experimental:BOOL=ON
-D MueLu_ENABLE_Experimental:BOOL=ON
-D MueLu_ENABLE_Kokkos_Refactor:BOOL=ON
-D Xpetra_ENABLE_Experimental:BOOL=ON
-D Xpetra_ENABLE_Kokkos_Refactor:BOOL=ON
-D Trilinos_ENABLE_Xpetra=ON
-D Trilinos_ENABLE_Isorropia=ON
-D Trilinos_ENABLE_Galeri=ON
-D Trilinos_ENABLE_Teuchos=ON
-D Trilinos_ENABLE_Epetra=ON
-D Trilinos_ENABLE_EpetraExt=ON
-D Trilinos_ENABLE_Anasazi=ON
-D Anasazi_ENABLE_RBGen=ON
-D Trilinos_ENABLE_Stokhos=OFF
-D Trilinos_ENABLE_Amesos2=ON
-D Amesos2_ENABLE_KLU2=ON
-D Trilinos_ENABLE_Ifpack2=ON
-D Trilinos_ENABLE_Zoltan2=ON
-D Trilinos_ENABLE_Panzer=ON
-D Panzer_ENABLE_TESTS=ON
-D Panzer_ENABLE_EXAMPLES=ON
/global/homes/j/jjellio/src/Trilinos-dev
```
<!---
Replace <teamName> below with the appropriate Trilinos package/team name.
-->
@trilinos/tpetra
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1998Build failed at "Built target kokkoskernel"2017-11-28T20:28:09ZJames WillenbringBuild failed at "Built target kokkoskernel"*Created by: lkkevin*
Hi,
I am trying to build Trilinos 12.12.1 for Peridigm on CentOS 7.4. The "make -j 2" failed at [59] Built target kokkoskernel. I have been working on this issue for a couple day but could not figure out. P...*Created by: lkkevin*
Hi,
I am trying to build Trilinos 12.12.1 for Peridigm on CentOS 7.4. The "make -j 2" failed at [59] Built target kokkoskernel. I have been working on this issue for a couple day but could not figure out. Please help. Thanks.
#------ These are the prerequisites that i installed on my VM: -----#
boost_1_65_1 , cmake-3.10.0-rc5, hdf5-1.10.1, matio-1.5.11, netcdf-c-4.5.0, openmpi-3.0.0, blas 3.8.0, lapack 3.8.0.
#------ This is the CMakeError.log ------#
Performing C++ SOURCE FILE Test HAVE_TEUCHOS_LAPACKLARND failed with the following output:
Change Dir: /root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_2fa60/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_2fa60.dir/build.make CMakeFiles/cmTC_2fa60.dir/build
gmake[1]: Entering directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_2fa60.dir/src.cxx.o
/usr/local/bin/mpicxx -O2 -std=c++11 -pedantic -ftrapv -Wall -Wno-long-long -std=c++11 -DHAVE_TEUCHOS_LAPACKLARND -O3 -DNDEBUG -o CMakeFiles/cmTC_2fa60.dir/src.cxx.o -c /root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp/src.cxx
/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp/src.cxx: In function ‘int main()’:
/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp/src.cxx:12:38: warning: narrowing conversion of ‘0.0’ from ‘double’ to ‘int’ inside { } [-Wnarrowing]
int seed[4] = { 0.0, 0.0, 0.0, 1.0 };
^
/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp/src.cxx:12:38: warning: narrowing conversion of ‘0.0’ from ‘double’ to ‘int’ inside { } [-Wnarrowing]
/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp/src.cxx:12:38: warning: narrowing conversion of ‘0.0’ from ‘double’ to ‘int’ inside { } [-Wnarrowing]
/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp/src.cxx:12:38: warning: narrowing conversion of ‘1.0e+0’ from ‘double’ to ‘int’ inside { } [-Wnarrowing]
Linking CXX executable cmTC_2fa60
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_2fa60.dir/link.txt --verbose=1
/usr/local/bin/mpicxx -O2 -std=c++11 -pedantic -ftrapv -Wall -Wno-long-long -std=c++11 -DHAVE_TEUCHOS_LAPACKLARND -O3 -DNDEBUG -rdynamic CMakeFiles/cmTC_2fa60.dir/src.cxx.o -o cmTC_2fa60 /usr/local/lib/liblapack.a
CMakeFiles/cmTC_2fa60.dir/src.cxx.o: In function `main':
src.cxx:(.text.startup+0x22): undefined reference to `dlarnd_'
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_2fa60] Error 1
gmake[1]: Leaving directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_2fa60/fast] Error 2
Return value: 1
Source file was:
#define F77_BLAS_MANGLE(name,NAME) name ## _
#define DLARND_F77 F77_BLAS_MANGLE(dlarnd,DLARND)
extern "C" { double DLARND_F77(const int* idist, int* seed); }
int main()
{
const int idist = 1;
int seed[4] = { 0.0, 0.0, 0.0, 1.0 };
double val = DLARND_F77(&idist, seed);
return (val < 0.0 ? 1 : 0);
}
Performing C++ SOURCE FILE Test HAVE_CXX_PRAGMA_WEAK failed with the following output:
Change Dir: /root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_82e34/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_82e34.dir/build.make CMakeFiles/cmTC_82e34.dir/build
gmake[1]: Entering directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_82e34.dir/src.cxx.o
/usr/local/bin/mpicxx -O2 -std=c++11 -pedantic -ftrapv -Wall -Wno-long-long -std=c++11 -DHAVE_CXX_PRAGMA_WEAK -O3 -DNDEBUG -o CMakeFiles/cmTC_82e34.dir/src.cxx.o -c /root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp/src.cxx
Linking CXX executable cmTC_82e34
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_82e34.dir/link.txt --verbose=1
/usr/local/bin/mpicxx -O2 -std=c++11 -pedantic -ftrapv -Wall -Wno-long-long -std=c++11 -DHAVE_CXX_PRAGMA_WEAK -O3 -DNDEBUG -rdynamic CMakeFiles/cmTC_82e34.dir/src.cxx.o -o cmTC_82e34
CMakeFiles/cmTC_82e34.dir/src.cxx.o: In function `main':
src.cxx:(.text.startup+0x1c): undefined reference to `A::theFunction()'
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_82e34] Error 1
gmake[1]: Leaving directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_82e34/fast] Error 2
Source file was:
#include <iostream>
namespace A {
// theFunction never gets defined, because we
// don't link with a library that defines it.
// That's OK, because it's weak linkage.
#pragma weak theFunction
extern void theFunction ();
}
int main() {
std::cout << "Hi! I am main." << std::endl;
if (A::theFunction != NULL) {
// Should never be called, since we don't link
// with a library that defines A::theFunction.
A::theFunction ();
}
return 0;
}
Determining if the function dggsvd3 exists failed with the following output:
Change Dir: /root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_427b3/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_427b3.dir/build.make CMakeFiles/cmTC_427b3.dir/build
gmake[1]: Entering directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_427b3.dir/CheckFunctionExists.c.o
/usr/local/bin/mpicc -DCHECK_FUNCTION_EXISTS=dggsvd3 -O3 -DNDEBUG -o CMakeFiles/cmTC_427b3.dir/CheckFunctionExists.c.o -c /usr/local/share/cmake-3.10/Modules/CheckFunctionExists.c
Linking C executable cmTC_427b3
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_427b3.dir/link.txt --verbose=1
/usr/local/bin/mpicc -DCHECK_FUNCTION_EXISTS=dggsvd3 -O3 -DNDEBUG -rdynamic CMakeFiles/cmTC_427b3.dir/CheckFunctionExists.c.o -o cmTC_427b3 /usr/local/lib/liblapack.a /usr/local/lib/libblas.a
CMakeFiles/cmTC_427b3.dir/CheckFunctionExists.c.o: In function `main':
CheckFunctionExists.c:(.text.startup+0xc): undefined reference to `dggsvd3'
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_427b3] Error 1
gmake[1]: Leaving directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_427b3/fast] Error 2
Determining if the function dggsvd3_ exists failed with the following output:
Change Dir: /root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_01e16/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_01e16.dir/build.make CMakeFiles/cmTC_01e16.dir/build
gmake[1]: Entering directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_01e16.dir/CheckFunctionExists.c.o
/usr/local/bin/mpicc -DCHECK_FUNCTION_EXISTS=dggsvd3_ -O3 -DNDEBUG -o CMakeFiles/cmTC_01e16.dir/CheckFunctionExists.c.o -c /usr/local/share/cmake-3.10/Modules/CheckFunctionExists.c
Linking C executable cmTC_01e16
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_01e16.dir/link.txt --verbose=1
/usr/local/bin/mpicc -DCHECK_FUNCTION_EXISTS=dggsvd3_ -O3 -DNDEBUG -rdynamic CMakeFiles/cmTC_01e16.dir/CheckFunctionExists.c.o -o cmTC_01e16 /usr/local/lib/liblapack.a /usr/local/lib/libblas.a
/usr/bin/ld: /usr/local/lib/liblapack.a(dlaqps.o): undefined reference to symbol 'lround@@GLIBC_2.2.5'
/usr/lib64/libm.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_01e16] Error 1
gmake[1]: Leaving directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_01e16/fast] Error 2
Determining if the function DGGSVD3 exists failed with the following output:
Change Dir: /root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_e1f49/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_e1f49.dir/build.make CMakeFiles/cmTC_e1f49.dir/build
gmake[1]: Entering directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_e1f49.dir/CheckFunctionExists.c.o
/usr/local/bin/mpicc -DCHECK_FUNCTION_EXISTS=DGGSVD3 -O3 -DNDEBUG -o CMakeFiles/cmTC_e1f49.dir/CheckFunctionExists.c.o -c /usr/local/share/cmake-3.10/Modules/CheckFunctionExists.c
Linking C executable cmTC_e1f49
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_e1f49.dir/link.txt --verbose=1
/usr/local/bin/mpicc -DCHECK_FUNCTION_EXISTS=DGGSVD3 -O3 -DNDEBUG -rdynamic CMakeFiles/cmTC_e1f49.dir/CheckFunctionExists.c.o -o cmTC_e1f49 /usr/local/lib/liblapack.a /usr/local/lib/libblas.a
CMakeFiles/cmTC_e1f49.dir/CheckFunctionExists.c.o: In function `main':
CheckFunctionExists.c:(.text.startup+0xc): undefined reference to `DGGSVD3'
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_e1f49] Error 1
gmake[1]: Leaving directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_e1f49/fast] Error 2
Determining if the function DGGSVD3_ exists failed with the following output:
Change Dir: /root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_6d0e5/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_6d0e5.dir/build.make CMakeFiles/cmTC_6d0e5.dir/build
gmake[1]: Entering directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_6d0e5.dir/CheckFunctionExists.c.o
/usr/local/bin/mpicc -DCHECK_FUNCTION_EXISTS=DGGSVD3_ -O3 -DNDEBUG -o CMakeFiles/cmTC_6d0e5.dir/CheckFunctionExists.c.o -c /usr/local/share/cmake-3.10/Modules/CheckFunctionExists.c
Linking C executable cmTC_6d0e5
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_6d0e5.dir/link.txt --verbose=1
/usr/local/bin/mpicc -DCHECK_FUNCTION_EXISTS=DGGSVD3_ -O3 -DNDEBUG -rdynamic CMakeFiles/cmTC_6d0e5.dir/CheckFunctionExists.c.o -o cmTC_6d0e5 /usr/local/lib/liblapack.a /usr/local/lib/libblas.a
CMakeFiles/cmTC_6d0e5.dir/CheckFunctionExists.c.o: In function `main':
CheckFunctionExists.c:(.text.startup+0xc): undefined reference to `DGGSVD3_'
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_6d0e5] Error 1
gmake[1]: Leaving directory `/root/Downloads/trilinos-12.12.1-Source/build/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_6d0e5/fast] Error 2
#------ This is my build.sh -----#
#!/bin/bash
#===========================================================
make clean
rm -rf CMake*
cmake \
-D CMAKE_INSTALL_PREFIX:PATH=/usr/local/trilinos \
-D MPI_BASE_DIR:PATH=/usr/local/lib/openmpi \
-D CMAKE_CXX_FLAGS:STRING="-O2 -std=c++11 -pedantic -ftrapv -Wall -Wno-long-long" \
-D CMAKE_BUILD_TYPE:STRING=RELEASE \
-D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_Teuchos:BOOL=ON \
-D Trilinos_ENABLE_Shards:BOOL=ON \
-D Trilinos_ENABLE_Sacado:BOOL=ON \
-D Trilinos_ENABLE_Epetra:BOOL=ON \
-D Trilinos_ENABLE_EpetraExt:BOOL=ON \
-D Trilinos_ENABLE_Ifpack:BOOL=ON \
-D Trilinos_ENABLE_AztecOO:BOOL=ON \
-D Trilinos_ENABLE_Amesos:BOOL=ON \
-D Trilinos_ENABLE_Anasazi:BOOL=ON \
-D Trilinos_ENABLE_Belos:BOOL=ON \
-D Trilinos_ENABLE_ML:BOOL=ON \
-D Trilinos_ENABLE_Phalanx:BOOL=ON \
-D Trilinos_ENABLE_Intrepid:BOOL=ON \
-D Trilinos_ENABLE_NOX:BOOL=ON \
-D Trilinos_ENABLE_Stratimikos:BOOL=ON \
-D Trilinos_ENABLE_Thyra:BOOL=ON \
-D Trilinos_ENABLE_Rythmos:BOOL=ON \
-D Trilinos_ENABLE_MOOCHO:BOOL=ON \
-D Trilinos_ENABLE_TriKota:BOOL=OFF \
-D Trilinos_ENABLE_Stokhos:BOOL=ON \
-D Trilinos_ENABLE_Zoltan:BOOL=ON \
-D Trilinos_ENABLE_Piro:BOOL=ON \
-D Trilinos_ENABLE_Teko:BOOL=ON \
-D Trilinos_ENABLE_SEACASIoss:BOOL=ON \
-D Trilinos_ENABLE_SEACAS:BOOL=ON \
-D Trilinos_ENABLE_SEACASBlot:BOOL=ON \
-D Trilinos_ENABLE_Pamgen:BOOL=ON \
-D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \
-D Trilinos_ENABLE_TESTS:BOOL=ON \
-D TPL_ENABLE_HDF5:BOOL=ON \
-D HDF5_INCLUDE_DIRS:PATH=/usr/local/HDF5/include \
-D HDF5_LIBRARY_DIRS:PATH=/usr/local/HDF5/lib \
-D TPL_ENABLE_Netcdf:BOOL=ON \
-D Netcdf_INCLUDE_DIRS:PATH=/usr/local/netcdf/include \
-D Netcdf_LIBRARY_DIRS:PATH=/usr/local/netcdf/lib \
-D TPL_ENABLE_MPI:BOOL=ON \
-D TPL_ENABLE_BLAS:BOOL=ON \
-D TPL_ENABLE_LAPACK:BOOL=ON \
-D TPL_ENABLE_Boost:BOOL=ON \
-D Boost_INCLUDE_DIRS:PATH=/usr/local/boost/include \
-D Boost_LIBRARY_DIRS:PATH=/usr/local/boost/lib \
-D BLAS_LIBRARY_NAMES:STRING=libblas.a \
-D BLAS_LIBRARY_DIRS:STRING=/usr/local/lib \
-D LAPACK_LIBRARY_NAMES:STRING=liblapack.a \
-D LAPACK_LIBRARY_DIRS:STRING=/usr/local/lib \
-D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
-D Trilinos_VERBOSE_CONFIGURE:BOOL=OFF \
../
#---------------------------------------------------------------------#
<!---
Note that anything between these delimiters is a comment that will not appear
in the issue description once created. Click on the Preview tab to see what
everything will look like when you submit.
-->
<!---
Feel free to delete anything from this template that is not applicable to the
issue you are submitting.
-->
<!---
Replace <teamName> below with the appropriate Trilinos package/team name.
-->
@trilinos/<teamName>
<!---
Assignees: If you know anyone who should likely tackle this issue, select them
from the Assignees drop-down on the right.
-->
<!---
Lables: Choose any applicable package names from the Labels drop-down on the
right. Additionally, choose a label to indicate the type of issue, for
instance, bug, build, documentation, enhancement, etc.
-->
## Expectations
<!---
Tell us what you think should happen, how you think things should work, what
you would like to see in the documentation, etc.
-->
## Current Behavior
<!---
Tell us how the current behavior fails to meet your expectations in some way.
-->
## Motivation and Context
<!---
How has this expectation failure affected you? What are you trying to
accomplish? Why do we need to address this? What does it have to do with
anything? Providing context helps us come up with a solution that is most
useful in the real world.
-->
## Definition of Done
<!---
Tell us what needs to happen. If necessary, give us a task lisk along the
lines of:
- [ ] First do this.
- [ ] Then do that.
- [ ] Also this other thing.
-->
## Possible Solution
<!---
Not obligatory, but suggest a fix for the bug or documentation, or suggest
ideas on how to implement the addition or change.
-->
## Steps to Reproduce
<!---
Provide a link to a live example, or an unambiguous set of steps to reproduce
this issue. Include code to reproduce, if relevant.
1. Do this.
1. Do that.
1. Shake fist angrily at computer.
-->
## Your Environment
<!---
Include relevant details about your environment such that we can replicate this
issue.
-->
- **Relevant repo SHA1s:**
- **Relevant configure flags or configure script:**
- **Operating system and version:**
- **Compiler and TPL versions:**
## Related Issues
<!---
If applicable, let us know how this bug is related to any other open issues:
-->
* Blocks
* Is blocked by
* Follows
* Precedes
* Related to
* Part of
* Composed of
## Additional Information
<!---
Anything else that might be helpful for us to know in addressing this issue:
* Configure log file:
* Build log file:
* Test log file:
* When was the last time everything worked (date/time; SHA1s; etc.)?
* What did you do that made the bug rear its ugly head?
* Have you tried turning it off and on again?
-->
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2116kokkos-kernels: Build error when MKL TPL is enabled2018-01-17T16:43:22ZJames Willenbringkokkos-kernels: Build error when MKL TPL is enabled*Created by: mhoemmen*
```
.../Trilinos/packages/kokkos-kernels/src/sparse/impl/KokkosSparse_spgemm_mkl_impl.hpp:165:24: error: use of undeclared identifier 'valuesA'
value_type *a_ew = valuesA.ptr_on_device();
...*Created by: mhoemmen*
```
.../Trilinos/packages/kokkos-kernels/src/sparse/impl/KokkosSparse_spgemm_mkl_impl.hpp:165:24: error: use of undeclared identifier 'valuesA'
value_type *a_ew = valuesA.ptr_on_device();
^
.../Trilinos/packages/kokkos-kernels/src/sparse/impl/KokkosSparse_spgemm_mkl_impl.hpp:166:24: error: use of undeclared identifier 'valuesB'
value_type *b_ew = valuesB.ptr_on_device();
^
```
This tracks progress on https://github.com/kokkos/kokkos-kernels/issues/135 .
<!---
Replace <teamName> below with the appropriate Trilinos package/team name.
-->
@trilinos/kokkos-kernels
## Expectations
Trilinos needs to build correctly when the MKL TPL is enabled.
## Motivation and Context
MKL provides many useful computational kernels, but we can't use them in Trilinos if Trilinos won't build.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2115KokkosKernels: build failure when enabling double, complex<double>, and share...2018-02-13T17:28:55ZJames WillenbringKokkosKernels: build failure when enabling double, complex<double>, and shared libraries*Created by: jhux2*
Xpetra and KokkosKernels builds started failing on the MueLu nightly test machine "geminga" on December ~~14th~~ 16th, but I only noticed today. As a result, MueLu isn't being tested with CUDA. Here's the dashboard ...*Created by: jhux2*
Xpetra and KokkosKernels builds started failing on the MueLu nightly test machine "geminga" on December ~~14th~~ 16th, but I only noticed today. As a result, MueLu isn't being tested with CUDA. Here's the dashboard [link](https://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=3278400&filtercount=3&showfilters=1&field1=buildstamp&compare1=61&value1=20171216-1441-Experimental&filtercombine=and).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2221Pamgen,kokkos-kernels: NVCC build error with -D CMAKE_CXX_USE_RESPONSE_FILE_F...2019-03-05T22:14:10ZJames WillenbringPamgen,kokkos-kernels: NVCC build error with -D CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON*Created by: mhoemmen*
kokkos-kernels requires `-D CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON` for CUDA builds, even with complex arithmetic disabled.
https://github.com/trilinos/Trilinos/issues/2115#issuecomment-357750048
However...*Created by: mhoemmen*
kokkos-kernels requires `-D CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON` for CUDA builds, even with complex arithmetic disabled.
https://github.com/trilinos/Trilinos/issues/2115#issuecomment-357750048
However, Pamgen does not build, given those settings.
@trilinos/kokkos-kernels @trilinos/pamgen @trilinos/framework
CC: @rrdrake @prwolfe
## Expectations
1. Whatever flags kokkos-kernels needs to build on CUDA, they need not to block building other essential packages.
2. kokkos-kernels needs to document its CMake requirements.
3. kokkos-kernels needs to fail at configure time, with an informative message, if the CMake variables that it needs are not set.
## Current Behavior
```
$ make
[ 0%] Linking CXX shared library libpamgen.so
nvcc fatal : No input files specified; use option --help for more information
make[2]: *** [packages/pamgen/src/libpamgen.so.12.13] Error 1
make[1]: *** [packages/pamgen/src/CMakeFiles/pamgen.dir/all] Error 2
make: *** [all] Error 2
```
## Motivation and Context
Many downstream tests, including Belos and Ifpack2, depend on Pamgen. This blocks adequate Trilinos testing on CUDA.
## Steps to Reproduce
```
$ module list
Currently Loaded Modulefiles:
1) sems-env 3) sems-cmake/3.3.2 5) kokkos-cuda/8.0.44
2) kokkos-env 4) sems-gcc/5.3.0 6) kokkos-openmpi/2.0.1/cuda
```
CMake configuration:
```
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-D BUILD_SHARED_LIBS:BOOL=ON
-D Trilinos_ENABLE_OpenMP:BOOL=ON
-D Kokkos_ENABLE_OpenMP:BOOL=ON
-D Tpetra_INST_OPENMP:BOOL=ON
-D Trilinos_SHOW_DEPRECATED_WARNINGS:BOOL=ON
-D Trilinos_ENABLE_Fortran:BOOL=OFF
-D TPL_ENABLE_CUDA:BOOL=ON
-D KOKKOS_ARCH="SNB;Kepler35"
-D Kokkos_ENABLE_Cuda:BOOL=ON
-D Kokkos_ENABLE_Cuda_UVM:BOOL=ON
-D Tpetra_INST_CUDA:BOOL=ON
-D Kokkos_ENABLE_Cuda_Lambda:BOOL=ON
-D CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS:BOOL=ON
-D CMAKE_CXX_FLAGS:STRING="-Wall"
-D Trilinos_CXX11_FLAGS:STRING="-std=c++11 --expt-extended-lambda"
-D TPL_ENABLE_MKL:BOOL=OFF
-D TPL_ENABLE_Matio:BOOL=OFF
-D TPL_ENABLE_SuperLU:BOOL=OFF
-D TPL_ENABLE_Zlib:BOOL=OFF
-D TPL_ENABLE_Netcdf:BOOL=OFF
-D TPL_ENABLE_HDF5:BOOL=OFF
-D TPL_ENABLE_ParMETIS:BOOL=OFF
-D TPL_ENABLE_Boost:BOOL=OFF
-D TPL_ENABLE_BoostLib:BOOL=OFF
-D TPL_ENABLE_yaml-cpp:BOOL=OFF
-D TPL_ENABLE_MPI:BOOL=ON
```
## Your Environment
- develop branch, commit 400765e21e17dfb995e0f4a2759ce9c5f961b685 (likely not related)
## Related Issues
* Blocks #2115 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2439Kokkos_ENABLE_Debug_Bounds_Check=ON and KOKKOS_ENABLE_DEBUG=ON by default if ...2018-10-27T00:51:55ZJames WillenbringKokkos_ENABLE_Debug_Bounds_Check=ON and KOKKOS_ENABLE_DEBUG=ON by default if Trilinos_ENABLE_DEBUG=ON *Created by: bartlettroscoe*
**CC:** @trilinos/kokkos
## Next Action Status
`KOKKOS_ENABLE_DEBUG=ON` is enabled in `Trilinos_ENABLE_DEBUG=ON` builds and in ATDM builds. New failures caused by this in ATDM builds is being addresse...*Created by: bartlettroscoe*
**CC:** @trilinos/kokkos
## Next Action Status
`KOKKOS_ENABLE_DEBUG=ON` is enabled in `Trilinos_ENABLE_DEBUG=ON` builds and in ATDM builds. New failures caused by this in ATDM builds is being addressed in #2471.
## Description
Currently, enabling `Trilinos_ENABLE_DEBUG=ON` does **not** by default enable `KOKKOS_ENABLE_DEBUG=ON` or `Kokkos_ENABLE_Debug_Bounds_Check=ON`. This is a problem that means that **none** of the automated builds for Trilinos seem to be testing Trilinos with these enabled a shown by:
```
$ cd Trilinos/
$ find cmake/ctest/drivers/ -name "*.cmake" \
-exec grep -nH Kokkos_ENABLE_Debug_Bounds_Check {} \;
... empty ...
```
This is a problem for ATDM because the ATDM builds of Trilinos are explicitly setting `Kokkos_ENABLE_Debug_Bounds_Check=ON` for debug builds and no other automated builds of Trilinos (including the CI and auto PR testing builds) ever test with this turned on. This means that all of auto PR, CI and other builds of Trilinos that pass with `Trilinos_ENABLE_DEBUG=ON` will still allow code to fail when `KOKKOS_ENABLE_DEBUG=ON` and `Kokkos_ENABLE_Debug_Bounds_Check=ON` are set.
## Related Issues
* Blocks: #2317 (need a full debug build)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2471New failing tests in ATDM debug builds of Trilinos due to KOKKOS_ENABLE_DEBUG...2018-11-30T03:09:20ZJames WillenbringNew failing tests in ATDM debug builds of Trilinos due to KOKKOS_ENABLE_DEBUG=ON being set*Created by: bartlettroscoe*
**CC:** @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/amesos2 , @trilinos/anasazi, @trilinos/panzer
## Next Action Status
The PR #2476 fixed two of the tests on 3/30/2018 and PR #2494 disabled...*Created by: bartlettroscoe*
**CC:** @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/amesos2 , @trilinos/anasazi, @trilinos/panzer
## Next Action Status
The PR #2476 fixed two of the tests on 3/30/2018 and PR #2494 disabled one single unit test on 4/3/2018 not appropriate to run on GPUs.
## Description
As shown in the query:
* https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-28&limit=0&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=-debug&field3=status&compare3=62&value3=passed
several tests are timing out today and failing in the ATDM `-debug` builds of Trilinos:
* `Amesos2_KLU2_UnitTests_MPI_2`
* `Anasazi_Epetra_ModalSolversTester_MPI_4`
* `KokkosCore_UnitTest_Cuda_MPI_1`
* `KokkosKernels_sparse_cuda_MPI_1`
* `PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4`
The set of tests that are failing and which platforms they are failing shown in the above query are shown in the below table:
<details>
<summary>
<b>Table of failing tests</b> (click to expend)
</summary>
| Site | Build Name | Test Name | Status | Time | Details |
| :-- | :-- | :-- | :-- | --: | :-- |
| hansen | Trilinos-atdm-hansen-shiller-cuda-debug | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.09 | Completed (Timeout) |
| hansen | Trilinos-atdm-hansen-shiller-gnu-debug-openmp | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 8.51 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-gnu-debug-serial | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.51 | Completed (Timeout) |
| hansen | Trilinos-atdm-hansen-shiller-intel-debug-openmp | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 2.39 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-intel-debug-serial | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.1 | Completed (Timeout) |
| ride | Trilinos-atdm-white-ride-cuda-debug | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.05 | Completed (Timeout) |
| white | Trilinos-atdm-white-ride-cuda-debug | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 600.04 | Completed (Timeout) |
| white | Trilinos-atdm-white-ride-gnu-debug-openmp | Amesos2_KLU2_UnitTests_MPI_2 | Failed | 1.31 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | Anasazi_Epetra_ModalSolversTester_MPI_4 | Failed | 0.84 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4 | Failed | 0.71 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-cuda-debug | KokkosCore_UnitTest_Cuda_MPI_1 | Failed | 103.19 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | KokkosCore_UnitTest_Cuda_MPI_1 | Failed | 213.56 | Completed (Failed) |
| white | Trilinos-atdm-white-ride-cuda-debug | KokkosCore_UnitTest_Cuda_MPI_1 | Failed | 213.74 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-cuda-debug | KokkosKernels_sparse_cuda_MPI_1 | Failed | 16.49 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | KokkosKernels_sparse_cuda_MPI_1 | Failed | 2.43 | Completed (Failed) |
| white | Trilinos-atdm-white-ride-cuda-debug | KokkosKernels_sparse_cuda_MPI_1 | Failed | 2.39 | Completed (Failed) |
| hansen | Trilinos-atdm-hansen-shiller-cuda-debug | PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4 | Failed | 21.95 | Completed (Failed) |
| ride | Trilinos-atdm-white-ride-cuda-debug | PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4 | Failed | 24.31 | Completed (Failed) |
| white | Trilinos-atdm-white-ride-cuda-debug | PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4 | Failed | 24.9 | Completed (Failed) |
</details>
*
Except for the failing Anasazi tests in the build `Trilinos-atdm-white-ride-cuda-debug` (which I will write another GitHub issue for), all of these tests (even the timeouts) seem to be failing due to debug-mode checks from `KOKKOS_ENABLE_DEBUG=ON` being set (see #2439) failing and throwing exceptions. In the case of the failing tests `Amesos2_KLU2_UnitTests_MPI_2`, for example, it shows:
```
5. KLU2_double_int_int_NonContgGID_UnitTest ...
p=0: *** Caught standard std::exception of type 'std::runtime_error' :
View bounds error of view MV::DualView ( -1 < 6 , 0 < 1 )
Traceback functionality not available
[FAILED] (0.0885 sec) KLU2_double_int_int_NonContgGID_UnitTest
Location: /home/rabartl/WHITE/ATDM_Driver/Trilinos-atdm-white-ride-cuda-debug/SRC_AND_BUILD/Trilinos/packages/amesos2/test/solvers/KLU2_UnitTests.cpp:383
```
This exception causes a hang and a timeout in some cases and fails quickly and aborts in other cases. (So much for assuming that one MPI process throwing an excpetion will bring down an MPI job in all cases.)
Many of these builds have been promoted to the "ATDM" CDash group/track and therefore triggered CDash error emails today. Therefore, this must get fixed quickly if possible (or we will need to demote these builds again).
## Steps to Reproduce
One can log onto `white` (SON) or `ride` (SRN) and then reproduce the build and tests as described at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
I just reproduced many of these failures on 'white' using
```
$ ssh white
$ cd ~/rilinos.base/BUILD/WHITE/CHECKIN/
$ bsub -x -I -q rhel7F -n 16 \
./checkin-test-atdm.sh cuda-debug --enable-packages=Kokkos,KokkosKernels,Amesos2,Panzer --local-do-all
...
FAILED (NOT READY TO PUSH): Trilinos: white22
Wed Mar 28 11:03:05 MDT 2018
Enabled Packages: Kokkos, KokkosKernels, Amesos2, Panzer
Build test results:
-------------------
0) MPI_RELEASE_DEBUG_SHARED_PT => Test case MPI_RELEASE_DEBUG_SHARED_PT was not run! => Does not affect push readiness! (-1.00 min)
1) cuda-debug => FAILED: passed=189,notpassed=5 => Not ready to push! (120.13 min)
REQUESTED ACTIONS: FAILED
```
This showed the test results:
```
97% tests passed, 5 tests failed out of 194
Subproject Time Summary:
Amesos2 = 1232.97 sec*proc (8 tests)
Kokkos = 954.94 sec*proc (26 tests)
KokkosKernels = 870.46 sec*proc (8 tests)
Panzer = 7490.79 sec*proc (152 tests)
Total Test time (real) = 1518.22 sec
The following tests FAILED:
2 - KokkosCore_UnitTest_Cuda_MPI_1 (Failed)
28 - KokkosKernels_sparse_cuda_MPI_1 (Failed)
35 - Amesos2_KLU2_UnitTests_MPI_2 (Timeout)
174 - PanzerAdaptersSTK_PoissonInterfaceExample_2d_diffsideids_MPI_1 (Timeout)
192 - PanzerMiniEM_MiniEM-BlockPrec_Augmentation_MPI_4 (Failed)
Errors while running CTest
Total time for cuda-debug = 120.13 min
```
The test failure timeout `PanzerAdaptersSTK_PoissonInterfaceExample_2d_diffsideids_MPI_1` was also seen in #2446 as well. Not sure why that test timed out when run locally but not in the driver jobs. But otherwise, this one build reproduced all of the failing tests shown on CDash except for the test `Anasazi_Epetra_ModalSolversTester_MPI_4` (which does not look to be related to `KOKKOS_ENABLE_DEBUG=ON`).
## Related Issues
* Related to: #2439, #2464
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2434Building pytrilinos with GCC 6.3.0 fails with "execvp: /bin/sh: Argument list...2018-08-24T22:27:28ZJames WillenbringBuilding pytrilinos with GCC 6.3.0 fails with "execvp: /bin/sh: Argument list too long"*Created by: samfux84*
@trilinos/pytrilinos
I am trying to build Trilinos 12.10.1 with python support. When compiling, it crashes with the error
```
make[2]: execvp: /bin/sh: Argument list too long
packages/PyTrilinos/src/CMakeF...*Created by: samfux84*
@trilinos/pytrilinos
I am trying to build Trilinos 12.10.1 with python support. When compiling, it crashes with the error
```
make[2]: execvp: /bin/sh: Argument list too long
packages/PyTrilinos/src/CMakeFiles/PyTrilinos_LOCA_Epetra_Interface.dir/build.make:192: recipe for target 'packages/PyTrilinos/src/LOCA.Epetra.InterfacePYTHON_wrap.cpp' failed
make[2]: *** [packages/PyTrilinos/src/LOCA.Epetra.InterfacePYTHON_wrap.cpp] Error 127
```
The command (38741 characters) that causes this error is:
```
cd /scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/muelu/src && /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/openmpi-2.1.0-sjd6nhdb3d4y6hhk2qtx23r3ypdi7coy/bin/mpic++ -Dmuelu_EXPORTS -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/BrickAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/UncoupledAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/UserAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Headers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Interface -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Interface/FacadeClasses -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Misc -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/MueCentral -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Rebalancing -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Smoothers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/BaseClass -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/BlockedTransfers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Generic -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Energy-Minimization -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Energy-Minimization/Solvers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Geometric-Interpolation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Smoothed-Aggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Petrov-Galerkin-SA -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/User -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/SemiCoarsen -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Utils/ForwardDeclaration -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/../adapters/epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/../adapters/tpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/muelu/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teko/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/NS -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/Epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/Tpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/stratimikos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/belos/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/stratimikos/adapters/belos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/aztecoo/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/ml/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/ifpack/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/tpetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/tpetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/epetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/aztecoo/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/aztecoo/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/triutils/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/triutils/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/kokkoscomm/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/kokkoscomm/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/kokkoscompat/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/kokkoscompat/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/parameterlist/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/core/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/core/src -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/boost-1.63.0-w3vrb2dy7hu4x5n5mgqxyw7xjxjnmhtk/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/comm/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/remainder/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/remainder/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/numerics/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/tpetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_vector/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_vector/extended -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_vector/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_vector/adapter_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_solve/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_solve/extended -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_solve/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/nonlinear/model_evaluator/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/nonlinear/model_evaluator/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/nonlinear/solvers/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/nonlinear/solvers/client_support -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/thyra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/example/operator_vector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/interfaces -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/ops_lib -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/lapack -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/rtop/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/ext -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/ext -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/inout -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/inout -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/src/kokkos_refactor -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/tsqr/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/tsqr/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/LinAlg -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/NodeAPI -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/classic/NodeAPI -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/classic/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/kernels/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/kernels/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/kernels/src/impl -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/algorithms/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/algorithms/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/containers/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/containers/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetraext/src/model_evaluator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetraext/src/transformer -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/epetraext/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/transform -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/inout -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/coloring -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/model_evaluator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/block -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/restrict -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/hypre -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/python-2.7.13-kz2b5ll34ua4qdjbflypwhtk3ujlitr4/include/python2.7 -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/hypre-2.11.1-nqgf2yzso76zjo53ad7yf7jrls2teaqc/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/suite-sparse-4.5.5-rlv2pziqccmce2ppuztg5rybahktvih7/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/hdf5-1.10.1-cn37gr2d2rhtcw3ep5neqy7ech3n7oit/include -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/ml/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Comm -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Coarsen -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/FEGrid -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Main -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Smoother -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Krylov -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Operator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/RefMaxwell -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/MatrixFree -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/MLAPI -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/LevelWrap -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/ifpack/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/../amesos/src/SuiteSparse/AMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/../amesos/src/SuiteSparse/UFconfig -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/AMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/COLAMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CHOLMOD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/UFconfig -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/KLU/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/BTF/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CAMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CCOLAMD/Include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/mumps-5.0.1-cp5ucvw44w6y5izcns3mgcw6l4qmb3pf/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/superlu-dist-5.1.1-l766mdbscb27rd5y2si6nhy57gnipm6f/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/parmetis-4.0.3-kshbkdzjzhfp324mzyqu5dcyv7aacis2/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-xpetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src-epetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-xpetra/../src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src/Headers -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/Maps -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/CRSMatrices -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/FiniteElements -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Matrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/StridedMap -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/BlockedCrsMatrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/BlockedMultiVector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Cloner -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Utils -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/xpetra/sup -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/CrsGraph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/CrsMatrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/DistObject -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Export -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Headers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Import -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Map -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/MultiVector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Operator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Platform -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/RowGraph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpe-12-10-1/packages/xpetra/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Vector -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Tril-release-12-10-1/spack-build/packages/isorropia/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6kgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/isorropia/src/epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko7stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/zoltan/src -I/cluster/apps/spack/var/spack/stage/trilinos-ack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/all -I/cluster/apps/spac-I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan//packages/zoltan/src/ha -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-rilinos-release-12-10-1/packages/zoltan/src/hsfc -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Tzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/matrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/par -I/cluster/apps/spack/var/spack/stage/t/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/tpls -I/cluster/ahg -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zolt1/packages/zoltan/src/reftree -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-reilinos-trilinos-release-12-10-1/packages/zoltan/src/timer -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwis-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/Utilities/DDirectory -I/cluster/apps/spac/Timer -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/os-release-12-10-1/packages/zoltan/src/zz -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/metis-5.1.0-s3cgl2pdmunmqnqksn4l7eyklibma7y/packages/ifpack2/adapters/thyra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packagee-12-10-1/packages/ifpack2/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/-1 -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/amesos2/src -I/cluster/apps/luster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos2/src-release-12-10-1/packages/amesos2/src/SuiteSparse/KLU2/Source -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzar/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos2/src/../../amesos/src/S-release-12-10-1/packages/amesos2/src/../../amesos/src/SuiteSparse/BTF/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77r/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src -I/sspack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms zoltan2/src/algorithms/partition -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinosyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms/color -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms/map -I/cln2/src/algorithms/zoltan -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-releaserilinos-trilinos-release-12-10-1/packages/zoltan2/src/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrz7bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/models -I/cluster/apps/spack/var/spack/stage/trilinos/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/util -I/scratch/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/anasazi/tpetra/src -I/scrat-I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/anasazi/anasazi/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-1ackages/anasazi/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-1release-12-10-1/packages/anasazi/src/../thyra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/G -fPIC -o CMakeFiles/muelu.dir/Interface/MueLu_ML2MueLuParameterTranslator.cpp.o -c /cluster/apps/spack/var/spack/stage/trilinos-12.10..cpp
```
Is this a known problem and is there any solution for it ?
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2511Address test timeouts on 'ride' for build Trilinos-atdm-white-ride-gnu-debug-...2018-11-30T03:09:20ZJames WillenbringAddress test timeouts on 'ride' for build Trilinos-atdm-white-ride-gnu-debug-openmp*Created by: bartlettroscoe*
**CC:** @trilinos/intrepid2, @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/panzer, @fryeguy52,
## Next Action Status
Pushed commit d0bfab7 and updated the Jenkins job `Trilinos-atdm-white-ride-...*Created by: bartlettroscoe*
**CC:** @trilinos/intrepid2, @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/panzer, @fryeguy52,
## Next Action Status
Pushed commit d0bfab7 and updated the Jenkins job `Trilinos-atdm-white-ride-gnu-debug-openmp` on 'ride' to send results to the "Specialized" CDash Group/Track on 4/5/2018. After major upgrade of 'ride' (see [TRIL-215](https://software-sandbox.sandia.gov/jira/browse/TRIL-215)) we have closed this since it is likely invalid.
## Description
A few different tests from the packages:
* Intrepid2
* Kokkos
* KokkosKernels
* Panzer
have been timing-out on the ATDM build `Trilinos-atdm-white-ride-gnu-debug-openmp` on 'ride' the last couple of days as can be seen by the query:
* https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercount=6&showfilters=1&filtercombine=and&field1=status&compare1=62&value1=Passed&field2=buildname&compare2=64&value2=Trilinos-atdm-white-ride-cuda-opt&field3=buildname&compare3=64&value3=Trilinos-atdm-white-ride-gnu-opt&field4=buildname&compare4=65&value4=Trilinos-atdm-white-ride&field5=buildstarttime&compare5=84&value5=2018-04-06&field6=buildstarttime&compare6=83&value6=2018-04-04
These builds are promoted to the "ATDM" CDash Group so they sent out CDash error emails to these packages lists
Note that that above query does not show any of these tests timing out for the **idential** builds on 'white'. The package test suites for these packages all passed just fine on 'white' the last two days as shown at:
* https://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=3483346
* https://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=3481469
Therefore, that suggests that these timeouts were caused by problems with 'ride', and not the Trilinos code and tests themselves.
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2598Set up "Specialized" ATDM CUDA build of Trilinos that enables RDC2019-03-11T21:14:14ZJames WillenbringSet up "Specialized" ATDM CUDA build of Trilinos that enables RDC*Created by: bartlettroscoe*
**CC**: @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/shylu, @trilinos/framework
## Description
The current ATDM CUDA builds of Trilinos do not set `Kokkos_ENABLE_Cuda_Relocatable_Device_Code...*Created by: bartlettroscoe*
**CC**: @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/shylu, @trilinos/framework
## Description
The current ATDM CUDA builds of Trilinos do not set `Kokkos_ENABLE_Cuda_Relocatable_Device_Code=ON` (see discussion in #2464 and #2580). However, this is needed for task-based algorithms like in ShyLU_NodeTacho and the ATDM codes may take advantage of this in the future (more likely in SPARC than EMPIRE it would seem).
Therefore, this story is to set up a new nightly build of some CUDA configuration of Trilinos that enables `Kokkos_ENABLE_Cuda_Relocatable_Device_Code=ON` so that the Trilinos developers can work to get this build cleaned up and ready for the ATDM customers to use on the ATDM platforms.
Internal SNL conversations about this topic are in companion issue:
* https://sems-atlassian-son.sandia.gov/jira/browse/TRIL-262
ToDo: Define targeted platforms and base build configurations, etc.
## Related Issues
* Related to: #2464, #2580
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2683MueLu failing to compile on cuda-92018-11-30T03:09:57ZJames WillenbringMueLu failing to compile on cuda-9*Created by: bathmatt*
Not sure how long it's been failing
```
muelu/src/Utils/ExplicitInstantiation/MueLu_Utilities_kokkos.cpp
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(369): warning...*Created by: bathmatt*
Not sure how long it's been failing
```
muelu/src/Utils/ExplicitInstantiation/MueLu_Utilities_kokkos.cpp
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(369): warning: calling a __host__ function("__builtin_cabs") from a __host__ __device__ function(" const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(387): warning: calling a __host__ function("__builtin_cabs") from a __host__ __device__ function(" const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(369): warning: calling a __host__ function("__builtin_cabs") from a __host__ __device__ function(" const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/muelu/src/Utils/MueLu_Utilities_kokkos_def.hpp(387): warning: calling a __host__ function("__builtin_cabs") from a __host__ __device__ function(" const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(385): error: calling a __host__ function("__builtin_cabs") from a __device__ function("Kokkos::Impl::ParallelFor< ::, ::Kokkos::RangePolicy<int, ::Kokkos::Cuda > , ::Kokkos::Cuda> ::operator () const") is not allowed
/home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp(385): error: calling a __host__ function("__builtin_cabs") from a __device__ function("Kokkos::Impl::ParallelFor< ::, ::Kokkos::RangePolicy<int, ::Kokkos::Cuda > , ::Kokkos::Cuda> ::operator () const") is not allowed
2 errors detected in the compilation of "/tmp/tmpxft_00008577_00000000-6_MueLu_Utilities_kokkos.cpp1.ii".
ninja: build stopped: subcommand failed.
```Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2735Kokkos Promotion: 2.6 -> 2.72018-05-25T17:09:16ZJames WillenbringKokkos Promotion: 2.6 -> 2.7*Created by: crtrott*
This is to track progress on the Kokkos promotion. Part of this promotion is to remove utilization of deprecated features from Trilinos. See https://github.com/kokkos/kokkos/wiki/DeprecationPage for a list of featu...*Created by: crtrott*
This is to track progress on the Kokkos promotion. Part of this promotion is to remove utilization of deprecated features from Trilinos. See https://github.com/kokkos/kokkos/wiki/DeprecationPage for a list of features (may not yet be complete). The branch for this is kokkos-promotion. Add a symlink into the root Trilinos directory to a separate git clone of kokkos and Kokkos-kernels with their develop branches checked out. Then add
```
-D Kokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \
-D KokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels \
-D KOKKOS_ENABLE_DEPRECATED_CODE:BOOL=OFF \
```
To your configure script. https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2772Tpetra: Micro-Benchmark MultiVector Update2018-05-23T21:06:28ZJames WillenbringTpetra: Micro-Benchmark MultiVector Update*Created by: csiefer2*
1) Write a Tpetra vs. KokkosBlas vs. Raw Lambda micro-benchmark to compare MultiVector Update performance in serial. Check into Tpetra @csiefer2
2) Study to understand performance @crtrott
Other potentially...*Created by: csiefer2*
1) Write a Tpetra vs. KokkosBlas vs. Raw Lambda micro-benchmark to compare MultiVector Update performance in serial. Check into Tpetra @csiefer2
2) Study to understand performance @crtrott
Other potentially interested parties @jhux2 @srajama1 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2827New Kokkos, KokkosKernels, and Panzer test failures on CUDA 8.0 and CUDA 9.0 ...2018-11-30T03:09:20ZJames WillenbringNew Kokkos, KokkosKernels, and Panzer test failures on CUDA 8.0 and CUDA 9.0 builds after Kokkos and KokkosKernels update*Created by: bartlettroscoe*
CC: @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/panzer, @ndellingwood
## Next Action Status
Kokkos, KokkosKernels, and Panzer failing and timing-out tests have been fixed by PRs #2863, #2874,...*Created by: bartlettroscoe*
CC: @trilinos/kokkos, @trilinos/kokkos-kernels, @trilinos/panzer, @ndellingwood
## Next Action Status
Kokkos, KokkosKernels, and Panzer failing and timing-out tests have been fixed by PRs #2863, #2874, #2927, and #2964 . No Panzer, Kokkos or KokkosKernels failures observed 6/19 or 6/20/2018.
## Description
The Kokkos and KokkosKernels updates in the recent commits 51cb7c5 and 816e703:
```
51cb7c5: Merge branch 'develop' into kokkos-promotion
Author: ndellingwood <ndellin@sandia.gov>
Date: Thu May 24 23:55:26 2018 -0600
816e703: Snapshot of kokkos-kernels.git from commit 1a7b524ba38fdfab6c1058065af06cbcb4a2ce6f
Author: Nathan Ellingwood <ndellin@sandia.gov>
Date: Thu May 24 23:30:27 2018 -0600
```
seem to have triggered several new test failures and timeouts in the packages in Kokkos, KokkosKernels, and Panzer as shown in:
* https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&date=2018-05-26&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=Trilinos-atdm-&field2=status&compare2=62&value2=passed&field3=status&compare3=62&value3=notrun&field4=buildname&compare4=63&value4=cuda&field5=buildname&compare5=62&value5=Trilinos-atdm-white-ride-cuda-debug-pt-all-at-once
The new failing and timing-out tests are:
| Test | Status | Details |
| :-- | :-- | :-- |
| KokkosContainers_UnitTest_Serial_MPI_1 | Failed | Completed (Timeout) |
| KokkosCore_UnitTest_Cuda_MPI_1 | Failed | Completed (Failed) |
| KokkosKernels_sparse_serial_MPI_1 | Failed | Completed (Timeout) |
| PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-2 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_PoissonExample-ConvTest-Quad-Order-3 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_PoissonExample-ConvTest-Quad-Order-4 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_PoissonExample-ConvTest-Tri-Order-3 | Failed | Completed (Failed) |
| PanzerAdaptersSTK_PoissonExample-ConvTest-Tri-Order-4 | Failed | Completed (Failed) |
which failed in one or more of the unique builds:
* Trilinos-atdm-hansen-shiller-cuda-8.0-debug
* Trilinos-atdm-hansen-shiller-cuda-8.0-opt
* Trilinos-atdm-white-ride-cuda-debug
* Trilinos-atdm-white-ride-cuda-opt
These are all basically CUDA 8.0 builds.
These commits were shown pulled in this testing day at:
* https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3547898#!#note0
## Steps to Reproduce
The most failures are produced on the `Trilinos-atdm-white-ride-cuda-debug` build on 'white' and 'ride' so that is likely the bet bet to use to reproduce these failures. Therefore, as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
after logging into 'white' or 'ride' and cloning the Trilinos Git repo (pointed to by `TRILINOS_DIR`) and getting on the 'develop' branch, one would do:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Kokkos=ON \
-DTrilinos_ENABLE_KokkosKernels=ON \
-DTrilinos_ENABLE_Panzer=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2877Tpetra: Norm2 (dot) is substantially slower than direct calls to TPL BLAS2018-06-04T21:26:32ZJames WillenbringTpetra: Norm2 (dot) is substantially slower than direct calls to TPL BLAS*Created by: jjellio*
When strong scaling an app, we noticed the norm2 (among other things) is 'blowing' up. In this case, the customer has a challenging problem that they want to strong scale, while using 2 processes per node and 16...*Created by: jjellio*
When strong scaling an app, we noticed the norm2 (among other things) is 'blowing' up. In this case, the customer has a challenging problem that they want to strong scale, while using 2 processes per node and 16 threads per process (On Haswell nodes). The problem, is that this results in relatively small work per process for the dense linear algebra, and KokkosKernels::BLAS does not impose a reasonable minimum chunk size for its operations.
Considering that vendor BLAS is Intel's MKL, it isn't surprising that is performs very well. This leaves a reasonable question: Why doesn't Tpetra prefer vendor BLAS?
To support my argument, I modified Belos' MultiVectorTrait::norm2 to call TPL BLAS ddot rather than Tpetra::norm2 (which if you follow the rabbit hole, eventually calls KokkosKernels::BLAS:dot).
I wrote a simple test code that calls MVT::Norm2 a few thousand times in a loop. I profiled this code linked against my modified MVT and the vanilla Trilinos one. (I.e., TPL ddot vs Kokkos dot). For the experiment, I fixed the data per MPI process to 1000 elements (i.e., a very small work size). I then weak scale this perfectly, incrementally filling nodes with 2 processes.
I also profiled the cost of Teuchos::reduceAll, with a single scalar. I ran this with OMP_NUM_THREADS=1 and 16.
## Regular MVT::norm2 and All Reduce (1 thread)
![baseline1-1](https://user-images.githubusercontent.com/21248657/40936039-351eb936-67f7-11e8-9d0d-18496b9b4b3d.png)
## Regular MVT::norm2 and All Reduce (16 thread)
![baseline16-1](https://user-images.githubusercontent.com/21248657/40936049-3cffa28c-67f7-11e8-8103-dd71b81a0545.png)
## All Reduce unaffected by threads
![baselinethreadcomp2-1](https://user-images.githubusercontent.com/21248657/40936072-54554c66-67f7-11e8-98fc-873a66d82fd0.png)
## Using TPL dot with 1 or 16 threads
![modbelos-1](https://user-images.githubusercontent.com/21248657/40936110-7755e310-67f7-11e8-8596-5e21dc86dcd6.png)
## TPL ddot vs Kokkos
![modbelos-2](https://user-images.githubusercontent.com/21248657/40936122-858f7c0c-67f7-11e8-948a-3585142f5505.png)
![modbelos-3](https://user-images.githubusercontent.com/21248657/40936126-899cff22-67f7-11e8-89ed-dd5f65d9d921.png)
While I profiled norm2, the issue is really the underlying call to ddot. In this case, MKL is doing a much better job of throttling back it's thread count. Still, calling the threaded BLAS is an overall loss, and in this case calling a purely serial MKL would have been better.
One option to mitigate the lack of thread scaling we see, is to enforce compute ensure that BLAS operations are called with a meaningful chunk size. A simpler solution, that would reduce the Trilinos code base, would be to call TPL BLAS for Serial/Thread/and OpenMP execution spaces.
This contradicts the recommendation in #2850
@trilinos/tpetra
@trilinos/kokkos-kernels https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2894MueLu: types mismatch in Driver.cpp equilibration2018-06-06T17:13:43ZJames WillenbringMueLu: types mismatch in Driver.cpp equilibration*Created by: lucbv*
@trilinos/muelu
## Expectations
The vector and matrix Scalar types forming a linear system should be consistent.
## Current Behavior
It seems that a call to KokkosBlas::abs is done on a `Scalar` type vector ...*Created by: lucbv*
@trilinos/muelu
## Expectations
The vector and matrix Scalar types forming a linear system should be consistent.
## Current Behavior
It seems that a call to KokkosBlas::abs is done on a `Scalar` type vector and a `magnitude` type vector.
## Motivation and Context
The code is not compiling properly when `std::complex<>` is used
## Definition of Done
- [ ] MueLu Driver compiles
## Possible Solution
My guess is that in MueLu_Driver.cpp on line 167 where the call to KokkosBlas::abs() is made, the two vectors should use the same `Scalar` type. Most likely the magnitude type needs to be replaced by a Scalar type even if this means that in the case of complex numbers only the real part is non zero.
## Steps to Reproduce
Build MueLu with tests and examples on and with `Trilinos_ENABLE_Complex=ON`.
## Your Environment
See builds on cdashhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2921Test TpetraCore_MatrixMatrix_UnitTests_MPI_4 failing in all ATDM Trilinos CUD...2018-11-30T03:09:57ZJames WillenbringTest TpetraCore_MatrixMatrix_UnitTests_MPI_4 failing in all ATDM Trilinos CUDA builds starting 6/7/2018*Created by: bartlettroscoe*
CC: @trilinos/tpetra, @fryeguy52, @kddevin (Data Services Product Lead)
## Next Action Status
Merged PR #2122 fixed this in all ATDM builds on 6/11/2018.
## Description
As shown in [this query fo...*Created by: bartlettroscoe*
CC: @trilinos/tpetra, @fryeguy52, @kddevin (Data Services Product Lead)
## Next Action Status
Merged PR #2122 fixed this in all ATDM builds on 6/11/2018.
## Description
As shown in [this query for the test TpetraCore_MatrixMatrix_UnitTests_MPI_4 between 6/5/2018 and 6/8/2018](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-06-07&filtercombine=and&filtercombine=and&filtercount=14&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=62&value2=Trilinos-atdm-mutrino-intel-debug-openmp&field3=buildname&compare3=62&value3=Trilinos-atdm-mutrino-intel-opt-openmp&field4=buildname&compare4=62&value4=Trilinos-atdm-white-ride-cuda-debug-pt-all-at-once&field5=buildname&compare5=62&value5=Trilinos-atdm-serrano-intel-debug-openmp&field6=buildname&compare6=62&value6=Trilinos-atdm-serrano-intel-opt-openmp&field7=buildname&compare7=62&value7=Trilinos-atdm-chama-intel-opt-openmp&field8=buildname&compare8=62&value8=Trilinos-atdm-chama-intel-debug-openmp-panzer&field9=buildname&compare9=62&value9=Trilinos-atdm-chama-intel-debug-openmp&field10=buildname&compare10=62&value10=Trilinos-atdm-chama-intel-opt-openmp-panzer&field11=site&compare11=62&value11=ride&field12=testname&compare12=61&value12=TpetraCore_MatrixMatrix_UnitTests_MPI_4&field13=buildstarttime&compare13=84&value13=2018-06-09&field14=buildstarttime&compare14=83&value14=2018-06-05), this test started failing in all of the CUDA builds:
* Trilinos-atdm-hansen-shiller-cuda-8.0-debug
* Trilinos-atdm-hansen-shiller-cuda-8.0-opt
* Trilinos-atdm-hansen-shiller-cuda-9.0-debug
* Trilinos-atdm-hansen-shiller-cuda-9.0-opt
* Trilinos-atdm-white-ride-cuda-debug
* Trilinos-atdm-white-ride-cuda-opt
starting on 6/7/2018.
The failing test output, for example, for the build `Trilinos-atdm-hansen-shiller-cuda-9.0-opt` on 'hansen' on 2018-06-07T18:36:47 UTC shown at:
* https://testing-vm.sandia.gov/cdash/testDetails.php?test=48129338&build=3591841
showed
```
p=0: *** Caught standard std::exception of type 'std::runtime_error' :
Invalid SPGEMMAlgorithm name
[FAILED] (6.84 sec) Tpetra_MatMat_double_int_int_Kokkos_Compat_KokkosCudaWrapperNode_operations_test_UnitTest
Location: /home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-9.0-opt/SRC_AND_BUILD/Trilinos/packages/tpetra/core/test/MatrixMatrix/MatrixMatrix_UnitTests.cpp:789
...
p=0: *** Caught standard std::exception of type 'std::runtime_error' :
Invalid SPGEMMAlgorithm name
Tpetra sparse matrix-matrix multiply: range row test
getIdentityMatrix
Create row Map
Create CrsMatrix
[FAILED] (0.865 sec) Tpetra_MatMat_double_int_longlong_Kokkos_Compat_KokkosCudaWrapperNode_operations_test_UnitTest
Location: /home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-9.0-opt/SRC_AND_BUILD/Trilinos/packages/tpetra/core/test/MatrixMatrix/MatrixMatrix_UnitTests.cpp:789
...
The following tests FAILED:
0. Tpetra_MatMat_double_int_int_Kokkos_Compat_KokkosCudaWrapperNode_operations_test_UnitTest ...
10. Tpetra_MatMat_double_int_longlong_Kokkos_Compat_KokkosCudaWrapperNode_operations_test_UnitTest ...
Total Time: 11.1 sec
Summary: total = 20, run = 20, passed = 18, failed = 2
End Result: TEST FAILED
```
All of the other failed test runs showed about identical test output.
When the test passed for the build `Trilinos-atdm-hansen-shiller-cuda-9.0-opt` on 'hansen' on 2018-06-05T20:22:00 UTC shown at:
* https://testing-vm.sandia.gov/cdash/testDetails.php?test=47987300&build=3582617
it showed the test output:
```
Total Time: 29.4 sec
Summary: total = 20, run = 20, passed = 20, failed = 0
End Result: TEST PASSED
```
## Steps to reproduce
Following the instructions at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
one should be able to reproduce this failure on 'hansen', 'shiller', 'white', or 'ride'. Given that 'white' is on the SON and is pretty unloaded, one can reproduce this as described at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
with:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Tpetra=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -VV -R TpetraCore_MatrixMatrix_UnitTests_MPI_4
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3168Test KokkosKernels_sparse_openmp_MPI_1 still (randomly) timing out in in buil...2018-12-01T04:23:59ZJames WillenbringTest KokkosKernels_sparse_openmp_MPI_1 still (randomly) timing out in in build Trilinos-atdm-white-ride-gnu-debug-openmp on 'white'*Created by: bartlettroscoe*
@trilinos/kokkos-kernels, @srajama1 (Trilinos Linear Solver Product Lead)
## Next Action Status
PR #3173 disabled this test in this build. There have been no timeouts of this test on any system since...*Created by: bartlettroscoe*
@trilinos/kokkos-kernels, @srajama1 (Trilinos Linear Solver Product Lead)
## Next Action Status
PR #3173 disabled this test in this build. There have been no timeouts of this test on any system since 9/21/2018 except in the the build `Trilinos-atdm-mutrino-intel-opt-openmp-KNL` (which is being handled #3864).
## Description
As shown in [this query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&date=2018-07-22&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-white-ride-gnu-debug-openmp&field2=site&compare2=61&value2=white&field3=testname&compare3=61&value3=KokkosKernels_sparse_openmp_MPI_1&field4=buildstarttime&compare4=83&value4=2018-06-26) the test `KokkosKernels_sparse_openmp_MPI_1` timed out at 10 minutes in the build `Trilinos-atdm-white-ride-gnu-debug-openmp` on 'white'. But as one can see, the test was taking upwards of nearly 10 minutes to complete before in this build going back to 7/1/2018.
## Steps to Reproduce
Use the build name `gnu-debug-openmp` (**not** `cuda-debug`) on the machine 'white' and enable the package `KokkosKernels` (**not** `MueLu`) using the commands shown at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
NOTE: One cannot currently reproduce this on 'ride' because of the upgrade of 'white' but not 'ride'. See [TRIL-215](https://software-sandbox.sandia.gov/jira/browse/TRIL-215).
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3336KokkosKernels and KokkosContainers tests timing out on ATDM waterman builds s...2018-11-30T03:09:57ZJames WillenbringKokkosKernels and KokkosContainers tests timing out on ATDM waterman builds starting 8/16/2018*Created by: fryeguy52*
CC: @trilinos/kokkos-kernels , @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe
## Next Action Status
PR #3660 merged on 10/18/2018 disabled the test `KokkosContainers_UnitTest_Serial_MPI_1...*Created by: fryeguy52*
CC: @trilinos/kokkos-kernels , @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe
## Next Action Status
PR #3660 merged on 10/18/2018 disabled the test `KokkosContainers_UnitTest_Serial_MPI_1` in the build `Trilinos-atdm-waterman-cuda-9.2-debug` and all Kokkos and KokkosKernels tests are passing on 10/19/2018.
## Description
Some KokkosKernals test are timing out on the debug builds on waterman
The following tests are failing in the job `Trilinos-atdm-waterman-cuda-9.2-debug`:
* KokkosKernels_sparse_serial_MPI_1
* KokkosContainers_UnitTest_Serial_MPI_1
as shown [here](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-08-21&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-debug&field2=status&compare2=62&value2=passed&field3=status&compare3=62&value3=notrun)
And the following tests are failing in the job `Trilinos-atdm-waterman-gnu-debug-openmp`:
* KokkosKernels_sparse_serial_MPI_1
* KokkosContainers_UnitTest_Serial_MPI_1
* KokkosKernels_sparse_openmp_MPI_1
* KokkosKernels_graph_serial_MPI_1
as shown [here](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-08-20&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-gnu-debug-openmp&field2=status&compare2=62&value2=passed&field3=status&compare3=62&value3=notrun)
## Steps to Reproduce
One should be able to reproduce this failure on the machine <supported-atdm-machine> as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system <supported-atdm-system> are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Kokkos=ON \
$TRILINOS_DIR
$ make NP=20
$ bsub -x -Is -n 20 ctest -j20
```Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3438KokkosKernels_sparse_* tests are failing on ATDM cuda builds2018-11-30T03:09:58ZJames WillenbringKokkosKernels_sparse_* tests are failing on ATDM cuda builds*Created by: fryeguy52*
CC: @trilinos/kokkos-kernels , @kddevin (Trilinos Data Services Product Lead)
## Next Action Status
The test `KokkosKernels_sparse_serial_MPI_1` on 'waterman' has been passing without timing out in each '...*Created by: fryeguy52*
CC: @trilinos/kokkos-kernels , @kddevin (Trilinos Data Services Product Lead)
## Next Action Status
The test `KokkosKernels_sparse_serial_MPI_1` on 'waterman' has been passing without timing out in each 'debug' build on 'waterman' since 10/9/2018 as shown [here](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&date=2018-10-19&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-waterman-&field2=buildname&compare2=63&value2=debug&field3=testname&compare3=61&value3=KokkosKernels_sparse_serial_MPI_1&field4=buildstarttime&compare4=83&value4=2018-10-01).
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-09-12&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=Trilinos-atdm-&field2=testname&compare2=63&value2=KokkosKernels_sparse_&field3=status&compare3=62&value3=Passed) the tests:
* KokkosKernels_sparse_serial_MPI_1
* KokkosKernels_sparse_cuda_MPI_1
are failing in all the cuda builds on white, ride, hansen, and waterman:
* Trilinos-atdm-waterman-gnu-debug-openmp
* Trilinos-atdm-waterman-cuda-9.2-debug
* Trilinos-atdm-waterman-cuda-9.2-opt
* Trilinos-atdm-white-ride-cuda-9.2-opt
* Trilinos-atdm-white-ride-cuda-9.2-debug-pt
* Trilinos-atdm-white-ride-cuda-9.2-debug
* Trilinos-atdm-hansen-shiller-cuda-8.0-opt
* Trilinos-atdm-hansen-shiller-cuda-9.0-debug
* Trilinos-atdm-hansen-shiller-cuda-9.0-opt
[Here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-09-04&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=7&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=cuda&field3=buildname&compare3=64&value3=hansen&field4=buildname&compare4=64&value4=waterman&field5=testname&compare5=63&value5=KokkosKernels_sparse_cuda&field6=buildstarttime&compare6=84&value6=2018-09-13&field7=buildstarttime&compare7=83&value7=2018-09-05) you can see that these tests started failing on 9/7/2018
at the bottom of [this page](https://testing.sandia.gov/cdash/viewNotes.php?buildid=3912160) is a list of commits that were new on that day.
## Steps to Reproduce
One should be able to reproduce this failure as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_KokkosKernels=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3456PyTrilinos: Conflicting types in Teuchos_BLAS_wrapper.hpp2018-10-25T16:48:26ZJames WillenbringPyTrilinos: Conflicting types in Teuchos_BLAS_wrapper.hpp*Created by: wfspotz*
@trilinos/pytrilinos
@trilinos/teuchos
@trilinos/kokkos-kernels
## Expectations
I expect to build PyTrilinos without compilation errors
## Current Behavior
I have a wrapper file that `#include`s the he...*Created by: wfspotz*
@trilinos/pytrilinos
@trilinos/teuchos
@trilinos/kokkos-kernels
## Expectations
I expect to build PyTrilinos without compilation errors
## Current Behavior
I have a wrapper file that `#include`s the header `MLAPI_MultiVector.h` which gives the compilation error
/Development/Trilinos/packages/teuchos/numerics/src/Teuchos_BLAS_wrappers.hpp:173:13: error: conflicting types for 'daxpy_'
void PREFIX DAXPY_F77(const int* n, const double* alpha, const double x[], const int* incx, double y[], const int* incy);
^
/Development/Trilinos/packages/teuchos/numerics/src/Teuchos_BLAS_wrappers.hpp:78:21: note: expanded from macro 'DAXPY_F77'
#define DAXPY_F77 F77_BLAS_MANGLE(daxpy,DAXPY)
^
/Development/Trilinos/MPI/packages/teuchos/core/src/Teuchos_config.h:10:37: note: expanded from macro 'F77_BLAS_MANGLE'
#define F77_BLAS_MANGLE(name,NAME) name ## _
^
<scratch space>:23:1: note: expanded from here
daxpy_
^
/Development/Trilinos/packages/kokkos-kernels/src/impl/tpls/KokkosBlas1_axpby_tpl_spec_decl.hpp:49:17: note: previous declaration is here
extern "C" void daxpy_( const int* N, const double* alpha,
## Definition of Done
I can get the PyTrilinos package to build and all of the PyTrilinos tests to pass.
## Possible Solution
I'm not sure why this conflicting type declaration is occurring, but it is clearly related to macro expansion. Based on `git blame` of `Teuchos_BLAS_wrapper.h` I'm hoping either @jwillenbring or @hkthorn might have some idea what the problem could be. PyTrilinos can present some unique configuration issues.
## Steps to Reproduce
If it gets to this, I can help someone set up their environment to build PyTrilinos.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3493Tpetra::MultiVector unit tests fail for Scalar=complex<double>, likely due to...2018-10-15T15:56:48ZJames WillenbringTpetra::MultiVector unit tests fail for Scalar=complex<double>, likely due to kokkos-kernels changes*Created by: mhoemmen*
https://testing.sandia.gov/cdash/testDetails.php?test=55163267&build=3970561
```
The following tests FAILED:
1. MultiVector_double_default_local_ordinal_type_default_global_ordinal_type_Kokkos_Compat_Kokk...*Created by: mhoemmen*
https://testing.sandia.gov/cdash/testDetails.php?test=55163267&build=3970561
```
The following tests FAILED:
1. MultiVector_double_default_local_ordinal_type_default_global_ordinal_type_Kokkos_Compat_KokkosSerialWrapperNode_ComplexDotOneColumn_UnitTest ...
56. MultiVector_int_int_std_complex0double0_Kokkos_Compat_KokkosSerialWrapperNode_NonContigView_UnitTest ...
```
Further inspection shows large relative errors, so this isn't just an issue of needing to adjust the tolerance for complex numbers. When I run the tests locally on a Mac, I get a segfault.
I'm curious if this is related to the recent kokkos-kernels update.
@trilinos/tpetra
## Motivation and Context
Salinas depends on Tpetra with complex Scalar types.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3574Ifpack2 + Kokkos + complex float : compilation error2019-03-08T14:28:23ZJames WillenbringIfpack2 + Kokkos + complex float : compilation error*Created by: davydden*
## Expectations
Trilinos `trilinos-release-12-14-branch` builds with ETI and complex and float types
## Current Behavior
```
8051 In file included from /Users/davydden/spack/var/spack/stage/trilinos-12...*Created by: davydden*
## Expectations
Trilinos `trilinos-release-12-14-branch` builds with ETI and complex and float types
## Current Behavior
```
8051 In file included from /Users/davydden/spack/var/spack/stage/trilinos-12.14-hpmbucm6pphwdcf3p6hhqcac2cm5qts6/Trilinos/spack-build/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_Serial.cpp:50:
8052 In file included from /Users/davydden/spack/var/spack/stage/trilinos-12.14-hpmbucm6pphwdcf3p6hhqcac2cm5qts6/Trilinos/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_def.hpp:52:
>> 8053 /Users/davydden/spack/var/spack/stage/trilinos-12.14-hpmbucm6pphwdcf3p6hhqcac2cm5qts6/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Util.hpp:195:7: error: static_assert failed "KokkosKernels:: Invalid SIMD<> type."
8054 static_assert( std::is_same<T,bool>::value ||
8055 ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8056 /Users/davydden/spack/var/spack/stage/trilinos-12.14-hpmbucm6pphwdcf3p6hhqcac2cm5qts6/Trilinos/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_impl.hpp:1568:28: note: in instantiation of template class 'KokkosBatched::Experimental::SIMD<Kokkos::complex<floa
t> >' requested here
8057 B.assign_data( &vector_values(i0+1,0,0) );
8058 ^
8059 /Users/davydden/spack/var/spack/stage/trilinos-12.14-hpmbucm6pphwdcf3p6hhqcac2cm5qts6/Trilinos/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_impl.hpp:1651:9: note: in instantiation of member function 'Ifpack2::BlockTriDiContainerDetails::ExtractAndFactori
zeTridiags<Tpetra::Classes::RowMatrix<std::__1::complex<float>, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::factorize' requested here
```
## Steps to Reproduce
configure and build with:
```
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-DTpetra_INST_DOUBLE:BOOL=ON
-DTpetra_INST_INT_LONG:BOOL=ON
-DTpetra_INST_COMPLEX_DOUBLE=ON
-DTpetra_INST_COMPLEX_FLOAT=ON
-DTpetra_INST_FLOAT=ON
-DTpetra_INST_SERIAL=ON
-DTeuchos_ENABLE_COMPLEX=ON
-DTeuchos_ENABLE_FLOAT=ON
```
## Your Environment
macOS Mojave
Apple Clang 10.0.0
gfortran 8.2.0
## Additional Information
full config/build logs:
[logs.zip](https://github.com/trilinos/Trilinos/files/2454254/logs.zip)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3664KokkosKernels: missing doxygen documentation2018-10-18T21:24:43ZJames WillenbringKokkosKernels: missing doxygen documentation*Created by: jhux2*
KokkosKernels doesn't have any links to its Doxygen documentation on trilinos.org, at least not that I could find. I think it would be useful to have doxygen available, similarly to what other Trilinos packages prov...*Created by: jhux2*
KokkosKernels doesn't have any links to its Doxygen documentation on trilinos.org, at least not that I could find. I think it would be useful to have doxygen available, similarly to what other Trilinos packages provide, e.g., [Tpetra](https://trilinos.org/docs/dev/packages/tpetra/doc/html/index.html).https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3773GCC 4.9.3-SERIAL fails in nightly clean test2018-11-07T15:48:47ZJames WillenbringGCC 4.9.3-SERIAL fails in nightly clean test*Created by: william76*
@trilinos/kokkos-kernels
An error showed up in the [gcc 4.9.3 SERIAL build on 10/25][1], which means the issue likely was introduced sometime on 10/24.
The errors I'm seeing are:
```
libkokkoskernels_g...*Created by: william76*
@trilinos/kokkos-kernels
An error showed up in the [gcc 4.9.3 SERIAL build on 10/25][1], which means the issue likely was introduced sometime on 10/24.
The errors I'm seeing are:
```
libkokkoskernels_gtest.so.12.13: undefined reference to `pthread_key_create'
libkokkoskernels_gtest.so.12.13: undefined reference to `pthread_getspecific'
libkokkoskernels_gtest.so.12.13: undefined reference to `pthread_key_delete'
libkokkoskernels_gtest.so.12.13: undefined reference to `pthread_setspecific'
collect2: error: ld returned 1 exit status
```
I was out of town for the past week... @jwillenbring do you know if anything @trilinos/framework related changed on our configurations that would cause an undefined reference to things like `pthread_key_create`?
@trilinos/kokkos-kernels was anything merged into Trilinos that is using any new features from pthreads that might require a newer version of pthreads on the testing machines?
I also found [this StackOverflow issue][2] which was related to someone linking libgtest incorrectly... were any changes that might relate to gtest stuff in KokkosKernels pushed into Trilinos on 10/24?
[1]: https://testing.sandia.gov/cdash/index.php?project=Trilinos&date=2018-10-25&filtercount=2&showfilters=1&filtercombine=or&field1=groupname&compare1=63&value1=Clean&field2=groupname&compare2=63&value2=Continuous
[2]: https://stackoverflow.com/questions/21116622/undefined-reference-to-pthread-key-create-linker-errorhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3864KokkosKernels tests randomly timing out on ATDM mutrino KNL build2019-04-02T19:15:07ZJames WillenbringKokkosKernels tests randomly timing out on ATDM mutrino KNL build*Created by: fryeguy52*
CC: @trilinos/<package-name>, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.ph...*Created by: fryeguy52*
CC: @trilinos/<package-name>, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=65&value2=KokkosKernels_&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00&field6=details&compare6=61&value6=Completed%20%28Timeout%29) the tests:
* [KokkosKernels_sparse_serial_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_sparse_serial_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00:00:00&field5=buildstarttime&compare5=83&value5=2018-10-14T00:00:00)
* [KokkosKernels_sparse_openmp_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_sparse_openmp_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
* [KokkosKernels_graph_serial_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_graph_serial_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
* [KokkosKernels_graph_openmp_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_graph_openmp_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00:00:00&field5=buildstarttime&compare5=83&value5=2018-10-14T00:00:00)
* [KokkosKernels_common_serial_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_common_serial_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
* [KokkosKernels_common_openmp_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_common_openmp_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00:00:00&field5=buildstarttime&compare5=83&value5=2018-10-14T00:00:00)
* [KokkosKernels_blas_serial_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_blas_serial_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
* [KokkosKernels_blas_openmp_MPI_1](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=61&value2=KokkosKernels_blas_openmp_MPI_1&field3=site&compare3=61&value3=mutrino&field4=buildstarttime&compare4=84&value4=2018-11-13T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-10-14T00%3A00%3A00)
are randomly timing out in the build:
* Trilinos-atdm-mutrino-intel-opt-openmp-KNL
Links above are to a 30 day history of each test. The `KokkosKernels_*_serial_MPI_1` tests are nearly 3 times as likely to timeout over the last 30 days vs the `KokkosKernels_*_openmp_MPI_1` tests. On average 3 of the 8 tests fail and there has been only one day in that last 30 when all passed.
## Current Status on CDash
The current status of these tests/builds for the current testing day can be found at:
[Current test status](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-mutrino-intel-opt-openmp-KNL&field2=testname&compare2=65&value2=KokkosKernels_&field3=buildstarttime&compare3=83&value3=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on the machine mutrino as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system mutrino are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#<mutrino
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-mutrino-intel-opt-openmp-KNL
$ cmake \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_KokkosKernels=ON \
$TRILINOS_DIR
$ make -j16
$ salloc -N 1 -p standard -J Trilinos-atdm-mutrino-intel-opt-openmp-KNL ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3912MueLu: spgemm OpenMP bug, now fixed2018-11-21T16:51:55ZJames WillenbringMueLu: spgemm OpenMP bug, now fixed*Created by: mhoemmen*
@trilinos/muelu @trilinos/tpetra
Mirror of the following kokkos-kernels issue, now fixed: https://github.com/kokkos/kokkos-kernels/issues/345*Created by: mhoemmen*
@trilinos/muelu @trilinos/tpetra
Mirror of the following kokkos-kernels issue, now fixed: https://github.com/kokkos/kokkos-kernels/issues/345https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4237kokkos-kernels: now requires fortran to configure2019-01-23T16:05:59ZJames Willenbringkokkos-kernels: now requires fortran to configure*Created by: rppawlo*
A change to kokkos-kernels last week now makes the package require fortran. This has brought down testing suite for Drekar that use the type2 stack but does not enable fortran. We are trying to avoid requiring user...*Created by: rppawlo*
A change to kokkos-kernels last week now makes the package require fortran. This has brought down testing suite for Drekar that use the type2 stack but does not enable fortran. We are trying to avoid requiring users to have to use the kitware hacked ninja to get fortran support. Was this dependency change intended? https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4242EMPIRE ATDM Trilinos builds broken that have Fortran disabled starting 1/22/20182019-02-01T18:19:35ZJames WillenbringEMPIRE ATDM Trilinos builds broken that have Fortran disabled starting 1/22/2018*Created by: bartlettroscoe*
CC: @trilinos/framework, @jwillenbring (Trilinos Framework Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
PR #4241 merged to 'develop' on 1/22/2019 which reverted PR #4228 fixed the...*Created by: bartlettroscoe*
CC: @trilinos/framework, @jwillenbring (Trilinos Framework Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
PR #4241 merged to 'develop' on 1/22/2019 which reverted PR #4228 fixed the Fortran builds.
## Description
As shown in [this query](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&date=2019-01-22&filtercount=1&showfilters=1&field1=buildname&compare1=65&value1=Trilinos-atdm-) all of the ATDM Trilinos builds that have Fortran disabled failed to configure today.
## Current Status on CDash
The current status of the ATDM Trilinos builds is found on CDash [here](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&filtercount=1&showfilters=1&field1=buildname&compare1=65&value1=Trilinos-atdm-).
## Steps to Reproduce
One should be able to reproduce this failure on any of the machines and builds shown above as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4394KokkosKernels: KokkosKernels_blas_serial_MPI_1 tests are unstable2019-02-21T19:32:59ZJames WillenbringKokkosKernels: KokkosKernels_blas_serial_MPI_1 tests are unstable*Created by: william76*
@trilinos/kokkos-kernels
The `KokkosKernels_blas_serial_MPI_1` test has been having some [intermittent failures the past few days][1]. Can someone on the @trilinos/kokkos-kernels have a look at this and see w...*Created by: william76*
@trilinos/kokkos-kernels
The `KokkosKernels_blas_serial_MPI_1` test has been having some [intermittent failures the past few days][1]. Can someone on the @trilinos/kokkos-kernels have a look at this and see what's happening?
The [CDash output][2] isn't terribly exciting:
```
Note: Google Test filter = -serial.gemm_double
[==========] Running 89 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 89 tests from serial
[ RUN ] serial.gemm_complex_double
```
@ndellingwood when you have a chance can you have a look at this? I've been seeing the error on the develop->master pull requests. It popped up after the merge of new kokkos stuff last week, but it seems to be intermittent so we don't have a ton of data points just yet.
[1]: https://testing.sandia.gov/cdash/testSummary.php?project=1&name=KokkosKernels_blas_serial_MPI_1&date=2019-02-13
[2]: https://testing-vm.sandia.gov/cdash/testDetails.php?test=65847315&build=4549908https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4571Long file names in KokkosKernels have broken git clone of Trilinos on Windows2019-03-07T19:52:38ZJames WillenbringLong file names in KokkosKernels have broken git clone of Trilinos on Windows*Created by: bartlettroscoe*
CC: @trilinos/framework, @trilinos/kokkos-kernels
## Description
It appears that the Trilinos git repo has gotten into a state where it can't even be cloned on Windows systems with git.
The CASL PH...*Created by: bartlettroscoe*
CC: @trilinos/framework, @trilinos/kokkos-kernels
## Description
It appears that the Trilinos git repo has gotten into a state where it can't even be cloned on Windows systems with git.
The CASL PHI INF lead @lefebvrera reported the following error when trying to clone the current Trilinos git repo just a short time ago:
```
G:\raq\scale_dev\dev>git clone https://github.com/trilinos/Trilinos.git Trilinos
Cloning into 'Trilinos'...
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 957120 (delta 17), reused 14 (delta 7), pack-reused 957073
Receiving objects: 100% (957120/957120), 562.05 MiB | 26.56 MiB/s, done.
Resolving deltas: 100% (771153/771153), done.
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_numeric/KokkosSparse_gauss_seidel_numeric_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
Checking out files: 100% (56578/56578), done.
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4630Tpetra::MultiVector not using cuBLAS for GEMM, making Belos slow2019-04-03T23:52:52ZJames WillenbringTpetra::MultiVector not using cuBLAS for GEMM, making Belos slow*Created by: mhoemmen*
@trilinos/tpetra @trilinos/belos @trilinos/kokkos-kernels @vbrunini
This is a Trilinos mirror of the following kokkos-kernels issue: https://github.com/kokkos/kokkos-kernels/issues/397 . Once we verify the pr...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/belos @trilinos/kokkos-kernels @vbrunini
This is a Trilinos mirror of the following kokkos-kernels issue: https://github.com/kokkos/kokkos-kernels/issues/397 . Once we verify the proposed fix, be sure to patch Trilinos and submit a fix to kokkos-kernels.
## Related Issues
* Related to https://github.com/kokkos/kokkos-kernels/issues/397, https://github.com/kokkos/kokkos-kernels/issues/396, https://github.com/trilinos/Trilinos/issues/4626
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4829KokkosKernels broke Albany build on Ride2019-04-10T14:34:42ZJames WillenbringKokkosKernels broke Albany build on Ride*Created by: ikalash*
It appears some changes to KokkosKernels broke Albany's Ride build over the weekend. Here is the error:
```
58%] Building C object packages/aztecoo/src/CMakeFiles/aztecoo.dir/az_gsumd_puma.c.o ...*Created by: ikalash*
It appears some changes to KokkosKernels broke Albany's Ride build over the weekend. Here is the error:
```
58%] Building C object packages/aztecoo/src/CMakeFiles/aztecoo.dir/az_gsumd_puma.c.o /.../repos/Trilinos/packages/kokkos-kernels/src/sparse/impl/KokkosSparse_spgemm_impl_color.hpp(516): error: namespace "KokkosGraph::Experimental" has no member "d2_graph_color"
--
```
http://cdash.sandia.gov/CDash-2-3-0/viewBuildError.php?buildid=83384
Could someone please have a look?
@trilinos/kokkos-kernels https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4847KokkosKernels_graph tests timing out in ATDM builds2019-05-01T19:20:49ZJames WillenbringKokkosKernels_graph tests timing out in ATDM builds*Created by: fryeguy52*
CC: @trilinos/kokkoskernels, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e...*Created by: fryeguy52*
CC: @trilinos/kokkoskernels, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=65&value2=KokkosKernels_graph_&field3=testname&compare3=66&value3=_MPI_1&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=83&value5=2019-04-07&field6=buildstarttime&compare6=84&value6=2019-04-10) the tests:
* KokkosKernels_graph_openmp_MPI_1
* KokkosKernels_graph_serial_MPI_1
started timing out on 2019-04-07 in the builds:
* Trilinos-atdm-hansen-shiller-gnu-debug-serial
* Trilinos-atdm-waterman-cuda-9.2-debug
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug
* Trilinos-atdm-white-ride-gnu-7.2.0-openmp-debug
## Current Status on CDash
Failures for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=65&value2=KokkosKernels_graph_&field3=testname&compare3=66&value3=_MPI_1&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=83&value5=yesterday&field6=buildstarttime&compare6=84&value6=today)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-gnu-7.2.0-openmp-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_KokkosKernels=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5014KokkosKernels: KokkosKernels_sparse_ tests timing out on ATDM complex build2019-04-25T18:53:43ZJames WillenbringKokkosKernels: KokkosKernels_sparse_ tests timing out on ATDM complex build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/kokkoskernels, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/kokkoskernels, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug&field2=testname&compare2=65&value2=KokkosKernels_sparse_&field3=buildstarttime&compare3=84&value3=2019-04-25T00%3A00%3A00&field4=buildstarttime&compare4=83&value4=2019-03-26T00%3A00%3A00) the tests:
* KokkosKernels_sparse_openmp_MPI_1
* KokkosKernels_sparse_serial_MPI_1
are failing timing out in the build:
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug
## Current Status on CDash
[The status of these tests for the current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug&field2=testname&compare2=65&value2=KokkosKernels_sparse_&field3=buildstarttime&compare3=84&value3=today&field4=buildstarttime&compare4=83&value4=2%20days%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_KokkosKernels=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Initial cleanup of new ATDM builds of Trilinos