Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2016-02-05T14:29:36Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/132Tpetra: Port Andrey's getLocalRowView benchmark from MueLu to Tpetra2016-02-05T14:29:36ZJames WillenbringTpetra: Port Andrey's getLocalRowView benchmark from MueLu to Tpetra*Created by: mhoemmen*
@trilinos/tpetra
@aprokop wrote a benchmark in MueLu that compares performance of getting local row views with Xpetra::CrsMatrix, Epetra_CrsMatrix, Tpetra::CrsMatrix, and KokkosSparse::CrsMatrix. See Issue #118 ...*Created by: mhoemmen*
@trilinos/tpetra
@aprokop wrote a benchmark in MueLu that compares performance of getting local row views with Xpetra::CrsMatrix, Epetra_CrsMatrix, Tpetra::CrsMatrix, and KokkosSparse::CrsMatrix. See Issue #118 for discussion. The problem is that the benchmark lives in MueLu, and requires enabling experimental build options which are off by default (see Issue #128, now closed, thanks Andrey!). This motivates me to port the benchmark to Tpetra. The main point of the benchmark will be to compare Tpetra vs. Epetra vs. "raw" Kokkos performance, so I will leave Xpetra out.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/131Ifpack2: More modular software architecture (esp. triangular solves & ILU)2016-08-24T22:40:54ZJames WillenbringIfpack2: More modular software architecture (esp. triangular solves & ILU)*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/amesos2
This relates to the recent addition of Basker's thread-parallel incomplete LU factorization (ILU) to `Ifpack2::RILUK`. I really appreciate all the work that people are doing ...*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/amesos2
This relates to the recent addition of Basker's thread-parallel incomplete LU factorization (ILU) to `Ifpack2::RILUK`. I really appreciate all the work that people are doing in Ifpack2 to improve support for thread parallelism.
What I see happening is that a lot of us are patching what amount to entirely new preconditioners, into existing classes. I've been doing that with BlockCrsMatrix, and others have been doing that with other classes. I think that's OK for prototyping, but I worry that eventually it will make the classes super complicated. It may also add complication time.
It would be good to remember LinearSolver and LinearSolverFactory at this point. We already have plenty of different ways to achieve abstraction. For example, maybe it's appropriate to treat thread-parallel sparse triangular solves as an "inner solver." If we don't want to expose a new solver in Ifpack2's public interface, we can put it in the `Ifpack2::Details` namespace. I'm not sure if we have to worry about it for Basker, but I want people to keep this in mind going forward.
Thanks all!
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/135Failed build with trilinos-release-12-62016-03-11T00:31:50ZJames WillenbringFailed build with trilinos-release-12-6*Created by: BarrySmith*
I'm unable to build the trilinos-release-12-6-branch
Obtaining errors of the form
../../../../../tpetra/core/src/libtpetra.so.12.6: undefined reference to `KokkosBlas::Impl::Update<Kokkos::View<int con\
st**...*Created by: BarrySmith*
I'm unable to build the trilinos-release-12-6-branch
Obtaining errors of the form
../../../../../tpetra/core/src/libtpetra.so.12.6: undefined reference to `KokkosBlas::Impl::Update<Kokkos::View<int con\
st**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u>, Kokkos::Impl::Vi\
ewDefault>, Kokkos::View<int const**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::Me\
moryTraits<1u>, Kokkos::Impl::ViewDefault>, Kokkos::View<int**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokk\
os::HostSpace>, Kokkos::MemoryTraits<1u>, Kokkos::Impl::ViewDefault>, 2>::update(int const&, Kokkos::View<int const**, \
Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u>, Kokkos::Impl::ViewDefa\
ult> const&, int const&, Kokkos::View<int const**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace\
> , Kokkos::MemoryTraits<1u>, Kokkos::Impl::ViewDefault> const&, int const&, Kokkos::View<int**, Kokkos::LayoutLeft, Kok\
> kos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u>, Kokkos::Impl::ViewDefault> const&)'
> ../../../../../tpetra/core/src/libtpetra.so.12.6: undefined reference to `KokkosSparse::Impl::SPMV_MV<double const*, Ko\
> kkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u>, double const, int, Kokko\
> s::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u>, unsigned long, double const**, Kokkos::LayoutLe\
> ft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<3u>, double const*, Kokkos::LayoutLeft, Kok\
> kos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u>, double**, Kokkos::LayoutLeft, Kokkos::Device<\
> Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >::spmv_mv(char const_, double const&, KokkosSparse::CrsMa\
> trix<double const, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u>, unsigned long> con\
> st&, Kokkos::View<double const__, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::Memory\
> Traits<3u>, Kokkos::Impl::ViewDefault> const&, double const&, Kokkos::View<double_*, Kokkos::LayoutLeft, Kokkos::Device\
> <Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u>, Kokkos::Impl::ViewDefault> const&)'
Full log including configure options and machine configuration can be found at
ftp://ftp.mcs.anl.gov/pub/petsc/trilinos-release-12-6-failure
The same configure options have worked successfully with Trilinos 'master' branch.
@jwillenbring
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/138Add Trilinos Developer Instructions for GitHub Issues and waffle.io2016-02-14T05:24:25ZJames WillenbringAdd Trilinos Developer Instructions for GitHub Issues and waffle.io*Created by: jwillenbring*
@maherou has asked me to write up some instructions for using GitHub Issues and waffle.io. I am going to do this on a Trilinos GitHub wiki page. I am going to use the IDEAS Jira instructions as a guide.
*Created by: jwillenbring*
@maherou has asked me to write up some instructions for using GitHub Issues and waffle.io. I am going to do this on a Trilinos GitHub wiki page. I am going to use the IDEAS Jira instructions as a guide.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/137ShyLU Build Error on KNL with Intel 16.1 (missing -lcamd?)2016-03-08T19:53:32ZJames WillenbringShyLU Build Error on KNL with Intel 16.1 (missing -lcamd?)*Created by: nmhamster*
I am getting the following build error on a KNL build but I cannot see -lcamd in the link line. I don't specify it in the cmake configure but then the configure does successfully complete. Am I missing something ...*Created by: nmhamster*
I am getting the following build error on a KNL build but I cannot see -lcamd in the link line. I don't specify it in the cmake configure but then the configure does successfully complete. Am I missing something obvious or is the build system missing a needed library?
```
[ 95%] Linking CXX executable symbolic_factor_serial.exe
cd /home/sdhammo/git/trilinos-github-repo/build-bddc/packages/shylu/tacho/example && /home/projects/x86-64-haswell/cmake/3.3.2/bin/cmake -E cmake_link_script CMakeFiles/symbolic_factor_serial.dir/link.txt --verbose=1
/home/projects/x86-64-knl/openmpi/1.10.1/intel/16.1.150/bin/mpicxx -O3 -g -xMIC-AVX512 -fopenmp -std=c++11 -qopenmp -L/home/projects/x86-64-knl/numa/2.0.10-static/lib -lnuma CMakeFiles/symbolic_factor_serial.dir/example_symbolic_factor_serial.cpp.o -o symbolic_factor_serial.exe -rdynamic ../src/libshylutacho.a ../../../teuchos/kokkoscomm/src/libteuchoskokkoscomm.a ../../../teuchos/kokkoscompat/src/libteuchoskokkoscompat.a ../../../teuchos/remainder/src/libteuchosremainder.a ../../../teuchos/numerics/src/libteuchosnumerics.a ../../../teuchos/comm/src/libteuchoscomm.a ../../../teuchos/parameterlist/src/libteuchosparameterlist.a ../../../teuchos/core/src/libteuchoscore.a ../../../kokkos/algorithms/src/libkokkosalgorithms.a ../../../kokkos/containers/src/libkokkoscontainers.a ../../../kokkos/core/src/libkokkoscore.a -Wl,-Bstatic -lptscotch -lptscotcherr -lscotch -lscotcherr -Wl,-Bdynamic -mkl -mkl -lhwloc -Wl,-Bstatic -lcholmod -lamd -lcolamd -lsuitesparseconfig -lptscotch -lptscotcherr -lscotch -lscotcherr -Wl,-Bdynamic -lhwloc
CMakeFiles/symbolic_factor_serial.dir/example_symbolic_factor_serial.cpp.o: In function `int Tacho::exampleSymbolicFactor<double, int, int, Kokkos::Serial, void>(std::string, int, int, int, int, int, bool, bool, bool, bool)':
/home/sdhammo/git/trilinos-github-repo/packages/shylu/tacho/src/graph_helper_camd.hpp:85: undefined reference to `camd_defaults'
/home/sdhammo/git/trilinos-github-repo/packages/shylu/tacho/src/graph_helper_camd.hpp:86: undefined reference to `camd_control'
/home/sdhammo/git/trilinos-github-repo/packages/shylu/tacho/src/graph_helper_camd.hpp:93: undefined reference to `camd_order'
make[2]: *** [packages/shylu/tacho/example/symbolic_factor_serial.exe] Error 1
make[2]: Leaving directory `/home/sdhammo/git/trilinos-github-repo/build-bddc'
make[1]: *** [packages/shylu/tacho/example/CMakeFiles/symbolic_factor_serial.dir/all] Error 2
make[1]: Leaving directory `/home/sdhammo/git/trilinos-github-repo/build-bddc'
make: *** [all] Error 2
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/139cmake rerun everytime I run make on chama2016-02-15T17:45:18ZJames Willenbringcmake rerun everytime I run make on chama*Created by: bathmatt*
For some reason Trilinos reconfigures every time I type make, adds about 5 minutes to the make cycle. Doesn't look like it is a filesystem time issue..
[mbetten@chama-login8 build]$ date
Thu Feb 11 14:19:31 MST ...*Created by: bathmatt*
For some reason Trilinos reconfigures every time I type make, adds about 5 minutes to the make cycle. Doesn't look like it is a filesystem time issue..
[mbetten@chama-login8 build]$ date
Thu Feb 11 14:19:31 MST 2016
[mbetten@chama-login8 build]$ touch a
[mbetten@chama-login8 build]$ date
Thu Feb 11 14:19:33 MST 2016
[mbetten@chama-login8 build]$ ls -l a
-rw------- 1 mbetten mbetten 0 Feb 11 14:19 a
[mbetten@chama-login8 build]$
[mbetten@chama-login8 build]$ make VERBOSE=1
/home/mbetten/cmake-2.8.11.2-Linux-i386/bin/cmake -H/ascldap/users/mbetten/work/Drekar/Trilinos -B/ascldap/users/mbetten/work/Drekar/build --check-build-system CMakeFiles/Makefile.cmake 0
Re-run cmake: build system dependency is missing
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/142Framework: Create GitHub tag for Thyra2016-02-15T17:56:10ZJames WillenbringFramework: Create GitHub tag for Thyra*Created by: mhoemmen*
Please :-) Issue #3 needs this.
*Created by: mhoemmen*
Please :-) Issue #3 needs this.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/143Tpetra: Debug build throws exception all the time.2016-02-16T16:58:32ZJames WillenbringTpetra: Debug build throws exception all the time.*Created by: lxmota*
Hi,
I am unable to use a debug version of Trilinos. When linked into Albany and run, the following exception is thrown regardless of the input file:
p=0: **\* Caught standard std::exception of type 'std::runtime_e...*Created by: lxmota*
Hi,
I am unable to use a debug version of Trilinos. When linked into Albany and run, the following exception is thrown regardless of the input file:
p=0: **\* Caught standard std::exception of type 'std::runtime_error' :
/home/amota/LCM/Trilinos/packages/tpetra/core/src/Tpetra_MultiVector_def.hpp:3098:
Throw number = 1
Throw test that evaluated to true: (this->vectorIndexOutOfRange (j))
Tpetra::MultiVector<double,int,int,Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>>::getVector(NonConst): Input index j (== 1) exceeds valid range [0, 1 - 1].
This exception is only thrown when Trilinos is compiled in debug mode. Any help would be appreciated.
Thanks,
Alejandro
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/145Anasazi tests fail with checkin script and secondary-stable code enabled2018-07-03T16:50:45ZJames WillenbringAnasazi tests fail with checkin script and secondary-stable code enabled*Created by: kddevin*
@trilinos/anasazi
I am trying to run the checkin script with secondary-stable code enabled. The following anasazi tests fail. All my code changes are in zoltan and zoltan2, so I don't suspect them as the problem...*Created by: kddevin*
@trilinos/anasazi
I am trying to run the checkin script with secondary-stable code enabled. The following anasazi tests fail. All my code changes are in zoltan and zoltan2, so I don't suspect them as the problem.
595 - Anasazi_Epetra_ModalSolversTester_MPI_4 (Failed)
609 - Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4 (Failed)
610 - Anasazi_Epetra_OrthoManagerGenTester_1_MPI_4 (Failed)
Error messages for 595:
ERROR: V*Q failed.
orthonorm error of applyHouse: 0.308401
ERROR: applyHouse failed.
error(VQ - house(V,H,tau): 3.5943e-16
Error messages for 609:
**\* Caught standard std::exception of type 'std::runtime_error' :
/home/kddevin/code/Trilinos/packages/anasazi/epetra/test/OrthoManager/cxx_gentest.cpp:261:
Throw number = 1
Throw test that evaluated to true: err > TOL
New X1 did not meet tolerance: orthog(X1,Y2) == 1.07011
Error messages for 610:
**\* Caught standard std::exception of type 'std::runtime_error' :
/home/kddevin/code/Trilinos/packages/anasazi/epetra/test/OrthoManager/cxx_gentest.cpp:261:
Throw number = 1
Throw test that evaluated to true: err > TOL
New X1 did not meet tolerance: orthog(X1,Y2) == 1.14092
My custom config options for these tests are listed below. Is there some other CMake magic that I need to include (or that could be made the default in the checkin script)?
Thanks for your help!
-D Trilinos_ENABLE_SECONDARY_STABLE_CODE=ON
-D Trilinos_ENABLE_Epetra:BOOL=ON
-D Trilinos_ENABLE_Galeri:BOOL=ON
-D Trilinos_ENABLE_Pamgen:BOOL=ON
-D Zoltan_ENABLE_EXAMPLES:BOOL=ON
-D Zoltan_ENABLE_TESTS:BOOL=ON
-D Zoltan2_ENABLE_EXAMPLES:BOOL=ON
-D Zoltan2_ENABLE_TESTS:BOOL=ON
-D Zoltan2_ENABLE_Experimental:BOOL=ON
-D Zoltan_ENABLE_Scotch:BOOL=ON
-D Zoltan2_ENABLE_Scotch:BOOL=ON
-D Scotch_LIBRARY_DIRS:FILEPATH="/home/kddevin/code/Scotch/scotch_6.0.3/32bit_openmpi/lib"
-D Scotch_INCLUDE_DIRS:FILEPATH="/home/kddevin/code/Scotch/scotch_6.0.3/32bit_openmpi//include"
-D Zoltan_ENABLE_ParMETIS:BOOL=ON
-D Zoltan2_ENABLE_ParMETIS:BOOL=ON
-D ParMETIS_LIBRARY_DIRS:FILEPATH="/home/kddevin/code/ParMETIS/ParMETIS-4.0.3/32bit_openmpi"
-D ParMETIS_INCLUDE_DIRS:FILEPATH="/home/kddevin/code/ParMETIS/ParMETIS-4.0.3/32bit_openmpi"
-D Teuchos_ENABLE_STACKTRACE=OFF
-D MPI_BIN_DIR:PATH=/usr/lib64/openmpi/bin
-D TPL_ENABLE_MPI:BOOL=ON
-D MPI_EXEC_MAX_NUMPROCS:STRING=12
-D TPL_ENABLE_Boost=OFF
-D TPL_ENABLE_Netcdf=OFF
-D TPL_ENABLE_BoostLib=OFF
-D Trilinos_ENABLE_SEACAS:BOOL=OFF
-D Trilinos_ENABLE_STK:BOOL=OFF
-D DART_TESTING_TIMEOUT:STRING=1300
-D Amesos2_ENABLE_KLU2=ON
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/146MueLu tests failing with checkin script and secondary-stable code enabled2016-12-14T20:54:21ZJames WillenbringMueLu tests failing with checkin script and secondary-stable code enabled*Created by: kddevin*
@trilinos/muelu
I am running the checkin script with secondary-stable code enabled. I see the following failures in MueLu. Since MueLu uses Zoltan and Zoltan2, I want to make sure that these errors are expected,...*Created by: kddevin*
@trilinos/muelu
I am running the checkin script with secondary-stable code enabled. I see the following failures in MueLu. Since MueLu uses Zoltan and Zoltan2, I want to make sure that these errors are expected, and not caused by my code changes. I list my custom configuration options at the bottom of this issue. Is there some other CMake magic that I need (and, if so, can it be made the default in the checkin script)?
```
1222 - MueLu_ParameterListInterpreterTpetra_MPI_1 (Failed)
1223 - MueLu_ParameterListInterpreterTpetra_MPI_4 (Failed)
```
Error message for 1222:
p=0: **\* Caught standard std::exception of type 'std::invalid_argument' :
/home/kddevin/code/Trilinos/packages/teuchos/remainder/src/Trilinos_Details_LinearSolverFactory.hpp:599:
Throw number = 209
Throw test that evaluated to true: factory.get () == NULL
Trilinos::Details::getLinearSolver: Package "Ifpack2" is valid, but it never registered a LinearSolverFactory for template parameters MV = Tpetra::MultiVector<double,int,long int,Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>>, OP = Tpetra::Operator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >, NormType = double. Trilinos_ENABLE_LINEAR_SOLVER_FACTORY_REGISTRATION = OFF.
Error message for 1223:
p=0: **\* Caught standard std::exception of type 'std::invalid_argument' :
/home/kddevin/code/Trilinos/packages/teuchos/remainder/src/Trilinos_Details_LinearSolverFactory.hpp:599:
Throw number = 132
Throw test that evaluated to true: factory.get () == NULL
Trilinos::Details::getLinearSolver: Package "Ifpack2" is valid, but it never registered a LinearSolverFactory for template parameters MV = Tpetra::MultiVector<double,int,long int,Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>>, OP = Tpetra::Operator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >, NormType = double. Trilinos_ENABLE_LINEAR_SOLVER_FACTORY_REGISTRATION = OFF.
My configuration options:
-D Trilinos_ENABLE_SECONDARY_STABLE_CODE=ON
-D Trilinos_ENABLE_Epetra:BOOL=ON
-D Trilinos_ENABLE_Galeri:BOOL=ON
-D Trilinos_ENABLE_Pamgen:BOOL=ON
-D Zoltan_ENABLE_EXAMPLES:BOOL=ON
-D Zoltan_ENABLE_TESTS:BOOL=ON
-D Zoltan2_ENABLE_EXAMPLES:BOOL=ON
-D Zoltan2_ENABLE_TESTS:BOOL=ON
-D Zoltan2_ENABLE_Experimental:BOOL=ON
-D Zoltan_ENABLE_Scotch:BOOL=ON
-D Zoltan2_ENABLE_Scotch:BOOL=ON
-D Scotch_LIBRARY_DIRS:FILEPATH="/home/kddevin/code/Scotch/scotch_6.0.3/32bit_openmpi/lib"
-D Scotch_INCLUDE_DIRS:FILEPATH="/home/kddevin/code/Scotch/scotch_6.0.3/32bit_openmpi//include"
-D Zoltan_ENABLE_ParMETIS:BOOL=ON
-D Zoltan2_ENABLE_ParMETIS:BOOL=ON
-D ParMETIS_LIBRARY_DIRS:FILEPATH="/home/kddevin/code/ParMETIS/ParMETIS-4.0.3/32bit_openmpi"
-D ParMETIS_INCLUDE_DIRS:FILEPATH="/home/kddevin/code/ParMETIS/ParMETIS-4.0.3/32bit_openmpi"
-D Teuchos_ENABLE_STACKTRACE=OFF
-D MPI_BIN_DIR:PATH=/usr/lib64/openmpi/bin
-D TPL_ENABLE_MPI:BOOL=ON
-D MPI_EXEC_MAX_NUMPROCS:STRING=12
-D TPL_ENABLE_Boost=OFF
-D TPL_ENABLE_Netcdf=OFF
-D TPL_ENABLE_BoostLib=OFF
-D Trilinos_ENABLE_SEACAS:BOOL=OFF
-D Trilinos_ENABLE_STK:BOOL=OFF
-D DART_TESTING_TIMEOUT:STRING=1300
-D Amesos2_ENABLE_KLU2=ON
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/147KokkosKernels,Tpetra: Plug in TPLs (MKL, cuSPARSE, ...) for sparse matrix-vec...2016-09-19T21:33:01ZJames WillenbringKokkosKernels,Tpetra: Plug in TPLs (MKL, cuSPARSE, ...) for sparse matrix-vector multiply*Created by: mhoemmen*
Please refer to the three attached patches. Their file names should end in .patch, but I had to rename them to .txt so Github would take them.
NOTE: If you want to use MKL for sparse matrix-vector multiply in ...*Created by: mhoemmen*
Please refer to the three attached patches. Their file names should end in .patch, but I had to rename them to .txt so Github would take them.
NOTE: If you want to use MKL for sparse matrix-vector multiply in Tpetra, you must do the following. First, apply the attached patches in order. Second, enable the MKL TPL in Trilinos. This is _not_ the same thing as building against MKL for BLAS and LAPACK. Here is an example of the required CMake options, assuming that `${MKLROOT}` is the path of the MKL installation:
-D TPL_ENABLE_MKL:BOOL=ON
-D MKL_LIBRARY_DIRS:FILEPATH="${MKLROOT}/lib/intel64"
-D MKL_LIBRARY_NAMES:STRING="mkl_rt\"
-D MKL_INCLUDE_DIRS:FILEPATH="${MKLROOT}/include"
Third, set the TPETRA_USE_MKL_SPMV environment variable to 1. If you set it to 0, Tpetra will /not/ use MKL for sparse matrix-vector multiply. The environment variable lets you compare performance of Trilinos' implementation and MKL without needing to recompile.
[0001-TpetraKernels-Add-comments-static_assert-to-single-v.txt](https://github.com/trilinos/Trilinos/files/136596/0001-TpetraKernels-Add-comments-static_assert-to-single-v.txt)
[0002-TpetraKernels-Add-MKL-TPL.txt](https://github.com/trilinos/Trilinos/files/136597/0002-TpetraKernels-Add-MKL-TPL.txt)
[0003-Tpetra-Call-MKL-for-single-vec-double-int-SpMV.txt](https://github.com/trilinos/Trilinos/files/136598/0003-Tpetra-Call-MKL-for-single-vec-double-int-SpMV.txt)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/153KokkosSparse::CrsMatrix Default SizeType2016-06-09T20:40:39ZJames WillenbringKokkosSparse::CrsMatrix Default SizeType*Created by: mndevec*
SizeType in the KokkosSparse::CrsMatrix is set with default
--- SizeType = typename ViewTraits< DataType*, Arg1Type, Arg2Type, void >::size_type.
This default appears to be size_t when run on Host, which is the d...*Created by: mndevec*
SizeType in the KokkosSparse::CrsMatrix is set with default
--- SizeType = typename ViewTraits< DataType*, Arg1Type, Arg2Type, void >::size_type.
This default appears to be size_t when run on Host, which is the default used by tpetra as well.
This makes default OrdinalType int, and SizeType as size_t at Kokkos CrsMatrix (Columns are stored as int's while row pointers are stored as size_t.). This causes problems when calling TPL's like MKL that expects same data type for those (Which is also the case for cuSPARSE).
As the matrix is sparse, wouldn't it be better to use the default SizeType as OrdinalType?
@crtrott @mhoemmen @srajama1 @egboman @aprokop @ambrad
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/149boostlib defines not being respected.2016-03-16T15:08:31ZJames Willenbringboostlib defines not being respected.*Created by: bathmatt*
On hansen I'm working on building trilinos and I have set every thing I can think of to set the boost library paths.
dir-debug (gcc) 20 $ grep BoostL configure-drekar.sh
-D TPL_ENABLE_BoostLib=ON \
-D TPL_BoostLi...*Created by: bathmatt*
On hansen I'm working on building trilinos and I have set every thing I can think of to set the boost library paths.
dir-debug (gcc) 20 $ grep BoostL configure-drekar.sh
-D TPL_ENABLE_BoostLib=ON \
-D TPL_BoostLib_INCLUDE_DIRS:FILEPATH="${BOOST_BASE_DIR}/include" \
-D TPL_BoostLib_LIBRARY_DIRS:FILEPATH="${BOOST_BASE_DIR}/lib" \
-D TPL_BoostLib_LIBRARIES="${BOOST_BASE_DIR}/lib/libboost_program_options.so;${BOOST_BASE_DIR}/lib/libboost_system.so" \
However, cmake ignores this and just uses -lboost_system as shown below. This happens on lots of different systems, hansen, chama are just two I can think of. I have a long email thread with Ross but that hit a dead end.
[config.out.txt](https://github.com/trilinos/Trilinos/files/138446/config.out.txt)
/home/projects/x86-64-haswell/openmpi/1.10.0/gnu/4.9.3/bin/mpicxx -std=c++11 -fopenmp -g -O0 CMakeFiles/STKClassic_stk_algsup_unit_tests.dir/UnitTestAlgorithmRunner.cpp.o CMakeFiles/STKClassic_stk_algsup_unit_tests.dir/UnitTestCudaMgr.cpp.o CMakeFiles/STKClassic_stk_algsup_unit_tests.dir/UnitTestMain.cpp.o CMakeFiles/STKClassic_stk_algsup_unit_tests.dir/UnitTest_helpers.cpp.o -o STKClassic_stk_algsup_unit_tests.exe -rdynamic ../stk_algsup/libstkclassic_algsup.a ../../stk_mesh/stk_mesh/fixtures/libstkclassic_mesh_fixtures.a ../../stk_mesh/stk_mesh/fem/libstkclassic_mesh_fem.a ../../stk_mesh/stk_mesh/base/libstkclassic_mesh_base.a ../../stk_util/stk_util/unit_test_support/libstkclassic_util_unit_test_support.a ../../stk_util/stk_util/parallel/libstkclassic_util_parallel.a ../../stk_util/stk_util/diag/libstkclassic_util_diag.a ../../stk_util/stk_util/environment/libstkclassic_util_env.a ../../stk_util/stk_util/util/libstkclassic_util_util.a ../../../../seacas/libraries/exodus/cbind/libexodus.a -lboost_program_options -lboost_system ../../../../fei/support-Trilinos/libfei_trilinos.a ../../../../fei/base/libfei_base.a ../../../../belos/tpetra/src/libbelostpetra.a ../../../../belos/epetra/src/libbelosepetra.a ../../../../belos/src/libbelos.a ../../../../ml/src/libml.a ../../../../galeri/src-epetra/libgaleri-epetra.a ../../../../tpetra/core/ext/libtpetraext.a ../../../../tpetra/core/inout/libtpetrainout.a ../../../../tpetra/core/src/libtpetra.a ../../../../tpetra/kernels/src/libtpetrakernels.a ../../../../kokkos/algorithms/src/libkokkosalgorithms.a ../../../../kokkos/containers/src/libkokkoscontainers.a ../../../../tpetra/classic/LinAlg/libtpetraclassiclinalg.a ../../../../tpetra/classic/NodeAPI/libtpetraclassicnodeapi.a ../../../../tpetra/classic/src/libtpetraclassic.a ../../../../ifpack/src/libifpack.a ../../../../amesos/src/libamesos.a ../../../../epetraext/src/libepetraext.a ../../../../seacas/libraries/ioss/src/init/libIonit.a ../../../../seacas/libraries/ioss/src/transform/libIotr.a ../../../../seacas/libraries/ioss/src/heartbeat/libIohb.a ../../../../seacas/libraries/ioss/src/generated/libIogn.a ../../../../seacas/libraries/ioss/src/pamgen/libIopg.a ../../../../seacas/libraries/ioss/src/exo_fac/libIoexo_fac.a ../../../../seacas/libraries/ioss/src/exo_par/libIopx.a ../../../../seacas/libraries/ioss/src/exo_fpp/libIofx.a ../../../../seacas/libraries/ioss/src/exodus/libIoex.a ../../../../seacas/libraries/ioss/src/libIoss.a ../../../../seacas/libraries/exodus/cbind/libexodus.a -Wl,-Bstatic -lnetcdf -Wl,-Bdynamic -L/home/projects/x86-64-haswell-nvidia/netcdf-exo/4.3.3.1/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.7/lib -lnetcdf -L/home/projects/x86-64-haswell-nvidia/hdf5/1.8.15/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.7/lib -lhdf5_hl -lhdf5 -lz -ldl ../../../../pamgen/src/libpamgen_extras.a ../../../../pamgen/src/libpamgen.a ../../../../aztecoo/src/libaztecoo.a ../../../../triutils/src/libtriutils.a ../../../../epetra/src/libepetra.a ../../../../shards/src/libshards.a ../../../../zoltan/src/libzoltan.a -lm ../../../../sacado/src/libsacado.a ../../../../teuchos/kokkoscomm/src/libteuchoskokkoscomm.a ../../../../teuchos/kokkoscompat/src/libteuchoskokkoscompat.a ../../../../teuchos/remainder/src/libteuchosremainder.a ../../../../teuchos/numerics/src/libteuchosnumerics.a -L/home/projects/x86-64-haswell/lapack/3.5.0/gcc/4.8.4 -llapack -L/home/projects/x86-64-haswell/blas/20150602/gcc/4.8.4 -lblas ../../../../teuchos/comm/src/libteuchoscomm.a ../../../../teuchos/parameterlist/src/libteuchosparameterlist.a ../../../../teuchos/core/src/libteuchoscore.a -lboost_program_options -lboost_system ../../../../kokkos/core/src/libkokkoscore.a -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -lquadmath
I even went so far as to edit the cmake dependencies file
-bash-4.1$ git diff
diff --git a/packages/stk/stk_classic/cmake/Dependencies.cmake b/packages/stk/stk_classic/cmake/Dependencies.cmake
index 0c16502..b7cd94d 100644
--- a/packages/stk/stk_classic/cmake/Dependencies.cmake
+++ b/packages/stk/stk_classic/cmake/Dependencies.cmake
@@ -4,5 +4,5 @@ SET(TEST_REQUIRED_DEP_PACKAGES SEACASExodus)
SET(TEST_OPTIONAL_DEP_PACKAGES)
SET(LIB_REQUIRED_DEP_TPLS Boost BoostLib)
SET(LIB_OPTIONAL_DEP_TPLS OpenNURBS)
-SET(TEST_REQUIRED_DEP_TPLS)
+SET(TEST_REQUIRED_DEP_TPLS BoostLib)
SET(TEST_OPTIONAL_DEP_TPLS gtest)
-bash-4.1$
dir-debug (gcc) 23 $ cmake --version
cmake version 3.3.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/152Intrepid2 Kokkos-View-ization2016-12-01T22:03:15ZJames WillenbringIntrepid2 Kokkos-View-ization*Created by: bathmatt*
I am adding a github issue for the intrepid2 work for more simplicity of tracking.
Summarizing the email below. The initial step is to remove the two MD fields in intrepid2 and replace them with runtime rank k...*Created by: bathmatt*
I am adding a github issue for the intrepid2 work for more simplicity of tracking.
Summarizing the email below. The initial step is to remove the two MD fields in intrepid2 and replace them with runtime rank kokkos views. @etphipp has a prototype dimension structure which handles FAD objects.
I'm adding all the poeple who have be involved in this in the past like
@eric-c-cyr @etphipp @crtrott @kyungjoo-kim @rppawlo
Here are some facts which led us to where we are today.
1: Intrepid2 has a lot of its functionality which uses run time ranked arrays.
2: Intrepid2 has two different multi-dimensional arrays which are sort of interchangeable (The API between these two are not identical)
2A: Intrepid_FieldContainer. This uses an ArrayRCP to store data and tracks dimensions and ranks and provides a multi->1D array. This can't be use in Kokkos functions
2B: Intrepid_FieldContainer_Kokkos, This uses a pointer and special code to allocate and reference data, but this does NOT use a kokkos::view under the hood. This partially works with Kokkos, however, if you have a Intrepid_FC_K<Fad> this won't work on kokkos,
3: Phalanx has a MDField which wraps a true kokkos view, this should work on both CPU/GPU and with kokkos functions. It should work with Kokkos, however, if it is using Fads, you need to take care to define the size when you great the MDField. If not you will get an empty MDField becuase the view length is based on the Fad derivative length.
4: Intrepid2 uses arrays (field containers or some other MD array) of Fads for performing calculations like grads. However, with either Kokkos:View, I_FC, or MDFields, this code will need to be modified so one defines the Fad length at decl. The MDField could be used for this, however, the other FC won't perform on the GPU and can cause deadlocks due to allocations.
5: The Intrepid FC types don't roll the FAD derivative lengths into the view dimension. This can make things slow
6: ArrayWrappers. These allow, in theory, one to send in a static sized array and use it in a dynamic sized array. It often copies data to temps in/out of parallel regions. These make it harder to debug and may not be needed.
7: We use the kokkos execution space based on the (first) array type passed into the function memory space
So, where do we go from here? Here is how I see things. Please
1: I still believe that we need to remove the Intrepid_FC classes and replace them with something like Phalanx::MDField, or better yet, Kokkos::MDView, which is a multi-dimensional array.
2: I think we need to remove all the array wrappers, we need to default to Kokkos::DefaultSpace
3: We need to make everything copy by value in Views,
4: We need to go through the 100 of basis types and fix up the Fad classes to make sure the length is known when we allocate temps.
This is a big undertaking, thankfully Intrepid2 has good test coverage. I k now there will be some pushback on someof this, but we need to do something IMHO. We need to figure how to move forward.
Matt
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/151Tpetra: Add symbolic & numeric analysis phases to KokkosSparse::CrsMatrix2016-06-09T20:45:23ZJames WillenbringTpetra: Add symbolic & numeric analysis phases to KokkosSparse::CrsMatrix*Created by: mhoemmen*
This would let us use TPLs like the MKL inspector-executor interface, that have separate analysis and apply phases.
*Created by: mhoemmen*
This would let us use TPLs like the MKL inspector-executor interface, that have separate analysis and apply phases.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/148Tpetra: Isolate sparse matrix-matrix multiply local kernel2017-02-20T23:11:30ZJames WillenbringTpetra: Isolate sparse matrix-matrix multiply local kernel*Created by: mhoemmen*
@trilinos/tpetra @mndevec @srajama1 @csiefer2
In Tpetra, we need to isolate the "local" part of the sparse matrix-matrix multiply kernel from the "global" (MPI communication) part. This will facilitate thread p...*Created by: mhoemmen*
@trilinos/tpetra @mndevec @srajama1 @csiefer2
In Tpetra, we need to isolate the "local" part of the sparse matrix-matrix multiply kernel from the "global" (MPI communication) part. This will facilitate thread parallelization and the use of third-party libraries' implementations of the local kernel.
Tpetra-FY17-Q2https://gitlab.osti.gov/jmwille/Trilinos/-/issues/154Intrepid2 test uses BasisSet and shouldn't2016-02-24T21:24:43ZJames WillenbringIntrepid2 test uses BasisSet and shouldn't*Created by: bathmatt*
It looks like when you modified this test you didn't ifdef out some of the calls and intrepid2 tests aren't compiling
Trilinos/packages/intrepid2/test/Discretization/Basis/HGRAD_TRI_Cn_FEM/test_04.cpp
It doesn't...*Created by: bathmatt*
It looks like when you modified this test you didn't ifdef out some of the calls and intrepid2 tests aren't compiling
Trilinos/packages/intrepid2/test/Discretization/Basis/HGRAD_TRI_Cn_FEM/test_04.cpp
It doesn't know what BasisSet is unless
#if defined( INTREPID_USING_EXPERIMENTAL_HIGH_ORDER )
so this probably should be ifdefed out
void build_element_matrix_and_rhs(FieldContainer<value_type> & A,
FieldContainer<value_type> & b,
DefaultCubatureFactory<value_type> & cubature_factory,
const BasisSet_HGRAD_TRI_Cn_FEM<value_type,FieldContainer<value_type> > &basis_set,
const int *element,
const int *boundary,
const FieldContainer<value_type> & nodes,
const Orientation ort,
const int nx,
const int ny);
[ 75%] Building CXX object packages/intrepid2/test/Discretization/Basis/HGRAD_TRI_Cn_FEM/CMakeFiles/Intrepid2_test_Discretization_Basis_HGRAD_TRI_Cn_FEM_Test_04.dir/test_04.cpp.o
/home/mbetten/Trilinos/Trilinos/packages/intrepid2/test/Discretization/Basis/HGRAD_TRI_Cn_FEM/test_04.cpp:101:41: error: ‘BasisSet_HGRAD_TRI_Cn_FEM’ does not name a type
const BasisSet_HGRAD_TRI_Cn_FEM<value_type,FieldContainer<value_type> > &basis_set,
^
/home/mbetten/Trilinos/Trilinos/packages/intrepid2/test/Discretization/Basis/HGRAD_TRI_Cn_FEM/test_04.cpp:101:66: error: expected ‘,’ or ‘...’ before ‘<’ token
const BasisSet_HGRAD_TRI_Cn_FEM<value_type,FieldContainer<value_type> > &basis_set,
^
/home/mbetten/Trilinos/Trilinos/packages/intrepid2/test/Discretization/Basis/HGRAD_TRI_Cn_FEM/test_04.cpp:125:34: error: ‘BasisSet_HGRAD_TRI_Cn_FEM’ does not name a type
const BasisSet_HGRAD_TRI_Cn_FEM<value_type,FieldContainer<value_type> > &basis_set,
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/150MueLu: dashboards failures2016-04-29T19:38:42ZJames WillenbringMueLu: dashboards failures*Created by: jhux2*
MueLu has a number of dashboard failures:
![screenshot](https://cloud.githubusercontent.com/assets/12970120/13185625/05ccbaea-d6f7-11e5-9d57-39e7b802f7e6.png)
@aprokop @tawiesn
*Created by: jhux2*
MueLu has a number of dashboard failures:
![screenshot](https://cloud.githubusercontent.com/assets/12970120/13185625/05ccbaea-d6f7-11e5-9d57-39e7b802f7e6.png)
@aprokop @tawiesn
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/156Netcdf and HDF5 ordering2017-03-30T11:51:37ZJames WillenbringNetcdf and HDF5 ordering*Created by: ambrad*
[configure.txt](https://github.com/trilinos/Trilinos/files/147581/configure.txt)
I'm creating an issue to track the problem of netcdf and hdf5 lib ordering.
For the record, let's remember that order matters only fo...*Created by: ambrad*
[configure.txt](https://github.com/trilinos/Trilinos/files/147581/configure.txt)
I'm creating an issue to track the problem of netcdf and hdf5 lib ordering.
For the record, let's remember that order matters only for .a libs. If we have at least libnetcdf.so, then order doesn't matter.
In what follows I summarize what seem to be the key points. I specify SEACAS, Netcdf, and HDF5 as follows:
-D Trilinos_ENABLE_SEACAS:BOOL=ON \
-D TPL_ENABLE_X11:BOOL=OFF \
-D TPL_ENABLE_Matio:BOOL=OFF \
-D Trilinos_ENABLE_SEACASIoss:BOOL=ON \
-D Trilinos_ENABLE_SEACASExodus:BOOL=ON \
-D TPL_ENABLE_Netcdf:BOOL=ON \
-D Netcdf_INCLUDE_DIRS:PATH="$NETCDFDIR/include" \
-D Netcdf_LIBRARY_DIRS:PATH="$NETCDFDIR/lib" \
-D TPL_ENABLE_HDF5:BOOL=ON \
-D HDF5_INCLUDE_DIRS:PATH="$HDF5DIR/include" \
-D HDF5_LIBRARY_DIRS:PATH="$HDF5DIR/lib" \
-D Trilinos_DUMP_LINK_LIBS=ON \
The DUMP_LINK_LIBS flag shows that exodus has the right order:
-- exodus:LINK_LIBS='/usr/local/parallel/lib/libnetcdf.so;/usr/local/parallel/lib/libhdf5.a;/usr/local/lib/libz.so'
and indeed the corresponding link.txt file has the right lib ordering. But, for example, nem_slice does not necessarily. Its dump output is
-- nem_slice:LINK_LIBS='suplib_cpp;suplib;chaco;exodus;zoltan'
which does not include the TPLs on which exodus depends. The corresponding link.txt has HDF5 and Netcdf in the wrong order:
... /usr/local/parallel/lib/libhdf5.a /usr/local/lib/libz.so /usr/local/parallel/lib/libnetcdf.so ...
(Note that libnetcdf is shared, so this particular build goes through without a problem.)
Historically, in the case that the libs are all .a, we have hardcoded lib ordering like this:
```
-DTPL_ENABLE_Netcdf:STRING=ON \
-DNetcdf_INCLUDE_DIRS:PATH="${NETCDF_ROOT}/include" \
-DNetcdf_LIBRARY_DIRS:PATH="${NETCDF_ROOT}/lib" \
-DTPL_Netcdf_LIBRARIES="${NETCDF_ROOT}/lib/libnetcdf.a;${HDF5_ROOT}/lib/libhdf5_hl.a;${HDF5_ROOT}/lib/libhdf5.a;${ZLIB_ROOT}/lib/libz.a" \
-DTPL_ENABLE_HDF5:STRING=ON \
-DHDF5_INCLUDE_DIRS:PATH="${HDF5_ROOT}/include" \
-DTPL_HDF5_LIBRARIES:PATH="${HDF5_ROOT}/lib/libhdf5_hl.a;${HDF5_ROOT}/lib/libhdf5.a;${ZLIB_ROOT}/lib/libz.a" \
```
This has assured that all link.txt files are correct. A fix for this issue would permit the simple specification without providing full-path libs explicitly.
Attached is a text file containing the cmake script, configuration output, and selected cats of link.txt files.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/160Zoltan2_AlgScotch.hpp:649:9: error: 'SCOTCH_STRATLEVELMAX' was not declared i...2016-10-05T20:43:20ZJames WillenbringZoltan2_AlgScotch.hpp:649:9: error: 'SCOTCH_STRATLEVELMAX' was not declared in this scope*Created by: nschloe*
As of recently, I'm getting the config error
```
/«PKGBUILDDIR»/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:649:9: error: 'SCOTCH_STRATLEVELMAX' was not declared in this scope
SCOTCH_S...*Created by: nschloe*
As of recently, I'm getting the config error
```
/«PKGBUILDDIR»/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:649:9: error: 'SCOTCH_STRATLEVELMAX' was not declared in this scope
SCOTCH_STRATLEVELMAX | SCOTCH_STRATLEVELMIN | SCOTCH_STRATLEAFSIMPLE | SCOTCH_STRATSEPASIMPLE,
^
```
Full log [here](https://launchpadlibrarian.net/243538118/buildlog_ubuntu-xenial-amd64.trilinos_12.7~20160228030214-xenial1_BUILDING.txt.gz).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/158Create a SEMSDevEnv.cmake file to automatically use loaded SEMS dev env2016-12-10T03:52:02ZJames WillenbringCreate a SEMSDevEnv.cmake file to automatically use loaded SEMS dev env*Created by: bartlettroscoe*
**Next Action Status:**
Working and providing value as part of new checkin-test-sems.sh script and CI build (see #482)
**CC list:** @rppawlo, @bathmatt, @jgfouca, @jwillenbring, @gdsjaar, @trilinos/fra...*Created by: bartlettroscoe*
**Next Action Status:**
Working and providing value as part of new checkin-test-sems.sh script and CI build (see #482)
**CC list:** @rppawlo, @bathmatt, @jgfouca, @jwillenbring, @gdsjaar, @trilinos/framework
**Blocking:** #410, #370
**Description:**
This story will be to create a SEMSDevEnv.cmake file that once included (with -DTrilinos_CONFIGURE_OPTIONS_FILES=<path>/SEMSDevEnv.cmake), then TriBITS will automatically pick up the right compilers and TPL locations.
In addition, it would be desirable for the Trilinos configure to automatically pick up the loaded SEAMS env (like is done on the ATTB machines using the `ATTB_ENV` env var, see #172).
In addition to just providing the `SEMSDevEnv.cmake` module, this story will also scope out what might be useful for a standard Trilinos dev env. However, a new story will be created to refine what a new expanded Trilinos Primary Tested build of TPLs and Packages will look like.
**List of TPLs and other requirements needed for a standard Trilinos CI build:**
This following list is the current consensus for this the standard Trilinos CI build (this list is updated as consensus changes).
- CMake 3.5.2 (There are huge speed advantages in using CMake 3.5.2 vs. 2.8.11) ([see](https://github.com/trilinos/Trilinos/issues/158#issuecomment-222986351))
- GCC 4.8.4 (@crtrott, [see](https://github.com/trilinos/Trilinos/issues/158#issuecomment-207526974))
- OpenMPI 1.8.7 (@crtrott, [see](https://github.com/trilinos/Trilinos/issues/158#issuecomment-207526974))
- Python 2.7.9 (the only version of 2.x, x > 6 provided by SEMS dev env)
- OpenMP (i.e. Pthreads) (@crtrott, [see](https://github.com/trilinos/Trilinos/issues/158#issuecomment-207526974))
- BLAS and LAPACK, provided by system? ([see](https://github.com/trilinos/Trilinos/issues/158#issuecomment-207500008))
- Boost v1.55.0? ([see](https://github.com/trilinos/Trilinos/issues/158#issuecomment-227600114))
- Zlib 1.2.8 (the version provided by SEMS)
- HDF5 v1.8.12 (the version provided by SEMS, used by SEMS NetCDF 4.3.2)
- NetCDF 4.3.2? (the version provided by SEMS)
- ParMETIS v4.0.3 built with 32-bit index types (@kddevin, [see](https://github.com/trilinos/Trilinos/issues/158#issuecomment-207566543))
- SuperLU 4.3 (used by several Trilinos packages)
- Scotch v6.0.3 built with 32-bit index types (@kddevin, [see](https://github.com/trilinos/Trilinos/issues/158#issuecomment-207566543))
**Tasks:**
1. Get opinions on what TPLs should be included in the `SEMSDevEnv.cmake` module (targeting a standard Trilinos pre-push CI build) [Done]
2. Write out `SEMSDevEnv.cmake` to automatically set up Compilers TPLs for a given loaded SEMS env (using `module load <avail>`) by reading env vars starting with `SEMS_`. (see [below](https://github.com/trilinos/Trilinos/issues/158#issuecomment-229527446)) [Done]
3. Create a `load_sems_dev_env.sh` module that can be called as `source load_sems_dev_env.sh [<compiler-and-version>] [<openmpi-and-version>]`, for example `source load_sems_dev_env.sh gcc/4.9.2 openmpi/1.10.1` (where `<compiler-and-version>` and `<openmpi-and-version>` are given defaults if not provided). (see [below](https://github.com/trilinos/Trilinos/issues/158#issuecomment-229527446)) [Done]
4. Test out on full shared lib MPI and serial builds of Trilinos with all Secondary Tested packages enabled that can be given this set of TPLs. (but forgot to enable Scott and ParMETIS TPLs, see [below](https://github.com/trilinos/Trilinos/issues/158#issuecomment-229527446)) [Done]
5. Automatically load `SEMSDevEnv.cmake` if it is detected that the SEMS dev env is loaded. (see [below](https://github.com/trilinos/Trilinos/issues/158#issuecomment-229527446)) [Done]
6. Enable Scotch and ParMETIS TPLs in shared lib build and test with packages that support them ... Zoltan and Zoltan2 tests fail as expected (see [below](https://github.com/trilinos/Trilinos/issues/158#issuecomment-229999257) and #475 and #476) [Done]
7. Test and get working all-static builds (i.e. build static libs and use static TPL libs) ... see commit c334dd6 [Done]
8. Investigate issues with Scotch and ParMETIS ... SEMS provides inconsistent 32-bit Scotch and 64-bit ParMETIS (see [below](https://github.com/trilinos/Trilinos/issues/158#issuecomment-230135927)) [Done]
9. Try to enable SuperLU 4.3 (see [below](https://github.com/trilinos/Trilinos/issues/158#issuecomment-230485338)) [Done]
10. Finish documentation on [Trilinos GitHub Wiki](https://github.com/trilinos/Trilinos/wiki/SEMS-Dev-Env) (see [below](https://github.com/trilinos/Trilinos/issues/158#issuecomment-230569893)) [Done]
11. Have documentation and implementation reviewed (see [a](https://github.com/trilinos/Trilinos/issues/158#issuecomment-232864091), [b](https://github.com/trilinos/Trilinos/issues/158#issuecomment-233073800), [c](https://github.com/trilinos/Trilinos/issues/158#issuecomment-233136006), [d](https://github.com/trilinos/Trilinos/issues/158#issuecomment-233315765), [e](https://github.com/trilinos/Trilinos/issues/158#issuecomment-237663100), and [f](https://github.com/trilinos/Trilinos/issues/158#issuecomment-237746406)) ... Will fix any problems if they come up. [Done]Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/162Extend status test framework of Belos by a general xml interface2016-05-05T19:14:57ZJames WillenbringExtend status test framework of Belos by a general xml interface*Created by: tawiesn*
I want to be able to define user-specified status tests in Belos using a general flexible xml interface. The definition of the status tests should be similar to how it is done in NOX. The tagging feature would be n...*Created by: tawiesn*
I want to be able to define user-specified status tests in Belos using a general flexible xml interface. The definition of the status tests should be similar to how it is done in NOX. The tagging feature would be nice to have, too.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/165Zoltan2: MJ memory leak when num_global_parts==12016-03-01T23:35:28ZJames WillenbringZoltan2: MJ memory leak when num_global_parts==1*Created by: kddevin*
@trilinos/zoltan2 @mndevec @tscoffe
Mehmet: would you look at this memory leak in MJ detected by STK testing?
The problem occurs in an obscure case, when MJ is called with num_global_parts=1.
The memory leak is ...*Created by: kddevin*
@trilinos/zoltan2 @mndevec @tscoffe
Mehmet: would you look at this memory leak in MJ detected by STK testing?
The problem occurs in an obscure case, when MJ is called with num_global_parts=1.
The memory leak is for array part_xadj.
part_xadj is allocated at line 2121.
In most cases, it is freed and reset at lines 1753-1754.
But when num_global_parts==1, the "continue" at line 1499 skips the free/reset, so part_xadj is never freed.
Given the comment at line 1492, I was afraid I might break cases like the P=4,5,1,2 case mentioned if I tried to fix it. Do we have a test that exercises that case?
I created a test problem that demonstrates the memory leak; I can see it in purify, and the STK team sees it in valgrind: test/driver/driverinputs/coffey
Thanks, Mehmet.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/167Performance of Intrepid2 for PIC2016-12-01T22:06:08ZJames WillenbringPerformance of Intrepid2 for PIC*Created by: bathmatt*
@trilinos/intrepid2 @rppawlo
In PIC methods we have lagrangian particles which migrate throughout the system. We have these particles with position x_i, velocity v_i. We can then derive a current which is the ...*Created by: bathmatt*
@trilinos/intrepid2 @rppawlo
In PIC methods we have lagrangian particles which migrate throughout the system. We have these particles with position x_i, velocity v_i. We can then derive a current which is the J_i = q v_i(x_i). This is a delta function, we will integrate J for the source of a Maxwell solve. Since J is just a sum of delta functions this integration is nothing more than an evaluation of the basis function at the point x_i.
The current typically use the HCurl basis, but for electrostatics we use the HGrad basis..
What we need is a fast way to perform a basis evaluation.
For example, if one wants the tet hcurl basis value at order 2 or higher one calls
Basis_HCURL_TET_In_FEM<Scalar, ArrayScalar>::getValues
which creates a temp array
ArrayScalar phisCur( scalarBigN , numPts );
and calls functions in a new basis functoin
Basis_HGRAD_TET_Cn_FEM_ORTH<Scalar,ArrayScalar > Phis_;
which in this constructor we create more temps.
```
Phis_.getValues( phisCur , inputPoints , OPERATOR_VALUE );
```
which creates a field container of sfads
FieldContainerSacado::Fad::SFad<Scalar,3 > dZ( z.dimension(0) , z.dimension(1) );
```
FieldContainer<Sacado::Fad::SFad<Scalar,3> > dResult(card,np);
```
which calls tabulate recursively.
Then once this is all done it multiplies some output with some local array coeffs_.
This chain skips some steps (I think their may be a human sacrifice in the code as well), but clearly this isn't really a performant call stack if I want to know what basis value is at point (3.1,4.5)...
What I need is a shallow call stack for this which doesn't allocate any memory and can be called on GPUs.
What I've done in the past is to make local versions of this call stack and optimize it with stack vars and other tricks to speed it up. It gave me a factor of 10x improvements on CPU and allowed me to run on GPU.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/170Panzer does instantiate "long long" global ordinal even if Tpetra doesn't2016-03-03T17:24:02ZJames WillenbringPanzer does instantiate "long long" global ordinal even if Tpetra doesn't*Created by: crtrott*
Panzer should check Tpetra macros for whether to use long long as global ordinal or not. If Tpetra does not enable it don't try to instantiate tpetra on it.
@trilinos/panzer
*Created by: crtrott*
Panzer should check Tpetra macros for whether to use long long as global ordinal or not. If Tpetra does not enable it don't try to instantiate tpetra on it.
@trilinos/panzer
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/176Add multi-repository support to Trilinos for inserted packages2017-01-06T17:44:29ZJames WillenbringAdd multi-repository support to Trilinos for inserted packages*Created by: bartlettroscoe*
**Next Action Status:**
Configure of MOOCHO, Mesquite, etc. fail due to wrong base directory. Next: Debug configure problems when running under CTest driver script.
**Blocked By:** #158
**Relates To:** SE...*Created by: bartlettroscoe*
**Next Action Status:**
Configure of MOOCHO, Mesquite, etc. fail due to wrong base directory. Next: Debug configure problems when running under CTest driver script.
**Blocked By:** #158
**Relates To:** SEMS JIRA Issue [TRIL-50](https://sems-jira.sandia.gov/browse/TRIL-50), #440, #452
**CC:** @bmpersc, @jwillenbring
**Description:**
As part of the effort to trim down the Trilinos git repo as part of the move to GitHub, several Trilinos packages (MOOCHO, CTrilinos, ForTrilinos, Sundance, Mesquite, etc.) were filtered out of the main Trilinos git repo and were given their own GitHub repos. The plan for adding them back into the automated builds is documented in the Google Doc [Proposal for trimming down Trilinos repo](https://docs.google.com/document/d/1hbki9KN4ErCChWcvAFBnOsJcPLuDJtPi8xaNq3ETyRo/edit#heading=h.k7j4h5jkz5en).
Note that all of the support for managing multiple repos already exists in TriBITS and has been used for CASL VERA development for many years. For an overview, see ["Multi-Repository Support"](https://tribits.org/doc/TribitsDevelopersGuide.html#multi-repository-support).
**Definition of Done:**
All Trilinos packages that were pulled out but still plan to be tested and released are back running under automated testing and will be automatically included in future releases (using `clone_extra_repos.py` and `gitdist`).
**Tasks:**
1. Add Trilinos packages back as ["Inserted Packages"](https://docs.google.com/document/d/1fLSz7FM8hzmIfr84jQ9B9-C7eXhdLBu0aUyXfKS3XCU/edit) (This is so that they will be defined as TriBITS packages in a way that does not interfere with other "Extra Packages" in "Extra Repos", e.g. through `-DTrilinos_EXTRA_REPOSITORIES=<repo0>,<repo1>,...`). See [below](https://github.com/trilinos/Trilinos/issues/176#issuecomment-220968040) [Done]
2. Add wiki page to Trilinos GitHub wiki site describing the handling of extra repos (e.g. `clone_extra-repos.py`, `gitdist`, etc.)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/175Amesos2 tests FTBFS with MUMPS2016-03-07T18:39:51ZJames WillenbringAmesos2 tests FTBFS with MUMPS*Created by: bavier*
With the following configure script at commit f95a33e826e1f13f080cba89676daf5772dcdc16
```
cmake \
-DCMAKE_CXX_COMPILER=mpic++ \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_Fortran_COMPILER=mpifort \
-DBUILD_SHARED_...*Created by: bavier*
With the following configure script at commit f95a33e826e1f13f080cba89676daf5772dcdc16
```
cmake \
-DCMAKE_CXX_COMPILER=mpic++ \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_Fortran_COMPILER=mpifort \
-DBUILD_SHARED_LIBS:BOOL=YES \
-DTrilinos_ENABLE_Amesos2:BOOL=YES \
-DTPL_ENABLE_MUMPS:BOOL=YES \
-DMUMPS_LIBRARY_NAMES='dmumps;mumps_common;scalapack;lapack;ptscotch;scotch;scotcherr;metis;ptesmumps;esmumps;pord' \
-DTPL_ENABLE_MPI:BOOL=YES \
-DAmesos2_ENABLE_MUMPS:BOOL=YES \
-DAmesos2_ENABLE_TESTS:BOOL=YES \
..
```
I get the following failure when building amesos2/test/solvers/Solver_Test.cpp:
```
In file included from /ptmp/Trilinos/packages/amesos2/src/Amesos2_MUMPS.hpp:49:0,
from /ptmp/Trilinos/packages/amesos2/src/Amesos2_Factory.hpp:120,
from /ptmp/Trilinos/packages/amesos2/src/Amesos2.hpp:45,
from /ptmp/Trilinos/packages/amesos2/test/solvers/Solver_Test.cpp:72:
/ptmp/Trilinos/packages/amesos2/src/Amesos2_MUMPS_def.hpp: In instantiation of 'int Amesos2::MUMPS< <template-parameter-1-1>, <template-parameter-1-2> >::ConvertToTriplet() [with Matrix = Tpetra::CrsMatrix<std::complex<double>, long long int, long long int, Kokkos::
Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>; Vector = Tpetra::MultiVector<std::complex<double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>]':
/ptmp/Trilinos/packages/amesos2/src/Amesos2_MUMPS_def.hpp:404:27: required from 'bool Amesos2::MUMPS< <template-parameter-1-1>, <template-parameter-1-2> >::loadA_impl(Amesos2::EPhase) [with Matrix = Tpetra::CrsMatrix<std::complex<double>, long long int, long long
int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>; Vector = Tpetra::MultiVector<std::complex<double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>]'
/ptmp/Trilinos/packages/amesos2/src/Amesos2_SolverCore_def.hpp:512:18: required from 'void Amesos2::SolverCore<ConcreteSolver, Matrix, Vector>::loadA(Amesos2::EPhase) [with ConcreteSolver = Amesos2::MUMPS; Matrix = Tpetra::CrsMatrix<std::complex<double>, long long
int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>; Vector = Tpetra::MultiVector<std::complex<double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>]'
/ptmp/Trilinos/packages/amesos2/src/Amesos2_SolverCore_def.hpp:105:8: required from 'Amesos2::Solver<Matrix, Vector>& Amesos2::SolverCore<ConcreteSolver, Matrix, Vector>::preOrdering() [with ConcreteSolver = Amesos2::MUMPS; Matrix = Tpetra::CrsMatrix<std::complex<
double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>; Vector = Tpetra::MultiVector<std::complex<double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>]'
/ptmp/Trilinos/packages/amesos2/test/solvers/Solver_Test.cpp:1610:1: required from here
/ptmp/Trilinos/packages/amesos2/src/Amesos2_MUMPS_def.hpp:420:19: error: cannot convert 'Amesos2::MUMPS<Tpetra::CrsMatrix<std::complex<double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>, Tpetra::MultiVector<std::co
mplex<double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false> >::local_ordinal_type* {aka long long int*}' to 'int*' in assignment
mumps_par.irn = (local_ordinal_type*)malloc(mumps_par.nz *sizeof(local_ordinal_type));
^
/ptmp/Trilinos/packages/amesos2/src/Amesos2_MUMPS_def.hpp:421:19: error: cannot convert 'Amesos2::MUMPS<Tpetra::CrsMatrix<std::complex<double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false>, Tpetra::MultiVector<std::co
mplex<double>, long long int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>, false> >::local_ordinal_type* {aka long long int*}' to 'int*' in assignment
mumps_par.jcn = (local_ordinal_type*)malloc(mumps_par.nz * sizeof(local_ordinal_type));
^
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/178Tpetra BCRS: Thread-parallelize sparse matrix-vector multiply2016-06-02T15:57:50ZJames WillenbringTpetra BCRS: Thread-parallelize sparse matrix-vector multiply*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2 @crtrott @kyungjoo-kim @amklinv
Thread-parallelize the sparse matrix-vector multiply in the apply() method of Tpetra::Experimental::BlockCrsMatrix. Please interact with Ryan E...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2 @crtrott @kyungjoo-kim @amklinv
Thread-parallelize the sparse matrix-vector multiply in the apply() method of Tpetra::Experimental::BlockCrsMatrix. Please interact with Ryan Eberhardt, who has an excellent CUDA implementation for column-major blocks.
It would be wise to do this in two passes. First, add a simple host execution space parallelization using a lambda. Then, implement an optimized kernel, using Ryan's as a start.
This affects Ifpack2 as well as Tpetra, because for Jacobi with > 1 sweep, Ifpack2 uses sparse mat-vec.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/180Tpetra BCRS: Improve vectorization of small dense linear algebra operations2016-06-04T23:50:26ZJames WillenbringTpetra BCRS: Improve vectorization of small dense linear algebra operations*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2 @crtrott @kyungjoo-kim @amklinv
Tpetra::Experimental::BlockCrsMatrix uses the small dense linear algebra operations currently implemented in Tpetra_Experimental_BlockView.hpp. ...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2 @crtrott @kyungjoo-kim @amklinv
Tpetra::Experimental::BlockCrsMatrix uses the small dense linear algebra operations currently implemented in Tpetra_Experimental_BlockView.hpp. These operations take Kokkos::View or LittleVector / LittleBlock. (Their interfaces are enough alike from the perspective of these operations, that we need only consider Kokkos::View in what follows, without loss of generality.) For example, Tpetra::Experimental::GEMV (small dense matrix times small dense vector) takes a rank-2 View (the matrix) and two rank-1 Views (input and output vectors).
Discussions a couple weeks ago with @nmhamster suggested that we could get outer loop vectorization by doing the following:
1. Change the storage layout so that the (i,j) entries of consecutive blocks (or the (i) entries of consecutive vectors) are stored contiguously
2. Linear algebra operations on those small dense blocks would then need to take a whichBlock / whichVector index argument, to tell which block / vector to use
The routines wouldn't change, except that instead of writing A(i,j) or x(k) (for example), we would write A(i,j,whichBlock) or x(k,whichBlock). We have to rely on Kokkos::View::operator() to inline, but this is a much easier approach than explicit SIMD.
This depends on #177 and #179.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/179Tpetra BCRS: Detach public interface from layout of little blocks2016-11-15T04:51:00ZJames WillenbringTpetra BCRS: Detach public interface from layout of little blocks*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2 @crtrott @kyungjoo-kim @amklinv
The public interface of Tpetra::Experimental::BlockCrsMatrix currently assumes that little blocks are row major (LayoutRight, in Kokkos terms), ...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/ifpack2 @crtrott @kyungjoo-kim @amklinv
The public interface of Tpetra::Experimental::BlockCrsMatrix currently assumes that little blocks are row major (LayoutRight, in Kokkos terms), regardless of the execution space. It would make sense to relax this restriction. This would have the following advantages:
1. It would let the layout be a function of the execution space, hopefully for best performance
2. It would allow strided layouts, so that the (i,j) entries of consecutive blocks could be stored contiguously, thus making outer loop vectorization (for Intel Xeon Phi) easier
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/187Tpetra: Add raw-pointer overloads of insert, replace, & sumInto for CrsGraph ...2016-03-09T12:29:27ZJames WillenbringTpetra: Add raw-pointer overloads of insert, replace, & sumInto for CrsGraph & CrsMatrix*Created by: mhoemmen*
@trilinos/tpetra @srajama1
This is a generalization of Bugzilla Bug 6447:
https://software.sandia.gov/bugzilla/show_bug.cgi?id=6447
*Created by: mhoemmen*
@trilinos/tpetra @srajama1
This is a generalization of Bugzilla Bug 6447:
https://software.sandia.gov/bugzilla/show_bug.cgi?id=6447
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/183Build failure with Ifpack2, Zoltan2, and AMD TPL2017-05-12T01:16:43ZJames WillenbringBuild failure with Ifpack2, Zoltan2, and AMD TPL*Created by: bavier*
At commit 17ccfa78c106fd86812563a4ab3d50ceace9c13a, I configure Trilinos with simply:
```
cmake \
-DTPL_ENABLE_AMD:BOOL=YES \
-DBUILD_SHARED_LIBS:BOOL=YES \
-DTrilinos_ENABLE_Zoltan2:BOOL=YES \
-DTrilinos_E...*Created by: bavier*
At commit 17ccfa78c106fd86812563a4ab3d50ceace9c13a, I configure Trilinos with simply:
```
cmake \
-DTPL_ENABLE_AMD:BOOL=YES \
-DBUILD_SHARED_LIBS:BOOL=YES \
-DTrilinos_ENABLE_Zoltan2:BOOL=YES \
-DTrilinos_ENABLE_Ifpack2:BOOL=YES \
..
```
During build, I get the following error:
```
In file included from /ptmp/Trilinos/packages/zoltan2/src/algorithms/order/Zoltan2_OrderingAlgorithms.hpp:53:0,
from /ptmp/Trilinos/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:54,
from /ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:68,
from /ptmp/Trilinos/build/build-zoltan2-amd/packages/ifpack2/src/Ifpack2_AdditiveSchwarz.hpp:2,
from /ptmp/Trilinos/packages/ifpack2/src/Ifpack2_Details_Factory_def.hpp:47,
from /ptmp/Trilinos/build/build-zoltan2-amd/packages/ifpack2/src/Ifpack2_Details_Factory.hpp:2,
from /ptmp/Trilinos/packages/ifpack2/src/Ifpack2_Factory_decl.hpp:48,
from /ptmp/Trilinos/build/build-zoltan2-amd/packages/ifpack2/src/Ifpack2_Factory.hpp:1,
from /ptmp/Trilinos/packages/ifpack2/src/Ifpack2_Details_LinearSolverFactory_def.hpp:54,
from /ptmp/Trilinos/build/build-zoltan2-amd/packages/ifpack2/src/Ifpack2_Details_LinearSolverFactory.hpp:2,
from /ptmp/Trilinos/packages/ifpack2/src/Ifpack2_Details_registerLinearSolverFactory.cpp:45:
/ptmp/Trilinos/packages/zoltan2/src/algorithms/order/Zoltan2_AlgAMD.hpp: In instantiation of 'int Zoltan2::AlgAMD<Adapter>::order(const Teuchos::RCP<Zoltan2::OrderingSolution<typename Adapter::lno_t, typename Adapter::gno_t> >&) [with Adapter = Zoltan2::MatrixAdapter<Xpetra::RowMatrix<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >; typename Adapter::gno_t = long long int; typename Adapter::lno_t = int]':
/ptmp/Trilinos/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:229:11: required from 'void Zoltan2::OrderingProblem<Adapter>::solve(bool) [with Adapter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >]'
/ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1517:5: required from 'void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >; LocalInverseType = Ifpack2::Preconditioner<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]'
/ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1134:11: required from 'void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::initialize() [with MatrixType = Tpetra::RowMatrix<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >; LocalInverseType = Ifpack2::Preconditioner<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]'
/ptmp/Trilinos/packages/ifpack2/src/Ifpack2_Details_registerLinearSolverFactory.cpp:76:1: required from here
/ptmp/Trilinos/packages/zoltan2/src/algorithms/order/Zoltan2_AlgAMD.hpp:154:70: error: no matching function for call to 'Zoltan2::AMDTraits<int>::order(const size_t&, const int*, const long long int*, lno_t*&, double [5], double [20])'
edgeIds.getRawPtr(), perm, Control, Info);
^
/ptmp/Trilinos/packages/zoltan2/src/algorithms/order/Zoltan2_AlgAMD.hpp:77:9: note: candidate: int Zoltan2::AMDTraits<int>::order(int, const int*, const int*, int*, double*, double*)
int order(int n, const int *Ap, const int *Ai, int *perm,
^
/ptmp/Trilinos/packages/zoltan2/src/algorithms/order/Zoltan2_AlgAMD.hpp:77:9: note: no known conversion for argument 3 from 'const long long int*' to 'const int*'
packages/ifpack2/src/CMakeFiles/ifpack2.dir/build.make:62: recipe for target 'packages/ifpack2/src/CMakeFiles/ifpack2.dir/Ifpack2_Details_registerLinearSolverFactory.cpp.o' failed
make[2]: *** [packages/ifpack2/src/CMakeFiles/ifpack2.dir/Ifpack2_Details_registerLinearSolverFactory.cpp.o] Error 1
CMakeFiles/Makefile2:6589: recipe for target 'packages/ifpack2/src/CMakeFiles/ifpack2.dir/all' failed
make[1]: *** [packages/ifpack2/src/CMakeFiles/ifpack2.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/184ArrayView in DEBUG build gives (probably) erroneous RCP message2017-02-03T23:04:23ZJames WillenbringArrayView in DEBUG build gives (probably) erroneous RCP message*Created by: ambrad*
Consider the following short program:
```
#include "Tpetra_Map.hpp"
#include "Teuchos_ArrayView.hpp"
#include "Tpetra_DefaultPlatform.hpp"
#include "Teuchos_GlobalMPISession.hpp"
typedef double ST;
typedef int GO;...*Created by: ambrad*
Consider the following short program:
```
#include "Tpetra_Map.hpp"
#include "Teuchos_ArrayView.hpp"
#include "Tpetra_DefaultPlatform.hpp"
#include "Teuchos_GlobalMPISession.hpp"
typedef double ST;
typedef int GO;
typedef int LO;
typedef Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> KokkosNode;
typedef Tpetra::Map<LO, GO, KokkosNode> Tpetra_Map;
int main (int argc, char** argv) {
Teuchos::GlobalMPISession mpiSession(&argc, &argv);
auto comm = Tpetra::DefaultPlatform::getDefaultPlatform().getComm();
static const int N = 10;
#if 1
// In a Trilinos DEBUG build, fails with RCP message.
GO* ids = new GO[N];
for (int i = 0; i < N; ++i) ids[i] = i;
Teuchos::ArrayView<const GO> ids_av = Teuchos::arrayView(ids, N);
Tpetra_Map m(N, ids_av, 0, comm);
delete[] ids;
#else
// OK (and better, anyway).
Teuchos::Array<GO> ids(N);
for (int i = 0; i < N; ++i) ids[i] = i;
Tpetra_Map m(N, ids, 0, comm);
#endif
}
```
In the `#if 1` branch, the program fails with:
```
terminate called after throwing an instance of 'Teuchos::DanglingReferenceError'
what(): /home/ambradl/bigcode/tril/Trilinos/install-shared-dbg/include/Teuchos_RCPNode.hpp:605:
Throw number = 1
Throw test that evaluated to true: true
Error, an attempt has been made to dereference the underlying object
from a weak smart pointer object where the underling object has already
been deleted since the strong count has already gone to zero.
```
The #if 0 branch is fine. This appears to be a bug in ArrayView in a DEBUG build.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/185Tpetra::MatrixMarket::Reader can't read graphs (pattern only)2017-09-06T20:02:25ZJames WillenbringTpetra::MatrixMarket::Reader can't read graphs (pattern only)*Created by: mhoemmen*
@trilinos/tpetra @egboman
Moved from Bugzilla Bug 6130: https://software.sandia.gov/bugzilla/show_bug.cgi?id=6130
Bug posted by @egboman . Original text:
I need to read MatrixMarket files with no values (patt...*Created by: mhoemmen*
@trilinos/tpetra @egboman
Moved from Bugzilla Bug 6130: https://software.sandia.gov/bugzilla/show_bug.cgi?id=6130
Bug posted by @egboman . Original text:
I need to read MatrixMarket files with no values (pattern only) into Tpetra for Zoltan2 testing. Ideally, I would like to get a CrsGraph returned from the Tpetra::MatrixMarket::Reader. I don't see how to do this?
A work-around would be for Tpetra::MatrixMarket::Reader to create a CrsMatrix with explicit zeros. Then I could extract the graph from the matrix.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/190Tpetra: Configure passes with no Node enabled2016-03-10T12:44:55ZJames WillenbringTpetra: Configure passes with no Node enabled*Created by: mhoemmen*
Moved from Bugzilla Bug 6474:
https://software.sandia.gov/bugzilla/show_bug.cgi?id=6474
@aprokop reported on 20 Nov 2015:
If ETI is on, and Tpetra_INST_Serial=OFF, Tpetra reports that all nodes are disabled, ...*Created by: mhoemmen*
Moved from Bugzilla Bug 6474:
https://software.sandia.gov/bugzilla/show_bug.cgi?id=6474
@aprokop reported on 20 Nov 2015:
If ETI is on, and Tpetra_INST_Serial=OFF, Tpetra reports that all nodes are disabled, but does not fail to configure. It really should! Configure script:
-D Trilinos_ENABLE_OpenMP=OFF
-D Tpetra_INST_SERIAL=OFF
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION=ON
-D Tpetra_INST_INT_INT=OFF
-D Tpetra_INST_INT_LONG=OFF
-D Tpetra_INST_INT_LONG_LONG=ON
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/192Tpetra: Rename TpetraKernels subpackage to KokkosKernels package (post copyri...2017-06-01T19:42:57ZJames WillenbringTpetra: Rename TpetraKernels subpackage to KokkosKernels package (post copyright)*Created by: mhoemmen*
@trilinos/tpetra @mndevec
This was originally Bugzilla Bug 6439: https://software.sandia.gov/bugzilla/show_bug.cgi?id=6439
Original text posted by me on 20 Oct 2015:
TpetraKernels, a subpackage of Tpetra, is a...*Created by: mhoemmen*
@trilinos/tpetra @mndevec
This was originally Bugzilla Bug 6439: https://software.sandia.gov/bugzilla/show_bug.cgi?id=6439
Original text posted by me on 20 Oct 2015:
TpetraKernels, a subpackage of Tpetra, is a temporary parking spot for code that will go into the (not yet existing) KokkosKernels package. I named it "TpetraKernels" only because I thought that TriBITS restricted subpackage names to start with the package name. It looks like this requirement doesn't exist, at least not anymore. It would be better to rename this subpackage to KokkosKernels, so that when KokkosKernels becomes its own package, users won't have to change their CMake configuration scripts.
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/194Tpetra: Simplify initialization paths2016-06-13T22:07:39ZJames WillenbringTpetra: Simplify initialization paths*Created by: mhoemmen*
THIS COMMENT SUPERSEDED BY MY LONG COMMENT BELOW (01:23 09 Mar 2016).
@trilinos/tpetra
Moved from Bugzilla Bug 6361: https://software.sandia.gov/bugzilla/show_bug.cgi?id=6361
That bug discussion centered around...*Created by: mhoemmen*
THIS COMMENT SUPERSEDED BY MY LONG COMMENT BELOW (01:23 09 Mar 2016).
@trilinos/tpetra
Moved from Bugzilla Bug 6361: https://software.sandia.gov/bugzilla/show_bug.cgi?id=6361
That bug discussion centered around whether Tpetra should initialize Kokkos and MPI (if enabled) for users automatically. The motivating problem is that users encountered unexpected, undesired Kokkos behavior, because Tpetra initialized Kokkos with defaults, when it should instead have used Kokkos' input command-line arguments. The main benefit of automatic initialization is that Tpetra users who don't otherwise use Kokkos or MPI, don't have to know about either. The risks of automatic initialization are:
1. Need to pass in command-line arguments through (argc, argv), for both Kokkos and MPI
2. If we silently initialize Kokkos with defaults, users might get undesired behavior
We didn't all come to full agreement, but I think we can agree that the following two options make sense:
- initialize(argc,argv): Tpetra initializes MPI (if not already) and Kokkos (if not already), passing in command-line arguments to both MPI_Init and Kokkos::initialize. Note that MPI_Init reserves the right to modify both argc and argv (e.g., to delete MPI's command-line arguments).
- initialize(): Tpetra requires that both MPI and Kokkos are initialized. If not, the function throws.
It is unnecessary to provide an initialize() function that takes an `MPI_Comm`, because Tpetra does not have a "default MPI communicator." `Tpetra::Map` takes and wraps the user's communicator. Similarly, initialize() need not take a Kokkos execution space instance; the proper place for that would be the constructor of `Tpetra::Map`.
Note that the zero-argument version of initialize() above changes current behavior with respect to Kokkos initialization. Current behavior is that initialize() initializes Kokkos with default arguments.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/141stk::io::StkMeshIoBroker in parallel2016-02-15T02:35:06ZJames Willenbringstk::io::StkMeshIoBroker in parallel*Created by: jandrej*
When running a minimal example in parallel (lets assume mpirun -n 2 on a single host) for example
```
stk::io::StkMeshIoBroker broker(MPI_COMM_WORLD);
broker.property_add(Ioss::Property("DECOMPOSITION_METHOD", "rc...*Created by: jandrej*
When running a minimal example in parallel (lets assume mpirun -n 2 on a single host) for example
```
stk::io::StkMeshIoBroker broker(MPI_COMM_WORLD);
broker.property_add(Ioss::Property("DECOMPOSITION_METHOD", "rcb"));
broker.add_mesh_database(absoluteFileName, stk::io::READ_MESH);
```
The add_mesh_database routine throws
```
terminate called after throwing an instance of 'std::runtime_error'
what(): Expr '!(Teuchos::is_null(m_database) || !m_database->ok(true))' eval'd to true, throwing.
Error occured at: juan/src/Trilinos/packages/stk/stk_io/stk_io/InputFile.cpp:104
Error: ERROR: Could not open database '/home/juan/src/Trilinos/_build/packages/trilinoscouplings/examples/scaling/unit_cube_10int_hex.exo' of type 'exodusii'
```
This also happens in the `packages/trilinoscouplings/examples/scaling/example_Poisson_stk.cpp` example.
I can not find any further documentation or hints about this issue. Is my assumption wrong, that it should work in parallel or do i have to configure further properties?
EDIT:
I'm testing this now with parallel-netcdf (which might be the issue)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/195Amesos2 SuperLUDist configure error in Morgan2016-03-09T00:18:49ZJames WillenbringAmesos2 SuperLUDist configure error in Morgan*Created by: srajama1*
[ 26%] Building CXX object packages/amesos/src/CMakeFiles/amesos.dir/Amesos_Superludist.cpp.o
/home/projects/sparc/tpls/Trilinos/packages/amesos/src/Amesos_Superludist.cpp(448): error: argument of type "int" is in...*Created by: srajama1*
[ 26%] Building CXX object packages/amesos/src/CMakeFiles/amesos.dir/Amesos_Superludist.cpp.o
/home/projects/sparc/tpls/Trilinos/packages/amesos/src/Amesos_Superludist.cpp(448): error: argument of type "int" is incompatible with parameter of type "LUstruct_t *"
LUstructInit(NumGlobalRows_, NumGlobalRows_, &PrivateSuperluData_->LUstruct_);
^
/home/projects/sparc/tpls/Trilinos/packages/amesos/src/Amesos_Superludist.cpp(448): error #140: too many arguments in function call
LUstructInit(NumGlobalRows_, NumGlobalRows_, &PrivateSuperluData_->LUstruct_);
^
/home/projects/sparc/tpls/Trilinos/packages/amesos/src/Amesos_Superludist.cpp(485): error: identifier "DOUBLE" is undefined
DOUBLE
^
/home/projects/sparc/tpls/Trilinos/packages/amesos/src/Amesos_Superludist.cpp(494): error: identifier "EXTRA" is undefined
EXTRA
^
compilation aborted for /home/projects/sparc/tpls/Trilinos/packages/amesos/src/Amesos_Superludist.cpp (code 2)
make[2]: **\* [packages/amesos/src/CMakeFiles/amesos.dir/Amesos_Superludist.cpp.o] Error 2
make[1]: **\* [packages/amesos/src/CMakeFiles/amesos.dir/all] Error 2
make: **\* [all] Error 2
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/197ShyLU and Ifpack2 : Add FastILU and TriSolve to Trilinos2017-10-02T18:12:30ZJames WillenbringShyLU and Ifpack2 : Add FastILU and TriSolve to Trilinos*Created by: srajama1*
@srajama1 @trilinos/shylu @trilinos/ifpack2
*Created by: srajama1*
@srajama1 @trilinos/shylu @trilinos/ifpack2
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/199BelosSolverManager.hpp update broke build.2016-03-09T18:49:07ZJames WillenbringBelosSolverManager.hpp update broke build.*Created by: bathmatt*
@amklinv
Your latest push broke the build
[ 65%] Building CXX object packages/belos/src/CMakeFiles/belos.dir/Belos_Details_registerLinearSolverFactory.cpp.o
[ 68%] Built target ml
In file included from https://j...*Created by: bathmatt*
@amklinv
Your latest push broke the build
[ 65%] Building CXX object packages/belos/src/CMakeFiles/belos.dir/Belos_Details_registerLinearSolverFactory.cpp.o
[ 68%] Built target ml
In file included from https://jenkins-srn.sandia.gov:8443/job/Drekar-opt/ws/Trilinos/packages/belos/src/BelosSolverFactory.hpp:47:0,
from https://jenkins-srn.sandia.gov:8443/job/Drekar-opt/ws/Trilinos/packages/belos/src/Belos_Details_LinearSolverFactory.hpp:48,
from https://jenkins-srn.sandia.gov:8443/job/Drekar-opt/ws/Trilinos/packages/belos/src/Belos_Details_registerLinearSolverFactory.cpp:43:
https://jenkins-srn.sandia.gov:8443/job/Drekar-opt/ws/Trilinos/packages/belos/src/BelosSolverManager.hpp:143:20: error: expected nested-name-specifier before ‘StatusTestCombo’
const typename StatusTestCombo<ScalarType,MV,OP>::ComboType &comboType =
^
https://jenkins-srn.sandia.gov:8443/job/Drekar-opt/ws/Trilinos/packages/belos/src/BelosSolverManager.hpp:143:35: error: expected ‘,’ or ‘...’ before ‘<’ token
const typename StatusTestCombo<ScalarType,MV,OP>::ComboType &comboType =
^
make[2]: **\* [packages/belos/src/CMakeFiles/belos.dir/Belos_Details_registerLinearSolverFactory.cpp.o] Error 1
make[1]: **\* [packages/belos/src/CMakeFiles/belos.dir/all] Error 2
make: **\* [all] Error 2
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/198Kokkos Serial execution space is not tested with TeamPolicy.2016-03-10T17:45:34ZJames WillenbringKokkos Serial execution space is not tested with TeamPolicy.*Created by: kyungjoo-kim*
The following error comes from when I use Serial space (OpenMP space is okay). I think that the TeamPolicy does not work here. The Tpetra block crs matrix is already submitted to Christian's Trilinos repo and ...*Created by: kyungjoo-kim*
The following error comes from when I use Serial space (OpenMP space is okay). I think that the TeamPolicy does not work here. The Tpetra block crs matrix is already submitted to Christian's Trilinos repo and you can reproduce this error in CI testing.
/home/kyukim/Work/lib/trilinos/christian/packages/kokkos/core/src/Kokkos_Serial.hpp:672:9: error: no match for call to ‘(const Tpetra::Experimental::BlockCrsMatrix<S, LO, GO, N>::localApplyBlockNoTrans(const Tpetra::Experimental::BlockMultiVector<Scalar, LO, GO, Node>&, Tpetra::Experimental::BlockMultiVector<Scalar, LO, GO, Node>&, Scalar, Scalar) [with Scalar = double; LO = int; GO = int; Node = Kokkos::Compat::KokkosDeviceWrapperNodeKokkos::Serial]::__lambda2) (Kokkos::Impl::ParallelFor<Tpetra::Experimental::BlockCrsMatrix<S, LO, GO, N>::localApplyBlockNoTrans(const Tpetra::Experimental::BlockMultiVector<Scalar, LO, GO, Node>&, Tpetra::Experimental::BlockMultiVector<Scalar, LO, GO, Node>&, Scalar, Scalar) [with Scalar = double; LO = int; GO = int; Node = Kokkos::Compat::KokkosDeviceWrapperNodeKokkos::Serial]::__lambda2, Kokkos::TeamPolicyKokkos::Schedule<Kokkos::Dynamic, Kokkos::Serial>, Kokkos::Serial>::Member)’
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/200Team reduction for array type2016-10-23T02:33:18ZJames WillenbringTeam reduction for array type*Created by: kyungjoo-kim*
Currently team reduction is designed for scalar type and I need team reduction for array and the array should be thread specific. The input array may have initial values.
template< typename iType, class Lamb...*Created by: kyungjoo-kim*
Currently team reduction is designed for scalar type and I need team reduction for array and the array should be thread specific. The input array may have initial values.
template< typename iType, class Lambda, typename ValueType, class JoinType >
KOKKOS_INLINE_FUNCTION
void parallel_reduce(const Impl::TeamThreadRangeBoundariesStruct<iType,Impl::ThreadsExecTeamMember>& loop_boundaries,
const Lambda & lambda, const JoinType& join, ValueType& init_result) {
ValueType result = init_result;
for( iType i = loop_boundaries.start; i < loop_boundaries.end; i+=loop_boundaries.increment) {
ValueType tmp = ValueType();
/// this tmp initialization should be reconsider to use thread local array.
```
lambda(i,tmp);
join(result,tmp);
```
}
init_result = loop_boundaries.thread.team_reduce(result,Impl::JoinLambdaAdapter<ValueType,JoinType>(join));
}
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/171Tpetra test depends on Epetra header file2016-03-08T04:14:35ZJames WillenbringTpetra test depends on Epetra header file*Created by: egboman*
When I build Trilinos with Tpetra and tests enabled, but without Epetra, I get the error:
/home/egboman/trilinos/Trilinos/packages/tpetra/core/example/advanced/Benchmarks/localView.cpp:90:31: fatal error: Epetra_S...*Created by: egboman*
When I build Trilinos with Tpetra and tests enabled, but without Epetra, I get the error:
/home/egboman/trilinos/Trilinos/packages/tpetra/core/example/advanced/Benchmarks/localView.cpp:90:31: fatal error: Epetra_SerialComm.h: No such file or directory
compilation terminated.
It looks like the line including that header file is not properly guarded by
# ifdef HAVE_TPETRACORE_EPETRA
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/212Tpetra::CrsGraph::getLocalDiagOffsets: Kokkos-ize2016-03-19T10:18:01ZJames WillenbringTpetra::CrsGraph::getLocalDiagOffsets: Kokkos-ize*Created by: mhoemmen*
This blocks #41, but is only a serious blocker if the matrix's graph structure changes frequently.
This depends on #205.
It's OK for `Tpetra::CrsGraph::getLocalDiagOffsets` only to have a full Kokkos parallelizat...*Created by: mhoemmen*
This blocks #41, but is only a serious blocker if the matrix's graph structure changes frequently.
This depends on #205.
It's OK for `Tpetra::CrsGraph::getLocalDiagOffsets` only to have a full Kokkos parallelization (e.g., on CUDA device) if the graph is fillComplete. If not, either the existing sequential code or a host execution space parallelization suffice. Thus, the code in CrsGraph should branch between these two cases.
Ifpack2 should only exercise the fillComplete case, otherwise it's a performance bug for Ifpack2 and possibly a correctness bug for the application. (Giving a non-fillComplete matrix to Ifpack2 is wrong, because one should always fillComplete the matrix before handing it off to a solver. If the matrix has never been fillComplete before, calling fillComplete could change the offsets of the diagonal entries, which is what getLocalDiagOffsets computes.)
Take care to make the whole Kokkos::StaticCrsGraph memory-unmanaged before entering the Kokkos kernel. Otherwise, it will be slow with non-Cuda execution spaces, due to memory management overhead.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/204Missing instantiations in muelu-adapters2016-03-11T16:54:17ZJames WillenbringMissing instantiations in muelu-adapters*Created by: tawiesn*
If, e.g., Tpetra is instantiated both on GO=int and GO=long long, the <double, int, int, ...> instantiations of classes in muelu-adapters (such as Xpetra::LinearOpBase, and MueLuPreconditionerFactory)
*Created by: tawiesn*
If, e.g., Tpetra is instantiated both on GO=int and GO=long long, the <double, int, int, ...> instantiations of classes in muelu-adapters (such as Xpetra::LinearOpBase, and MueLuPreconditionerFactory)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/210Amesos2_Solver_Test ETI2016-04-06T15:55:05ZJames WillenbringAmesos2_Solver_Test ETI*Created by: jdbooth*
The ETI for Tpetra is setup wrong, limiting the number of test.
@trilinos/amesos2
*Created by: jdbooth*
The ETI for Tpetra is setup wrong, limiting the number of test.
@trilinos/amesos2
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/217Use fence() instead of CUDA_LAUNCH_BLOCKING=1 in Tpetra2016-05-03T00:07:31ZJames WillenbringUse fence() instead of CUDA_LAUNCH_BLOCKING=1 in Tpetra*Created by: ambrad*
My understanding is that, right now, we need to set CUDA_LAUNCH_BLOCKING=1 to use Tpetra and possibly other Trilinos packages on the GPU.
This flag synchronizes launch of kernels; that is, one kernel launches and c...*Created by: ambrad*
My understanding is that, right now, we need to set CUDA_LAUNCH_BLOCKING=1 to use Tpetra and possibly other Trilinos packages on the GPU.
This flag synchronizes launch of kernels; that is, one kernel launches and completes, and then the next does.
In application development, we'd like to avoid that for two reasons.
First, we'd like to run with Tpetra but test that our own application code does not have a race condition between kernels. I believe blocking is being forced on the app by that env var, so that means we can't test for a race condition in our own code.
I'm not sure what effect the CUDA_LAUNCH_BLOCKING flag has in the case of having multiple GPUs per node. But, second, we'll certainly want to make sure we can have multiple kernels launched from one MPI rank running simultaneously across GPUs on a node.
I believe the proper fix for this is to track down places in Tpetra where fence() needs to be used. Once fence() is used in all the right spots, then CUDA_LAUNCH_BLOCKING can be set to 0.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/221Missing Sacado test dependency2016-03-17T17:39:22ZJames WillenbringMissing Sacado test dependency*Created by: jwillenbring*
@etphipp
When building Sacado for a nightly no-C++11 build with optional packages off, the following error occurred:
/jenkins/slave/workspace/Trilinos_XYCE_MPI/MPI_OPT_DEV_XYCE/Trilinos/packages/sacado/test...*Created by: jwillenbring*
@etphipp
When building Sacado for a nightly no-C++11 build with optional packages off, the following error occurred:
/jenkins/slave/workspace/Trilinos_XYCE_MPI/MPI_OPT_DEV_XYCE/Trilinos/packages/sacado/test/UnitTests/ConversionTests.cpp:30:39: error: Teuchos_UnitTestHarness.hpp: No such file or directory
/jenkins/slave/workspace/Trilinos_XYCE_MPI/MPI_OPT_DEV_XYCE/Trilinos/packages/sacado/test/UnitTests/ConversionTests.cpp:31:42: error: Teuchos_UnitTestRepository.hpp: No such file or directory
/jenkins/slave/workspace/Trilinos_XYCE_MPI/MPI_OPT_DEV_XYCE/Trilinos/packages/sacado/test/UnitTests/ConversionTests.cpp:32:40: error: Teuchos_GlobalMPISession.hpp: No such file or directory
/jenkins/slave/workspace/Trilinos_XYCE_MPI/MPI_OPT_DEV_XYCE/Trilinos/packages/sacado/test/UnitTests/ConversionTests.cpp:33:38: error: Teuchos_TestingHelpers.hpp: No such file or directory
/jenkins/slave/workspace/Trilinos_XYCE_MPI/MPI_OPT_DEV_XYCE/Trilinos/packages/sacado/test/UnitTests/ConversionTests.cpp:53: error: expected constructor, destructor, or type conversion before ‘(’ token
/jenkins/slave/workspace/Trilinos_XYCE_MPI/MPI_OPT_DEV_XYCE/Trilinos/packages/sacado/test/UnitTests/ConversionTests.cpp:70: error: template declaration of ‘bool test_ad_conversions’
/jenkins/slave/workspace/Trilinos_XYCE_MPI/MPI_OPT_DEV_XYCE/Trilinos/packages/sacado/test/UnitTests/ConversionTests.cpp:70: error: ‘Teuchos’ has not been declared
...
The full context can be seen here:
http://testing.sandia.gov/cdash/viewBuildError.php?buildid=2380725
I have a simple fix that I have tested which is to add TeuchosCore as a required test dependence for Sacado. I did not investigate if it would be possible to refactor the testing to optionally use Teuchos. It seems very likely that this case isn't commonly exercised, as Teuchos is an optional library dependence for Sacado, and Teuchos is included in the vast majority of Trilinos builds.
I plan to issue a pull request for this simple fix later today.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/223Zoltan2: Transfer number-of-message metrics from HOMME to Zoltan22018-07-03T16:52:46ZJames WillenbringZoltan2: Transfer number-of-message metrics from HOMME to Zoltan2*Created by: kddevin*
@mndevec has implemented metrics measuring the number of messages (neighbors) based on a graph. These metrics should be transferred into Zoltan2's EvaluatePartition class.
Some ideas:
- Add these metrics to Zolt...*Created by: kddevin*
@mndevec has implemented metrics measuring the number of messages (neighbors) based on a graph. These metrics should be transferred into Zoltan2's EvaluatePartition class.
Some ideas:
- Add these metrics to Zoltan2's GraphMetricValues class
- Create a new class for these metrics
Choice will depend upon how we can reuse the GraphModel constructed by or passed to EvaluatePartition.
@trilinos/zoltan2
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/206KokkosContainers_UnitTest_MPI_1 Time Out on Shepard using GCC 4.8.42016-03-18T08:49:54ZJames WillenbringKokkosContainers_UnitTest_MPI_1 Time Out on Shepard using GCC 4.8.4*Created by: nmhamster*
KokkosContainers_UnitTest_MPI_1 test is timing out on Shepard using GCC 4.8.4.
To reproduce:
- `module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18`
- Configure Trilinos
- `make`
- `make test`
*Created by: nmhamster*
KokkosContainers_UnitTest_MPI_1 test is timing out on Shepard using GCC 4.8.4.
To reproduce:
- `module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18`
- Configure Trilinos
- `make`
- `make test`
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/224Zoltan2: transfer new MappingProblem and MappingSolution interface into HOMME2016-11-29T19:45:19ZJames WillenbringZoltan2: transfer new MappingProblem and MappingSolution interface into HOMME*Created by: kddevin*
Once Zoltan2's new MappingProblem and MappingSolution interface stabilize, we'll need to update HOMME to use them.
*Created by: kddevin*
Once Zoltan2's new MappingProblem and MappingSolution interface stabilize, we'll need to update HOMME to use them.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/219Compilation error for missing operator >= for Kokkos_Views when called by Tp...2016-03-16T23:35:43ZJames WillenbringCompilation error for missing operator >= for Kokkos_Views when called by Tpetra_MultiVector*Created by: mperego*
It's probably related with new Kokkos Views. See my configuration script and error that I get.
Thanks!
Mauro
[do-trilinos-experimental.txt](https://github.com/trilinos/Trilinos/files/176892/do-trilinos-experimen...*Created by: mperego*
It's probably related with new Kokkos Views. See my configuration script and error that I get.
Thanks!
Mauro
[do-trilinos-experimental.txt](https://github.com/trilinos/Trilinos/files/176892/do-trilinos-experimental.txt)
[error.txt](https://github.com/trilinos/Trilinos/files/176893/error.txt)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/201Tpetra: Forbid default Node from being a disabled Node2016-03-10T12:44:55ZJames WillenbringTpetra: Forbid default Node from being a disabled Node*Created by: mhoemmen*
See discussion of Issue #190.
*Created by: mhoemmen*
See discussion of Issue #190.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/220Duplicate lib refs if one uses cmake Trilinos_ENABLE_INSTALL_CMAKE_CONFIG_FIL...2016-03-17T14:58:23ZJames WillenbringDuplicate lib refs if one uses cmake Trilinos_ENABLE_INSTALL_CMAKE_CONFIG_FILES:BOOL=ON*Created by: bathmatt*
First this is a low priority ticket, but when I do an install of trilinos and export the cmake stuff, and use it in my application I get a lot of duplicate lib paths, roughly 30 per lib. I've attached the link li...*Created by: bathmatt*
First this is a low priority ticket, but when I do an install of trilinos and export the cmake stuff, and use it in my application I get a lot of duplicate lib paths, roughly 30 per lib. I've attached the link line in the file. If there could be some consolidation that would be great, it is hard to see the tree from the forest.
[link.txt](https://github.com/trilinos/Trilinos/files/176948/link.txt)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/218Could not find load file STKDoc_testsConfig.cmake2017-11-02T02:27:23ZJames WillenbringCould not find load file STKDoc_testsConfig.cmake*Created by: bavier*
With Trilinos 12.6.1 configured with STK, I get the following error when using the installed cmake configuration fragments:
```
CMake Error at /tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/STK/STKConfig.cmake:143 (...*Created by: bavier*
With Trilinos 12.6.1 configured with STK, I get the following error when using the installed cmake configuration fragments:
```
CMake Error at /tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/STK/STKConfig.cmake:143 (INCLUDE):
include could not find load file:
/tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/STK/../STKDoc_tests/STKDoc_testsConfig.cmake
Call Stack (most recent call first):
/tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/TrilinosCouplings/TrilinosCouplingsConfig.cmake:146 (INCLUDE)
/tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/Trilinos/TrilinosConfig.cmake:146 (INCLUDE)
CMakeLists.txt:66 (FIND_PACKAGE)
CMake Error at /tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/TrilinosCouplings/TrilinosCouplingsConfig.cmake:147 (INCLUDE):
include could not find load file:
/tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/TrilinosCouplings/../STKDoc_tests/STKDoc_testsConfig.cmake
Call Stack (most recent call first):
/tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/Trilinos/TrilinosConfig.cmake:146 (INCLUDE)
CMakeLists.txt:66 (FIND_PACKAGE)
CMake Warning at /tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/Trilinos/TrilinosConfig.cmake:156 (MESSAGE):
Component "STKDoc_tests" NOT found.
Call Stack (most recent call first):
CMakeLists.txt:66 (FIND_PACKAGE)
CMake Error at /tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/Trilinos/TrilinosConfig.cmake:203 (include):
include could not find load file:
/tmp/bavier/trilinos-12.6.1-cce-x86/lib/cmake/Trilinos/../STKDoc_tests/STKDoc_testsConfig.cmake
Call Stack (most recent call first):
CMakeLists.txt:66 (FIND_PACKAGE)
```
STKConfig.cmake lists `STKDoc_tests` in `STK_PACKAGE_LIST` but there is no corresponding STKDoc_testsConfig.cmake.
This patch appears to fix the issue:
```
--- trilinos-12.6.1-Source/packages/stk/stk_doc_tests/CMakeLists.txt 2016-01-12 13:17:58.000000000 -0600
+++ trilinos-12.6.1-Source/packages/stk/stk_doc_tests/CMakeLists.txt 2016-03-16 12:12:46.000118000 -0500
@@ -1,3 +1,4 @@
+TRIBITS_SUBPACKAGE(Doc_tests)
TRIBITS_ADD_TEST_DIRECTORIES(stk_mesh)
@@ -7,3 +8,5 @@
TRIBITS_ADD_TEST_DIRECTORIES(stk_io)
TRIBITS_ADD_TEST_DIRECTORIES(stk_util)
+
+TRIBITS_SUBPACKAGE_POSTPROCESS()
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/203TriUtils build failure with Teuchos disabled2016-03-17T13:25:42ZJames WillenbringTriUtils build failure with Teuchos disabled*Created by: jwillenbring*
TriUtils currently has no stated dependence on Teuchos. It does have a stated dependence on Epetra. Epetra has an optional dependence on Teuchos, so Teuchos is typically available. On Sept 30, TriUtils was mad...*Created by: jwillenbring*
TriUtils currently has no stated dependence on Teuchos. It does have a stated dependence on Epetra. Epetra has an optional dependence on Teuchos, so Teuchos is typically available. On Sept 30, TriUtils was made dependent on Teuchos, although the dependencies file was not updated.
commit 60dd431da705a59158cecc30055b62b593024824
Author: Nico Schlömer nico.schloemer@gmail.com
Date: Tue Sep 30 11:39:16 2014 +0200
```
[TriUtils] char *data_file ==> const char *data_file
Using a `const char*` as function argument allows char literals
as arguments.
Besides this change, this commit contains a bunch of indentation corrections
and replacement of conditional `exit(...)` calls by their TEUCHOS_EXCEPT*
equivalents.
```
Basically the options are to make TriUtils depend on Teuchos, or make the Teuchos calls conditional.
@hkthorn - this is affecting the no C++11 nightly build.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/205Tpetra::CrsGraph::findLocalIndex: Port to use Kokkos2016-03-17T13:36:36ZJames WillenbringTpetra::CrsGraph::findLocalIndex: Port to use Kokkos*Created by: mhoemmen*
@trilinos/tpetra @amklinv
This blocks #41.
It may also help #118.
Tpetra::CrsGraph::findLocalIndex takes the column indices in a row of a CrsGraph / CrsMatrix, and a column index. If the column index exists ...*Created by: mhoemmen*
@trilinos/tpetra @amklinv
This blocks #41.
It may also help #118.
Tpetra::CrsGraph::findLocalIndex takes the column indices in a row of a CrsGraph / CrsMatrix, and a column index. If the column index exists in the row, it returns the relative offset of that column index. Otherwise, it returns a flag value. There are other details (e.g., the search hint) but that's the essence.
We want to make this fully Kokkos-ized -- we want this functionality, marked as a Kokkos device function, so that we can use it in Kokkos::parallel_*.
There is nothing in this method that requires knowing anything about a sparse graph or matrix! It's just search in an array. Thus, if we want to Kokkos-ize this fully, it shouldn't even be an instance method of KokkosSparse::CrsMatrix, nor does it need to take the matrix as an input argument. I don't want to add more instance methods to classes, especially not public ones.
Doing this will get us most of the way to finishing #41. It should also start to address #118, in that it will help us change Tpetra so that dispatch from replace / sumInto to the various storage options will happen at the top level, making the lower-level search code less general and therefore possibly faster.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/208build of Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o fails if you use nvcc and ha...2016-03-19T08:19:49ZJames Willenbringbuild of Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o fails if you use nvcc and have cuda disabled*Created by: bathmatt*
If I don't configure with cuda but still have OMPI_CXX=nvcc_wrapper I get the following. This is on hansen
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV...*Created by: bathmatt*
If I don't configure with cuda but still have OMPI_CXX=nvcc_wrapper I get the following. This is on hansen
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o
/home/mbetten/Trilinos/Trilinos/packages/tpetra/kernels/src/impl/Kokkos_Sparse_impl_spmv.hpp(885): error: namespace "Kokkos" has no member "shfl_down"
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/216Unable to Specify --map-by numa:PE=4 in Trilinos Configuration for MPI Tests2016-03-16T20:06:58ZJames WillenbringUnable to Specify --map-by numa:PE=4 in Trilinos Configuration for MPI Tests*Created by: nmhamster*
@bartlettroscoe I have been specifying `-D MPI_EXEC_POST_NUMPROCS_FLAGS:STRING="--map-by numa:PE=4"` in my Trilinos configuration but it appears that this is being passed to `mpirun` or `mpiexec` as a single argu...*Created by: nmhamster*
@bartlettroscoe I have been specifying `-D MPI_EXEC_POST_NUMPROCS_FLAGS:STRING="--map-by numa:PE=4"` in my Trilinos configuration but it appears that this is being passed to `mpirun` or `mpiexec` as a single argument (i.e. not two arguments).
I'd like to be able to run the Trilinos test in a sensible binding mode for OpenMP on `hansen`, `shiller` and `shepard`.
So far I have backed off to `--bind-to-socket` (a single argument) but it would be better to not have MPI ranks conflicting if we can avoid that (and use the true mapping option).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/209IFPACK2 Compile Failure - GCC 4.8.4 on Hansen Test System2016-03-11T19:39:11ZJames WillenbringIFPACK2 Compile Failure - GCC 4.8.4 on Hansen Test System*Created by: nmhamster*
Reproducer:
`module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18` (Hansen)
Configure Trilinos, Run `make` and output:
```
[ 90%] Building CXX object packages/ifpack2/src/CMakeFiles/ifpack2.dir/Ifpack2_Add...*Created by: nmhamster*
Reproducer:
`module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18` (Hansen)
Configure Trilinos, Run `make` and output:
```
[ 90%] Building CXX object packages/ifpack2/src/CMakeFiles/ifpack2.dir/Ifpack2_AdditiveSchwarz_Serial.cpp.o
In file included from /home/sdhammo/git/trilinos-github-repo/build-devpack-gcc-4.8.4/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_Serial.cpp:50:0:
/home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp: In instantiation of ‘void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >; LocalInverseType = Ifpack2::Preconditioner<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]’:
/home/sdhammo/git/trilinos-github-repo/build-devpack-gcc-4.8.4/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_Serial.cpp:61:1: required from here
/home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1531:75: error: conversion from ‘Teuchos::ArrayRCP<long int>’ to non-scalar type ‘Teuchos::ArrayRCP<int>’ requested
ArrayRCP<local_ordinal_type> perm = sol.getPermutationRCPConst (true);
^
/home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1532:74: error: conversion from ‘Teuchos::ArrayRCP<long int>’ to non-scalar type ‘Teuchos::ArrayRCP<int>’ requested
ArrayRCP<local_ordinal_type> revperm = sol.getPermutationRCPConst ();
^
In file included from /home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_OrderingAlgorithms.hpp:51:0,
from /home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:54,
from /home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:68,
from /home/sdhammo/git/trilinos-github-repo/build-devpack-gcc-4.8.4/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_Serial.cpp:50:
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_AlgRCM.hpp: In instantiation of ‘int Zoltan2::AlgRCM<Adapter>::order(const Teuchos::RCP<Zoltan2::OrderingSolution<typename Adapter::lno_t, typename Adapter::gno_t> >&) [with Adapter = Zoltan2::MatrixAdapter<Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >; typename Adapter::gno_t = long int; typename Adapter::lno_t = int]’:
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:199:7: required from ‘void Zoltan2::OrderingProblem<Adapter>::solve(bool) [with Adapter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >]’
/home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1517:5: required from ‘void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >; LocalInverseType = Ifpack2::Preconditioner<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]’
/home/sdhammo/git/trilinos-github-repo/build-devpack-gcc-4.8.4/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_Serial.cpp:61:1: required from here
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_AlgRCM.hpp:112:69: error: conversion from ‘const Teuchos::ArrayRCP<long int>’ to non-scalar type ‘const Teuchos::ArrayRCP<int>’ requested
const ArrayRCP<lno_t> invPerm = solution->getPermutationRCP(true);
^
In file included from /home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_OrderingAlgorithms.hpp:49:0,
from /home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:54,
from /home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:68,
from /home/sdhammo/git/trilinos-github-repo/build-devpack-gcc-4.8.4/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_Serial.cpp:50:
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_AlgNatural.hpp: In instantiation of ‘int Zoltan2::AlgNatural<Adapter>::order(const Teuchos::RCP<Zoltan2::OrderingSolution<typename Adapter::lno_t, typename Adapter::gno_t> >&) [with Adapter = Zoltan2::MatrixAdapter<Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >; typename Adapter::gno_t = long int; typename Adapter::lno_t = int]’:
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:205:7: required from ‘void Zoltan2::OrderingProblem<Adapter>::solve(bool) [with Adapter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >]’
/home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1517:5: required from ‘void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >; LocalInverseType = Ifpack2::Preconditioner<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]’
/home/sdhammo/git/trilinos-github-repo/build-devpack-gcc-4.8.4/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_Serial.cpp:61:1: required from here
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_AlgNatural.hpp:93:44: error: cannot convert ‘long int*’ to ‘Zoltan2::AlgNatural<Zoltan2::MatrixAdapter<Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > > >::lno_t* {aka int*}’ in initialization
lno_t *perm = solution->getPermutation();
^
In file included from /home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_OrderingAlgorithms.hpp:52:0,
from /home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:54,
from /home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:68,
from /home/sdhammo/git/trilinos-github-repo/build-devpack-gcc-4.8.4/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_Serial.cpp:50:
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_AlgSortedDegree.hpp: In instantiation of ‘int Zoltan2::AlgSortedDegree<Adapter>::order(const Teuchos::RCP<Zoltan2::OrderingSolution<typename Adapter::lno_t, typename Adapter::gno_t> >&) [with Adapter = Zoltan2::MatrixAdapter<Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >; typename Adapter::gno_t = long int; typename Adapter::lno_t = int]’:
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:217:7: required from ‘void Zoltan2::OrderingProblem<Adapter>::solve(bool) [with Adapter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >]’
/home/sdhammo/git/trilinos-github-repo/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1517:5: required from ‘void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >; LocalInverseType = Ifpack2::Preconditioner<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]’
/home/sdhammo/git/trilinos-github-repo/build-devpack-gcc-4.8.4/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_Serial.cpp:61:1: required from here
/home/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/order/Zoltan2_AlgSortedDegree.hpp:91:44: error: cannot convert ‘long int*’ to ‘Zoltan2::AlgSortedDegree<Zoltan2::MatrixAdapter<Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >, Xpetra::RowMatrix<double, int, long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > > >::lno_t* {aka int*}’ in initialization
lno_t *perm = solution->getPermutation();
^
make[2]: *** [packages/ifpack2/src/CMakeFiles/ifpack2.dir/Ifpack2_AdditiveSchwarz_Serial.cpp.o] Error 1
make[1]: *** [packages/ifpack2/src/CMakeFiles/ifpack2.dir/all] Error 2
make: *** [all] Error 2
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/225Zoltan / parmetis test failures2016-07-18T19:45:11ZJames WillenbringZoltan / parmetis test failures*Created by: jwillenbring*
@trilinos/zoltan
I was trying to set up a nightly no-C++11 build using a version of parmetis installed on the SEMS NFS mount. When I run the Zoltan tests, many of the parmetis tests fail.
http://testing.san...*Created by: jwillenbring*
@trilinos/zoltan
I was trying to set up a nightly no-C++11 build using a version of parmetis installed on the SEMS NFS mount. When I run the Zoltan tests, many of the parmetis tests fail.
http://testing.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=2380727
I was able to reproduce these failures on a machine that has access to the NFS mount using the following cmake arguments:
EXTRA_ARGS=$@
cmake \
-DTrilinos_ENABLE_TESTS:BOOL=ON \
-DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON \
-DTrilinos_ENABLE_SECONDARY_TESTED_CODE:BOOL=ON \
-DTrilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON \
-DTrilinos_ENABLE_Teuchos:BOOL= \
-DCMAKE_BUILD_TYPE:STRING=RELEASE \
-DTrilinos_ENABLE_CXX11=OFF \
-DCMAKE_CXX_FLAGS='-O3 -fPIC' \
-DCMAKE_C_FLAGS='-O3 -fPIC' \
-DParMETIS_LIBRARY_DIRS=/projects/install/rhel6-x86_64/sems/tpl/parmetis/4.0.3/gcc/4.4.7/openmpi/1.8.7/lib \
-DParMETIS_INCLUDE_DIRS=/projects/install/rhel6-x86_64/sems/tpl/parmetis/4.0.3/gcc/4.4.7/openmpi/1.8.7/include \
-DTPL_ENABLE_MPI:BOOL=ON \
-DMPI_BASE_DIR:PATH=/projects/install/rhel6-x86_64/sems/compiler/gcc/4.4.7/openmpi/1.8.7 \
-DEpetraExt_BUILD_BTF=O \
-DTeuchos_ENABLE_COMPLEX=ON \
-DNOX_ENABLE_LOCA=ON \
-DTPL_ENABLE_ParMETIS:BOOL=ON \
-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-DTrilinos_ENABLE_Zoltan:BOOL=ON \
$EXTRA_ARGS \
../Trilinos
It is possible, even likely, these errors have something to do with the version of parmetis I am using, the specific installation of parmetis I am using, or random error on my part. Please advise.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/226cuda build of Kokkos_Sparse_MV_impl_spmv takes 25 minutes for 1 file.2016-05-18T19:44:14ZJames Willenbringcuda build of Kokkos_Sparse_MV_impl_spmv takes 25 minutes for 1 file.*Created by: bathmatt*
Can we split up the build of this file? This is all on hansen with
-bash-4.1$ module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18
The build of
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeF...*Created by: bathmatt*
Can we split up the build of this file? This is all on hansen with
-bash-4.1$ module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18
The build of
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Cuda.cpp.o
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o
is very slow under cuda/debug (not -G though). By very slow, 25 minutes.
-bash-4.1$ time make -j
[ 0%] Built target kokkoscore
[ 0%] Built target kokkosalgorithms
[ 0%] Built target kokkoscontainers
[ 33%] Built target teuchoscore
[ 66%] Built target teuchosparameterlist
[ 66%] Built target teuchoscomm
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Cuda.cpp.o
[ 66%] Building CXX object packages/tpetra/kernels/src/CMakeFiles/tpetrakernels.dir/impl/Kokkos_Sparse_MV_impl_spmv_Serial.cpp.o
[ 66%] Linking CXX static library libtpetrakernels.a
[100%] Built target tpetrakernels
real 24m16.402s
user 29m35.614s
sys 0m58.005s
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/227Tpetra range issues in Kokkos subview with new view2016-05-03T23:48:18ZJames WillenbringTpetra range issues in Kokkos subview with new view*Created by: bathmatt*
@mhoemmen @crtrott @hcedwar
I'm getting this error when I have bounds checking on
```
Kokkos::subview bounds errorp=1 | *********** Caught Exception std::exception: Begin Error Report ***********
p=1 | /home/mb...*Created by: bathmatt*
@mhoemmen @crtrott @hcedwar
I'm getting this error when I have bounds checking on
```
Kokkos::subview bounds errorp=1 | *********** Caught Exception std::exception: Begin Error Report ***********
p=1 | /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2136:
p=1 |
p=1 | Throw number = 1
p=1 |
p=1 | Throw test that evaluated to true: static_cast<size_t> (imports.dimension_0 ()) < totalNumImportPackets
p=1 |
p=1 | Tpetra::Distributor::doPosts(3 args): The 'imports' array must have enough entries to hold the expected number of import packets. imports.dimension_0() = 0 < totalNumImportPackets = 63.
p=1 | ************ Caught Exception std::exception: End Error Report ************
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 5293 on node hansen02 exited on signal 11 (Segmentation fault).
```
I've run this code in serial and parallel through valgrind , it is clean
I've run it through cuda-memcheck in parallel and it doesn't show anything except for the exception shown below.
I've run it through cuda-gdb in parallel, I've attached the tracebacks.
I think this is an issue with tpetra now. This is on Hansen, I can make the build public.
On rank 1
```
(cuda-gdb) where
#0 __cxxabiv1::__cxa_throw (obj=0x1f81f6c0,
tinfo=0x1d4b1b60 <_ZTISt13runtime_error@@GLIBCXX_3.4>,
dest=0x49b40c0 <_ZNSt13runtime_errorD1Ev@plt>)
at ../../.././libstdc++-v3/libsupc++/eh_throw.cc:62
#1 0x000000000843950e in Tpetra::Distributor::doPosts<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > > (this=
0x1f7a9610, exports=..., numPackets=3, imports=...)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2131
#2 0x000000000842fb78 in Tpetra::Distributor::doPostsAndWaits<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > > (
this=0x1f7a9610, exports=..., numPackets=3, imports=...)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2009
#3 0x000000000842b41a in Tpetra::Distributor::doReversePostsAndWaits<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::Cuda> > (this=0x1f7a9610, exports=..., numPackets=3, imports=...)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distribut---Type <return> to continue, or q <return> to quit---
or.hpp:2803
#4 0x0000000008429bc2 in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doTransferNew
(this=0x1f7aea40, src=..., CM=Tpetra::REPLACE, numSameIDs=231,
permuteToLIDs_=..., permuteFromLIDs_=..., remoteLIDs_=...,
exportLIDs_=..., distor=...,
revOp=Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::DoReverse)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:856
#5 0x0000000008428519 in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doTransfer (
this=0x1f7aea40, src=..., CM=Tpetra::REPLACE, numSameIDs=231,
permuteToLIDs_=..., permuteFromLIDs_=..., remoteLIDs_=...,
exportLIDs_=..., distor=...,
revOp=Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::DoReverse)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:397
#6 0x00000000084270bd in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doImport (
this=0x1f7aea40, source=..., exporter=..., CM=Tpetra::REPLACE)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObjec---Type <return> to continue, or q <return> to quit---
t_def.hpp:333
#7 0x0000000006b7b330 in panzer::DOFManager<int, int>::buildGlobalUnknowns (
this=0x1f7a3780, geomPattern=...)
at /home/mbetten/Trilinos/Trilinos/packages/panzer/dof-mgr/src/Panzer_DOFManager_impl.hpp:505
#8 0x0000000006b7a074 in panzer::DOFManager<int, int>::buildGlobalUnknowns (
this=0x1f7a3780)
at /home/mbetten/Trilinos/Trilinos/packages/panzer/dof-mgr/src/Panzer_DOFManager_impl.hpp:317
#9 0x00000000064ad919 in panzer::DOFManagerFactory<int, int>::buildUniqueGlobalIndexer<panzer::DOFManager<int, int> > (this=0x7fffffff6a80, mpiComm=...,
physicsBlocks=..., connMngr=..., fieldOrder=...)
at /home/mbetten/Trilinos/Trilinos/packages/panzer/disc-fe/src/Panzer_DOFManagerFactory_impl.hpp:148
#10 0x00000000064ac4ba in panzer::DOFManagerFactory<int, int>::buildUniqueGlobalIndexer (this=0x7fffffff6a80, mpiComm=..., physicsBlocks=..., connMngr=...,
fieldOrder=...)
at /home/mbetten/Trilinos/Trilinos/packages/panzer/disc-fe/src/Panzer_DOFManagerFactory_impl.hpp:66
#11 0x0000000004bef189 in drekar::UniqueGlobalIndexerFactory<int>::buildUniqueGlobalIndexer (this=0x7fffffff6cb0, mpiComm=..., physicsBlocks_exp=...,
```
on rank 0
```
#1 0x00007fffeca13e15 in abort () from /lib64/libc.so.6
#2 0x0000000008d2e9c2 in Kokkos::Impl::host_abort (
message=0x9ec8f16 <Kokkos::Experimental::(anonymous namespace)::AllowPadding+25640> "Kokkos::subview bounds error")
at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/Kokkos_Error.cpp:64
#3 0x0000000004ff6eca in Kokkos::abort (
message=0x9ec8f16 <Kokkos::Experimental::(anonymous namespace)::AllowPadding+25640> "Kokkos::subview bounds error")
at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/Kokkos_Error.hpp:74
#4 0x0000000007ca65a2 in error<0ul, std::pair<unsigned long, unsigned long> >
(dim=..., this=0x7fffffff4630)
at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/KokkosExp_ViewMapping.hpp:552
#5 Kokkos::Experimental::Impl::SubviewExtents<1u, 1u>::SubviewExtents<0ul, std::pair<unsigned long, unsigned long> > (this=0x7fffffff4630, dim=...)
at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/KokkosExp_ViewMapping.hpp:587
#6 0x00000000084434ba in Kokkos::Experimental::Impl::ViewMapping<void, Kokkos::Experimental::ViewTraits<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, std::pair<unsigned lo---Type <return> to continue, or q <return> to quit---
ng, unsigned long> >::assign<Kokkos::Experimental::ViewTraits<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> > >(Kokkos::Experimental::Impl::ViewMapping<Kokkos::Experimental::ViewTraits<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> ><void> >&, Kokkos::Experimental::Impl::ViewMapping<Kokkos::Experimental::ViewTraits<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> ><void> > const&, std::pair<unsigned long, unsigned long>) (dst=..., src=...,
args#0=...)
at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/KokkosExp_ViewMapping.hpp:2788
#7 0x00000000084417c5 in Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >::View<int const*<Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, std::pair<unsigned long, unsigned long>> (this=0x7fffffff49e0, src_view=..., arg0=...)
at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/KokkosExp_View.hpp:1160
#8 0x000000000843f2bf in Kokkos::Experimental::subview<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u>, std::pair<unsigned long, unsigned long> > (src=...)
at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/KokkosExp_View.hpp:1462
---Type <return> to continue, or q <return> to quit---
#9 0x000000000843d56e in Kokkos::Compat::subview_offset<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, unsigned long> (view=..., offset=0,
size=63)
at /home/mbetten/Trilinos/Trilinos/packages/teuchos/kokkoscompat/src/KokkosCompat_View.hpp:221
#10 0x0000000008439d5d in Tpetra::Distributor::doPosts<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > > (this=
0x2b881130, exports=..., numPackets=3, imports=...)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2263
#11 0x000000000842fb78 in Tpetra::Distributor::doPostsAndWaits<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > > (
this=0x2b881130, exports=..., numPackets=3, imports=...)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2009
#12 0x000000000842b41a in Tpetra::Distributor::doReversePostsAndWaits<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*---Type <return> to continue, or q <return> to quit---
, Kokkos::Cuda> > (this=0x2b881130, exports=..., numPackets=3, imports=...)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2803
#13 0x0000000008429bc2 in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doTransferNew
(this=0x2b875ec0, src=..., CM=Tpetra::REPLACE, numSameIDs=10,
permuteToLIDs_=..., permuteFromLIDs_=..., remoteLIDs_=...,
exportLIDs_=..., distor=...,
revOp=Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::DoReverse)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:856
#14 0x0000000008428519 in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doTransfer (
this=0x2b875ec0, src=..., CM=Tpetra::REPLACE, numSameIDs=10,
permuteToLIDs_=..., permuteFromLIDs_=..., remoteLIDs_=...,
exportLIDs_=..., distor=...,
revOp=Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::DoReverse)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:397
#15 0x00000000084270bd in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doImport (
---Type <return> to continue, or q <return> to quit---
this=0x2b875ec0, source=..., exporter=..., CM=Tpetra::REPLACE)
at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:333
#16 0x0000000006b7b330 in panzer::DOFManager<int, int>::buildGlobalUnknowns (
this=0x2b87f8c0, geomPattern=...)
at /home/mbetten/Trilinos/Trilinos/packages/panzer/dof-mgr/src/Panzer_DOFManager_impl.hpp:505
#17 0x0000000006b7a074 in panzer::DOFManager<int, int>::buildGlobalUnknowns (
this=0x2b87f8c0)
at /home/mbetten/Trilinos/Trilinos/packages/panzer/dof-mgr/src/Panzer_DOFManager_impl.hpp:317
#18 0x00000000064ad919 in panzer::DOFManagerFactory<int, int>::buildUniqueGlobalIndexer<panzer::DOFManager<int, int> > (this=0x7fffffff6a90, mpiComm=...,
physicsBlocks=..., connMngr=..., fieldOrder=...)
at /home/mbetten/Trilinos/Trilinos/packages/panzer/disc-fe/src/Panzer_DOFManagerFactory_impl.hpp:148
#19 0x00000000064ac4ba in panzer::DOFManagerFactory<int, int>::buildUniqueGlobalIndexer (this=0x7fffffff6a90, mpiComm=..., physicsBlocks=..., connMngr=...,
fieldOrder=...)
at /home/mbetten/Trilinos/Trilinos/packages/panzer/disc-fe/src/Panzer_DOFManagerFactory_impl.hpp:66
#20 0x0000000004bef189 in drekar::UniqueGlobalIndexerFactory<int>::buildUniqueGlobalIndexer (this=0x7fffffff6cc0, mpiComm=..., physicsBlocks_exp=...,
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/228Trilinos Compilation with GCC 4.8 using AVX and AVX2 Fails2016-03-21T23:32:07ZJames WillenbringTrilinos Compilation with GCC 4.8 using AVX and AVX2 Fails*Created by: nmhamster*
Use of Trilinos in NALU with GCC 4.8 fails when using `-mavx`, `-mavx2` and `-march=core-avx2` flags.
Error produced is:
```
...
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/in...*Created by: nmhamster*
Use of Trilinos in NALU with GCC 4.8 fails when using `-mavx`, `-mavx2` and `-march=core-avx2` flags.
Error produced is:
```
...
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include/mmintrin.h: In function ‘__m64 _mm_xor_si64(__m64, __m64)’:
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include/mmintrin.h:761:41: error: cannot convert ‘__m64 {aka int}’ to ‘__vector(2) int’ for argument ‘1’ to ‘__vector(2) int __builtin_ia32_pxor(__vector(2) int, __vector(2) int)’
return __builtin_ia32_pxor (__m1, __m2);
^
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include/mmintrin.h: In function ‘__m64 _mm_cmpeq_pi8(__m64, __m64)’:
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include/mmintrin.h:775:68: error: cannot convert ‘__v8qi {aka char}’ to ‘__vector(8) char’ for argument ‘1’ to ‘__vector(8) char __builtin_ia32_pcmpeqb(__vector(8) char, __vector(8) char)’
return (__m64) __builtin_ia32_pcmpeqb ((__v8qi)__m1, (__v8qi)__m2);
^
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include/mmintrin.h: In function ‘__m64 _mm_cmpgt_pi8(__m64, __m64)’:
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include/mmintrin.h:787:68: error: cannot convert ‘__v8qi {aka char}’ to ‘__vector(8) char’ for argument ‘1’ to ‘__vector(8) char __builtin_ia32_pcmpgtb(__vector(8) char, __vector(8) char)’
return (__m64) __builtin_ia32_pcmpgtb ((__v8qi)__m1, (__v8qi)__m2);
^
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include/mmintrin.h: In function ‘__m64 _mm_cmpeq_pi16(__m64, __m64)’:
/home/projects/x86-64-haswell/gnu/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include/mmintrin.h:801:68: error: cannot convert ‘__v4hi {aka short int}’ to ‘__vector(4) short int’ for argument ‘1’ to ‘__vector(4) short int __builtin_ia32_pcmpeqw(__vector(4) short int, __vector(4) short int)’
return (__m64) __builtin_ia32_pcmpeqw ((__v4hi)__m1, (__v4hi)__m2);
...
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/232KokkosKernels: Make sure sparse mat-vec instantiations use same OFFSET_TYPE t...2016-03-19T08:50:26ZJames WillenbringKokkosKernels: Make sure sparse mat-vec instantiations use same OFFSET_TYPE that Tpetra does*Created by: mhoemmen*
@trilinos/tpetra KokkosKernels does explicit instantiations (not through the ETI system -- it's a different approach that works whether or not ETI is enabled) for the sparse matrix-vector multiply kernels that Tpe...*Created by: mhoemmen*
@trilinos/tpetra KokkosKernels does explicit instantiations (not through the ETI system -- it's a different approach that works whether or not ETI is enabled) for the sparse matrix-vector multiply kernels that Tpetra and downstream packages use. The instantiations specify the offset type (type of the `ptr` array in the CSR `ptr`, `ind`, `val` triple) explicitly as `size_t` for all device types, including for CUDA. However, with CUDA, Kokkos may use a different size_type than `size_t` -- in particular, it may use a 32-bit integer (such as `int`). This means that the instantiations are getting built, but not getting used!
Fixing this may not actually fix #226, but certainly goes along with its spirit.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/231KokkosKernels: Don't need both CudaSpace & CudaUVMSpace instantiations2016-03-19T08:19:49ZJames WillenbringKokkosKernels: Don't need both CudaSpace & CudaUVMSpace instantiations*Created by: mhoemmen*
@trilinos/tpetra
See #226. CUDA instantiations of sparse matrix-vector multiply kernel take a long time. I found that the kernels used by Tpetra for iterative solvers (sparse mat-vec and (multi)vector operatio...*Created by: mhoemmen*
@trilinos/tpetra
See #226. CUDA instantiations of sparse matrix-vector multiply kernel take a long time. I found that the kernels used by Tpetra for iterative solvers (sparse mat-vec and (multi)vector operations) are getting instantiated for both `Device<Cuda, CudaSpace>` and `Device<Cuda, CudaUVMSpace>`. Tpetra and downstream packages only need the latter. So, it's building twice as much code as needed.
KokkosKernels instantiates these kernels in a different way than Trilinos' ETI. With KokkosKernels, one can use template parameter combinations that haven't been instantiated. Thus, getting rid of instantiations that Tpetra and downstream packages don't use won't break any tests, examples, or downstream code.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/233KokkosKernels: Make sure sparse mat-vec instantiations use same OFFSET_TYPE t...2016-03-19T07:33:46ZJames WillenbringKokkosKernels: Make sure sparse mat-vec instantiations use same OFFSET_TYPE that Tpetra does*Created by: mhoemmen*
@trilinos/tpetra KokkosKernels does explicit instantiations (not through the ETI system -- it's a different approach that works whether or not ETI is enabled) for the sparse matrix-vector multiply kernels that Tpe...*Created by: mhoemmen*
@trilinos/tpetra KokkosKernels does explicit instantiations (not through the ETI system -- it's a different approach that works whether or not ETI is enabled) for the sparse matrix-vector multiply kernels that Tpetra and downstream packages use. The instantiations specify the offset type (type of the `ptr` array in the CSR `ptr`, `ind`, `val` triple) explicitly as `size_t` for all device types, including for CUDA. However, with CUDA, Kokkos may use a different size_type than `size_t` -- in particular, it may use a 32-bit integer (such as `int`). This means that the instantiations are getting built, but not getting used!
Fixing this may not actually fix #226, but certainly goes along with its spirit.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/235Tpetra: Deprecate ArrayRCP version of getLocalDiagOffsets from BlockCrsMatrix2016-03-20T01:32:05ZJames WillenbringTpetra: Deprecate ArrayRCP version of getLocalDiagOffsets from BlockCrsMatrix*Created by: mhoemmen*
@trilinos/tpetra
BlockCrsMatrix::getLocalDiagCopy (which uses the output of getLocalDiagOffsets) takes the offsets in as Kokkos::View, not Teuchos::ArrayRCP. Thus, we don't need the latter version of getLocalDi...*Created by: mhoemmen*
@trilinos/tpetra
BlockCrsMatrix::getLocalDiagCopy (which uses the output of getLocalDiagOffsets) takes the offsets in as Kokkos::View, not Teuchos::ArrayRCP. Thus, we don't need the latter version of getLocalDiagOffsets, so we can deprecate it.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/236Tpetra::CrsGraph::getLocalDiagOffsets: Deprecate ArrayRCP version2016-11-04T17:05:27ZJames WillenbringTpetra::CrsGraph::getLocalDiagOffsets: Deprecate ArrayRCP version*Created by: mhoemmen*
@trilinos/tpetra
BlockCrsMatrix::getLocalDiagCopy (which uses the output of getLocalDiagOffsets) takes the offsets in as Kokkos::View, not Teuchos::ArrayRCP. Thus, we don't need the latter version of getLocalDiag...*Created by: mhoemmen*
@trilinos/tpetra
BlockCrsMatrix::getLocalDiagCopy (which uses the output of getLocalDiagOffsets) takes the offsets in as Kokkos::View, not Teuchos::ArrayRCP. Thus, we don't need the latter version of getLocalDiagOffsets, so we can deprecate it.
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/234Ifpack2::Chebyshev: "chebyshev: boost factor" should be same type as magnitud...2016-08-15T03:10:17ZJames WillenbringIfpack2::Chebyshev: "chebyshev: boost factor" should be same type as magnitude of Scalar*Created by: mhoemmen*
@trilinos/ifpack2 @jhux2
I'm going to be "that guy" and point out a tiny issue with commit 0c051216295e1ec5abeae880de2bae58859a8549, namely that Ifpack2::Details::Chebyshev should accept the new "chebyshev: boost...*Created by: mhoemmen*
@trilinos/ifpack2 @jhux2
I'm going to be "that guy" and point out a tiny issue with commit 0c051216295e1ec5abeae880de2bae58859a8549, namely that Ifpack2::Details::Chebyshev should accept the new "chebyshev: boost factor" parameter either as `Teuchos::ScalarTraits<Scalar>::magnitudeType`, or as `double`. For example, if `Scalar = float`, users might reasonably expect to pass this in as `float`. However, in that case, the ParameterList would reject the parameter request, because the second argument of get() (the default value of the parameter) needs to have the same type as the stored value.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/240Add InvRowSums method to Tpetra 2019-02-05T21:42:37ZJames WillenbringAdd InvRowSums method to Tpetra *Created by: ikalash*
It looks like Epetra has an InvRowSums method, but not Tpetra. Would it be possible to add the InvRowSums method to Tpetra? It will faciliate some scaling implementations in Albany.
*Created by: ikalash*
It looks like Epetra has an InvRowSums method, but not Tpetra. Would it be possible to add the InvRowSums method to Tpetra? It will faciliate some scaling implementations in Albany.
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/237Tpetra::BlockCrsMatrix::getLocalDiagCopy (1-arg): Get rid of debug output2016-03-23T23:12:02ZJames WillenbringTpetra::BlockCrsMatrix::getLocalDiagCopy (1-arg): Get rid of debug output*Created by: mhoemmen*
@trilinos/tpetra @amklinv
Tpetra::Experimental::BlockCrsMatrix::getLocalDiagCopy (the 1-argument version) has debug output (to `std::cout`) in it, one line per diagonal entry on each process. I noticed this whe...*Created by: mhoemmen*
@trilinos/tpetra @amklinv
Tpetra::Experimental::BlockCrsMatrix::getLocalDiagCopy (the 1-argument version) has debug output (to `std::cout`) in it, one line per diagonal entry on each process. I noticed this when a fix for #235 brought up a deprecated warning in the method.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/239TeuchosCore_ScalarTraits_test_MPI_1 Tests Fail on POWER8 with GCC 4.9.2 and C...2016-03-23T19:44:32ZJames WillenbringTeuchosCore_ScalarTraits_test_MPI_1 Tests Fail on POWER8 with GCC 4.9.2 and CUDA 7.5*Created by: nmhamster*
@bartlettroscoe - I am getting these errors in the Trilinos bring up on POWER8 with GCC 4.9.2 and CUDA 7.5. This is being tested on the `white` Sandia system. Just want to check in whether these would be expected...*Created by: nmhamster*
@bartlettroscoe - I am getting these errors in the Trilinos bring up on POWER8 with GCC 4.9.2 and CUDA 7.5. This is being tested on the `white` Sandia system. Just want to check in whether these would be expected failures or not?
```
34: Testing: Teuchos::ScalarTraits<float> ...
34: Type chain (ascending) : float -> double
34: Type chain (descending): float
34:
34: Testing that squareroot(NaN) == NaN! ...
34: squareroot(nan) = nan == nan : failed
34:
34: Testing that squareroot(-NaN) == NaN! ...
34: squareroot(-nan) = -nan == nan : failed
34:
34: Testing that squareroot(-1) == NaN! ...
34: squareroot(-1) = nan == nan : failed
34:
34: Testing: Teuchos::ScalarTraits<double> ...
34: Type chain (ascending) : double
34: Type chain (descending): double -> float
34:
34: Testing that squareroot(NaN) == NaN! ...
34: squareroot(nan) = nan == nan : failed
34:
34: Testing that squareroot(-NaN) == NaN! ...
34: squareroot(-nan) = -nan == nan : failed
34:
34: Testing that squareroot(-1) == NaN! ...
34: squareroot(-1) = nan == nan : failed
```
Advanced Technology Test Bed Issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/243Ifpack2: Jacobi tests fail with 2 OpenMP threads, pass with 12016-05-11T15:49:53ZJames WillenbringIfpack2: Jacobi tests fail with 2 OpenMP threads, pass with 1*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/tpetra
I was warming up by working on #237 (easy fix), and found that some of Ifpack2's Jacobi tests fail with OpenMP, when OMP_NUM_THREADS=2, but pass when OMP_NUM_THREADS=1. The di...*Created by: mhoemmen*
@trilinos/ifpack2 @trilinos/tpetra
I was warming up by working on #237 (easy fix), and found that some of Ifpack2's Jacobi tests fail with OpenMP, when OMP_NUM_THREADS=2, but pass when OMP_NUM_THREADS=1. The difference in iteration counts was significant enough to be a real bug. My fix for #237 didn't affect it.
I'm wondering if there's something wrong with diagonal extraction (Tpetra::CrsMatrix::getLocalDiagCopy), but I'm not sure.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/238Zoltan2 Compilation Issue on POWER8 w/ CUDA 7.52016-03-22T02:24:47ZJames WillenbringZoltan2 Compilation Issue on POWER8 w/ CUDA 7.5*Created by: nmhamster*
@trilinos/zoltan2 I am getting this error with a Trilinos VOTD on `white.sandia.gov`. It looks like there may be a preprocessor issue. Have you guys seen something like this before?
```
/ascldap/users/sdhammo/gi...*Created by: nmhamster*
@trilinos/zoltan2 I am getting this error with a Trilinos VOTD on `white.sandia.gov`. It looks like there may be a preprocessor issue. Have you guys seen something like this before?
```
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgMultiJagged.hpp: In member function ‘void Zoltan2::AlgMJ<mj_scalar_t, mj_lno_t, mj_gno_t, mj_part_t>::mj_1D_part(mj_scalar_t*, mj_scalar_t, mj_part_t, mj_part_t, mj_scalar_t*, mj_part_t, std::vector<mj_part_t>&)’:
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgMultiJagged.hpp:2838:1: error: expected primary-expression before ‘}’ token
#endif
^
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgMultiJagged.hpp: In member function ‘void Zoltan2::AlgMJ<mj_scalar_t, mj_lno_t, mj_gno_t, mj_part_t>::mj_get_new_cut_coordinates(const size_t&, const mj_part_t&, const mj_scalar_t&, const mj_scalar_t&, const mj_scalar_t&, const mj_scalar_t&, mj_scalar_t*, const mj_scalar_t*, const mj_scalar_t*, bool*, mj_scalar_t*, mj_scalar_t*, mj_scalar_t*, mj_scalar_t*, mj_scalar_t*, mj_scalar_t*, mj_scalar_t*, mj_scalar_t*, mj_scalar_t*, mj_part_t*, mj_part_t&)’:
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgMultiJagged.hpp:3800:1: error: expected primary-expression before ‘}’ token
#endif
^
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/test/helpers/GeometricGenerator.hpp: In constructor ‘GeometricGen::GeometricGenerator<T1, T2, T3, T4>::GeometricGenerator(Teuchos::ParameterList&, const Teuchos::RCP<const Teuchos::Comm<int> >&)’:
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/test/helpers/GeometricGenerator.hpp:1825:1: error: for statement expected before ‘case’
case 1:
^
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/test/helpers/GeometricGenerator.hpp:1833:1: error: for statement expected before ‘case’
case 2:
^
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/test/helpers/GeometricGenerator.hpp:1841:1: error: for statement expected before ‘case’
case 3:
^
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/test/helpers/GeometricGenerator.hpp: In instantiation of ‘GeometricGen::GeometricGenerator<T1, T2, T3, T4>::GeometricGenerator(Teuchos::ParameterList&, const Teuchos::RCP<const Teuchos::Comm<int> >&) [with scalar_t = double; lno_t = int; gno_t = int; node_t = Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda>]’:
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/test/helpers/UserInputForTests.hpp:885:116: required from here
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/zoltan2/test/helpers/GeometricGenerator.hpp:1695:21: warning: non-constant array new length must be specified without parentheses around the type-id [-Wvla]
this->coords[i] = new scalar_t[myPointCount];
^
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/teuchos/numerics/src/Teuchos_SerialDenseMatrix.hpp: In instantiation of ‘Teuchos::SerialDenseMatrix<OrdinalType, ScalarType>& Teuchos::SerialDenseMatrix<OrdinalType, ScalarType>::operator=(const Teuchos::SerialDenseMatrix<OrdinalType, ScalarType>&) [with OrdinalType = int; ScalarType = double]’:
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/galeri/src-xpetra/Galeri_Elasticity3DProblem.hpp:219:4: required from ‘Teuchos::RCP<Matrix> Galeri::Xpetra::Elasticity3DProblem<Scalar, LocalOrdinal, GlobalOrdinal, Map, Matrix, MultiVector>::BuildMatrix() [with Scalar = double; LocalOrdinal = int; GlobalOrdinal = int; Map = Tpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda> >; Matrix = Tpetra::CrsMatrix<>; MultiVector = Tpetra::MultiVector<double, int, int>]’
/tmp/tmpxft_0000f6a0_00000000-4_AlltoAll.cudafe1.stub.c:12:27: required from here
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/teuchos/numerics/src/Teuchos_SerialDenseMatrix.hpp:659:11: warning: non-constant array new length must be specified without parentheses around the type-id [-Wvla]
values_ = new ScalarType[newsize];
^
/ascldap/users/sdhammo/git/trilinos-github-repo/packages/teuchos/numerics/src/Teuchos_SerialDenseMatrix.hpp:679:11: warning: non-constant array new length must be specified without parentheses around the type-id [-Wvla]
values_ = new ScalarType[newsize];
^
make[2]: *** [packages/zoltan2/test/unit/CMakeFiles/Zoltan2_AlltoAll.dir/util/AlltoAll.cpp.o] Error 1
make[1]: *** [packages/zoltan2/test/unit/CMakeFiles/Zoltan2_AlltoAll.dir/all] Error 2
make: *** [all] Error 2
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/245Amesos(2): SuperLUDist 5.0 is incompatible with current Amesos(2) interface2016-03-30T15:35:29ZJames WillenbringAmesos(2): SuperLUDist 5.0 is incompatible with current Amesos(2) interface*Created by: amklinv*
@trilinos/amesos2 @jdbooth @jwillenbring @srajama1 @bmpersc
Sherry recently renamed the superlu_options_t struct to superlu_dist_options_t, which means SuperLUDist 5.0 does not work with our current Amesos and Am...*Created by: amklinv*
@trilinos/amesos2 @jdbooth @jwillenbring @srajama1 @bmpersc
Sherry recently renamed the superlu_options_t struct to superlu_dist_options_t, which means SuperLUDist 5.0 does not work with our current Amesos and Amesos2 interfaces. I'm planning to address this issue the same way Siva addressed the issue of the LUstructInit parameters changing in Verson 4.0. I will add a check to FindTPLSuperLUDist.cmake to see whether we should use superlu_options_t or superlu_dist_options_t, then wrap all occurances of superlu_options_t with an ifdef.
Please let me know if this does not seem like a reasonable approach. We need to fix this soon in order to meet our xSDK release deadline.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/244Zoltan2_AlgScotch invalid cast with Sacado Vector2016-09-01T15:52:57ZJames WillenbringZoltan2_AlgScotch invalid cast with Sacado Vector*Created by: bavier*
Configuring Trilinos at 17ccfa78c106fd86812563a4ab3d50ceace9c13a with:
```
cmake \
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=YES \
-DTrilinos_ENABLE_Stokhos:BOOL=YES \
-DTrilinos_ENABLE_Zoltan2:BOOL...*Created by: bavier*
Configuring Trilinos at 17ccfa78c106fd86812563a4ab3d50ceace9c13a with:
```
cmake \
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=YES \
-DTrilinos_ENABLE_Stokhos:BOOL=YES \
-DTrilinos_ENABLE_Zoltan2:BOOL=YES \
-DTrilinos_ENABLE_Ifpack2:BOOL=YES \
-DTrilinos_ENABLE_Sacado:BOOL=YES \
-DTrilinos_ENABLE_OpenMP:BOOL=YES \
-DKokkos_ENABLE_OpenMP:BOOL=YES \
-DTPL_ENABLE_Scotch:BOOL=YES \
..
```
results in the following two build errors:
```
[ 90%] Building CXX object packages/stokhos/src/CMakeFiles/stokhos_ifpack2.dir/Ifpack2_AdditiveSchwarz_MP_Vector_OpenMP.cpp.o
In file included from /ptmp/Trilinos/packages/zoltan2/src/algorithms/order/Zoltan2_OrderingAlgorithms.hpp:54:0,
from /ptmp/Trilinos/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:54,
from /ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:68,
from /ptmp/Trilinos/build/build-ifpack2-zoltan2-scotch/packages/stokhos/src/Ifpack2_AdditiveSchwarz_MP_Vector_OpenMP.cpp:51:
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp: In instantiation of 'void Zoltan2::AlgPTScotch<Adapter>::scale_weights(size_t, Zoltan2::StridedData<typename Adapter::lno_t, typename Adapter::scalar_t>&, Zoltan2::SCOTCH_Num*) [with Ada
pter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 4, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedS
torage<int, double, 4, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >; size_t = long unsigned int; typename Adapter::scalar_t = Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 4, Kokkos::OpenMP> >; typename Adapt
er::lno_t = int; Zoltan2::SCOTCH_Num = int]':
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:740:42: required from 'int Zoltan2::AlgPTScotch<Adapter>::order(const Teuchos::RCP<Zoltan2::OrderingSolution<typename Adapter::lno_t, typename Adapter::gno_t> >&) [with Adapter = Zoltan
2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 4, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, d
ouble, 4, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >; typename Adapter::gno_t = int; typename Adapter::lno_t = int]'
/ptmp/Trilinos/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:237:5: required from 'void Zoltan2::OrderingProblem<Adapter>::solve(bool) [with Adapter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int,
double, 4, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 4, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP>
> >]'
/ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1517:5: required from 'void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 4, Kokk
os::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >; LocalInverseType = Ifpack2::Preconditioner<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 4, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos:
:OpenMP> >]'
/ptmp/Trilinos/build/build-ifpack2-zoltan2-scotch/packages/stokhos/src/Ifpack2_AdditiveSchwarz_MP_Vector_OpenMP.cpp:55:1: required from here
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:565:32: error: invalid cast from type 'Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 4, Kokkos::OpenMP> >' to type 'double'
double fw = double(fwgts[i]);
^
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:595:50: error: invalid cast from type 'Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 4, Kokkos::OpenMP> >' to type 'double'
iwgts[i] = (SCOTCH_Num) ceil(double(fwgts[i])*scale);
^
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp: In instantiation of 'void Zoltan2::AlgPTScotch<Adapter>::scale_weights(size_t, Zoltan2::StridedData<typename Adapter::lno_t, typename Adapter::scalar_t>&, Zoltan2::SCOTCH_Num*) [with Ada
pter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixed
Storage<int, double, 16, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >; size_t = long unsigned int; typename Adapter::scalar_t = Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::OpenMP> >; typename Ad
apter::lno_t = int; Zoltan2::SCOTCH_Num = int]':
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:740:42: required from 'int Zoltan2::AlgPTScotch<Adapter>::order(const Teuchos::RCP<Zoltan2::OrderingSolution<typename Adapter::lno_t, typename Adapter::gno_t> >&) [with Adapter = Zoltan
2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int,
double, 16, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >; typename Adapter::gno_t = int; typename Adapter::lno_t = int]'
/ptmp/Trilinos/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:237:5: required from 'void Zoltan2::OrderingProblem<Adapter>::solve(bool) [with Adapter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int,
double, 16, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP
> > >]'
/ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1517:5: required from 'void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kok
kos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >; LocalInverseType = Ifpack2::Preconditioner<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokko
s::OpenMP> >]'
/ptmp/Trilinos/build/build-ifpack2-zoltan2-scotch/packages/stokhos/src/Ifpack2_AdditiveSchwarz_MP_Vector_OpenMP.cpp:55:1: required from here
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:565:32: error: invalid cast from type 'Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::OpenMP> >' to type 'double'
double fw = double(fwgts[i]);
^
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:595:50: error: invalid cast from type 'Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::OpenMP> >' to type 'double'
iwgts[i] = (SCOTCH_Num) ceil(double(fwgts[i])*scale);
^
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp: In instantiation of 'void Zoltan2::AlgPTScotch<Adapter>::scale_weights(size_t, Zoltan2::StridedData<typename Adapter::lno_t, typename Adapter::scalar_t>&, Zoltan2::SCOTCH_Num*) [with Ada
pter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 32, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixed
Storage<int, double, 32, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >; size_t = long unsigned int; typename Adapter::scalar_t = Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 32, Kokkos::OpenMP> >; typename Ad
apter::lno_t = int; Zoltan2::SCOTCH_Num = int]':
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:740:42: required from 'int Zoltan2::AlgPTScotch<Adapter>::order(const Teuchos::RCP<Zoltan2::OrderingSolution<typename Adapter::lno_t, typename Adapter::gno_t> >&) [with Adapter = Zoltan
2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 32, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int,
double, 32, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >; typename Adapter::gno_t = int; typename Adapter::lno_t = int]'
/ptmp/Trilinos/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:237:5: required from 'void Zoltan2::OrderingProblem<Adapter>::solve(bool) [with Adapter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int,
double, 32, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 32, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP
> > >]'
/ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1517:5: required from 'void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 32, Kok
kos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >; LocalInverseType = Ifpack2::Preconditioner<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 32, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokko
s::OpenMP> >]'
/ptmp/Trilinos/build/build-ifpack2-zoltan2-scotch/packages/stokhos/src/Ifpack2_AdditiveSchwarz_MP_Vector_OpenMP.cpp:55:1: required from here
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:565:32: error: invalid cast from type 'Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 32, Kokkos::OpenMP> >' to type 'double'
double fw = double(fwgts[i]);
^
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:595:50: error: invalid cast from type 'Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 32, Kokkos::OpenMP> >' to type 'double'
iwgts[i] = (SCOTCH_Num) ceil(double(fwgts[i])*scale);
^
packages/stokhos/src/CMakeFiles/stokhos_ifpack2.dir/build.make:86: recipe for target 'packages/stokhos/src/CMakeFiles/stokhos_ifpack2.dir/Ifpack2_AdditiveSchwarz_MP_Vector_OpenMP.cpp.o' failed
make[2]: *** [packages/stokhos/src/CMakeFiles/stokhos_ifpack2.dir/Ifpack2_AdditiveSchwarz_MP_Vector_OpenMP.cpp.o] Error 1
```
and
```
[ 91%] Building CXX object packages/stokhos/src/CMakeFiles/stokhos_ifpack2.dir/Ifpack2_AdditiveSchwarz_UQ_PCE_OpenMP.cpp.o
In file included from /ptmp/Trilinos/packages/zoltan2/src/algorithms/order/Zoltan2_OrderingAlgorithms.hpp:54:0,
from /ptmp/Trilinos/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:54,
from /ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:68,
from /ptmp/Trilinos/build/build-ifpack2-zoltan2-scotch/packages/stokhos/src/Ifpack2_AdditiveSchwarz_UQ_PCE_OpenMP.cpp:51:
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp: In instantiation of 'void Zoltan2::AlgPTScotch<Adapter>::scale_weights(size_t, Zoltan2::StridedData<typename Adapter::lno_t, typename Adapter::scalar_t>&, Zoltan2::SCOTCH_Num*) [with Ada
pter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::UQ::PCE<Stokhos::DynamicStorage<int, doubl
e, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >; size_t = long unsigned int; typename Adapter::scalar_t = Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::OpenMP> >; typename Adapter::lno_t = int; Zoltan2::SCOT
CH_Num = int]':
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:740:42: required from 'int Zoltan2::AlgPTScotch<Adapter>::order(const Teuchos::RCP<Zoltan2::OrderingSolution<typename Adapter::lno_t, typename Adapter::gno_t> >&) [with Adapter = Zoltan
2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::Op
enMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >; typename Adapter::gno_t = int; typename Adapter::lno_t = int]'
/ptmp/Trilinos/packages/zoltan2/src/problems/Zoltan2_OrderingProblem.hpp:237:5: required from 'void Zoltan2::OrderingProblem<Adapter>::solve(bool) [with Adapter = Zoltan2::XpetraRowMatrixAdapter<Xpetra::RowMatrix<Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double
, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Xpetra::RowMatrix<Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >]'
/ptmp/Trilinos/packages/ifpack2/src/Ifpack2_AdditiveSchwarz_def.hpp:1517:5: required from 'void Ifpack2::AdditiveSchwarz<MatrixType, LocalInverseType>::setup() [with MatrixType = Tpetra::RowMatrix<Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::OpenMP
> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >; LocalInverseType = Ifpack2::Preconditioner<Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::OpenMP> >, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >]'
/ptmp/Trilinos/build/build-ifpack2-zoltan2-scotch/packages/stokhos/src/Ifpack2_AdditiveSchwarz_UQ_PCE_OpenMP.cpp:61:1: required from here
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:565:32: error: invalid cast from type 'Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::OpenMP> >' to type 'double'
double fw = double(fwgts[i]);
^
/ptmp/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_AlgScotch.hpp:595:50: error: invalid cast from type 'Sacado::UQ::PCE<Stokhos::DynamicStorage<int, double, Kokkos::OpenMP> >' to type 'double'
iwgts[i] = (SCOTCH_Num) ceil(double(fwgts[i])*scale);
^
packages/stokhos/src/CMakeFiles/stokhos_ifpack2.dir/build.make:854: recipe for target 'packages/stokhos/src/CMakeFiles/stokhos_ifpack2.dir/Ifpack2_AdditiveSchwarz_UQ_PCE_OpenMP.cpp.o' failed
make[2]: *** [packages/stokhos/src/CMakeFiles/stokhos_ifpack2.dir/Ifpack2_AdditiveSchwarz_UQ_PCE_OpenMP.cpp.o] Error 1
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/247The dependency target "MeshingGenie_libs" of target "Trilinos_libs" does not ...2016-03-30T16:34:29ZJames WillenbringThe dependency target "MeshingGenie_libs" of target "Trilinos_libs" does not exist.*Created by: nschloe*
When configuring with
```
-DTrilinos_ENABLE_MeshingGenie:BOOL=ON
```
I'm getting
```
-- Configuring done
CMake Error at cmake/tribits/core/package_arch/TribitsGlobalMacros.cmake:2255 (ADD_DEPENDENCIES):
The ...*Created by: nschloe*
When configuring with
```
-DTrilinos_ENABLE_MeshingGenie:BOOL=ON
```
I'm getting
```
-- Configuring done
CMake Error at cmake/tribits/core/package_arch/TribitsGlobalMacros.cmake:2255 (ADD_DEPENDENCIES):
The dependency target "MeshingGenie_libs" of target "Trilinos_libs" does
not exist.
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/252Zoltan2 serial build error2016-04-18T16:19:54ZJames WillenbringZoltan2 serial build error*Created by: jwillenbring*
@trilinos/zoltan2
I tried to build Zoltan2 using the following cmake arguments:
cmake \
-DTrilinos_ENABLE_TESTS:BOOL=ON \
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-DTrilinos_ENABLE_Zoltan2=ON \
../Trilinos...*Created by: jwillenbring*
@trilinos/zoltan2
I tried to build Zoltan2 using the following cmake arguments:
cmake \
-DTrilinos_ENABLE_TESTS:BOOL=ON \
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-DTrilinos_ENABLE_Zoltan2=ON \
../Trilinos
I got the following error:
[ 92%] Building CXX object packages/zoltan2/test/unit/CMakeFiles/Zoltan2_Mapping.dir/problems/Mapping.cpp.o
In file included from /scratch1/jmwille/TrilinosTest/Trilinos/packages/zoltan2/src/problems/Zoltan2_MappingProblem.hpp:60:0,
from /scratch1/jmwille/TrilinosTest/Trilinos/packages/zoltan2/test/unit/problems/Mapping.cpp:55:
/scratch1/jmwille/TrilinosTest/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_TaskMapping.hpp: In function ‘void Zoltan2::getCoarsenedPartGraph(const Zoltan2::Environment_, const Teuchos::Comm<int>_, const Zoltan2::GraphModel<typename Adapter::base_adapter_t>_, part_t, const part_t_, Teuchos::ArrayRCP<T>&, Teuchos::ArrayRCP<T>&, Teuchos::ArrayRCP<Packet>&)’:
/scratch1/jmwille/TrilinosTest/Trilinos/packages/zoltan2/src/algorithms/partition/Zoltan2_TaskMapping.hpp:298:24: error: ‘getRawMpiComm’ is not a member of ‘Teuchos’
make[2]: **\* [packages/zoltan2/test/unit/CMakeFiles/Zoltan2_Mapping.dir/problems/Mapping.cpp.o] Error 1
make[1]: **\* [packages/zoltan2/test/unit/CMakeFiles/Zoltan2_Mapping.dir/all] Error 2
make: **\* [all] Error 2
It looks like Zoltan2 is making a call to Teuchos that requires MPI to be enabled, but this is a serial build. I produced this error on a RHEL6 machine, using GCC 4.7.2, and specifically the SEMS NFS module:
module load gcc/4.7.2/base
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/254Intrepid2 Build errors with Intel2016-04-01T01:59:58ZJames WillenbringIntrepid2 Build errors with Intel*Created by: crtrott*
I get tons of errors like this with Intrepid2 using the Intel 15.0.2 compiler. This is a non-threaded build, using the sampleScripts/Sandia-SEMS/configure-all script with the addition of enabling Intrepid2.
Might ...*Created by: crtrott*
I get tons of errors like this with Intrepid2 using the Intel 15.0.2 compiler. This is a non-threaded build, using the sampleScripts/Sandia-SEMS/configure-all script with the addition of enabling Intrepid2.
Might be a non-standard conform usage of initializer lists which gcc allows, but intel doesn't.
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h: In instantiation of \u2018bool Intrepid::_GLOBAL__N__42_tmpxft_0000a64b_00000000_7_test_02_cpp1_ii_main::solveFN(FN&, const Intrepid::Vector<T, N>&) [with FN = Intrepid::_GLOBAL__N__42_tmpxft_0000a64b_00000000_7_test_02_cpp1_ii_main::SquareRootNLS<double>; T = double; unsigned int N = 2u]\u2019:
/home/crtrott/Trilinos/packages/intrepid/test/Shared/MiniTensor/test_02.cc:1017:51: required from here
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: too many braces around initializer for \u2018double\u2019 [-fpermissive]
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: invalid conversion from \u2018<brace-enclosed initializer list>\u2019 to \u2018double\u2019 [-fpermissive]
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: aggregate value used where a float was expected
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: too many braces around initializer for \u2018double\u2019 [-fpermissive]
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: invalid conversion from \u2018<brace-enclosed initializer list>\u2019 to \u2018double\u2019 [-fpermissive]
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: aggregate value used where a float was expected
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: too many braces around initializer for \u2018double\u2019 [-fpermissive]
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: invalid conversion from \u2018<brace-enclosed initializer list>\u2019 to \u2018double\u2019 [-fpermissive]
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: aggregate value used where a float was expected
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: too many braces around initializer for \u2018double\u2019 [-fpermissive]
/home/crtrott/Trilinos/packages/intrepid/src/Shared/MiniTensor/Intrepid_MiniTensor_Solvers.h:328:8: error: invalid conversion from \u2018<brace-enclosed initializer list>\u2019 to \u2018double\u2019 [-fpermissive]
@trilinos/intrepid2
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/255Prepare cmake arguments for parameterized Jenkins nightly builds2016-05-26T04:09:16ZJames WillenbringPrepare cmake arguments for parameterized Jenkins nightly builds*Created by: jwillenbring*
@bmpersc
As part of getting proper nightly testing back in place, and making jobs float on the Jenkins build farm, we are going to start by getting build arguments for working Trilinos configurations. These ...*Created by: jwillenbring*
@bmpersc
As part of getting proper nightly testing back in place, and making jobs float on the Jenkins build farm, we are going to start by getting build arguments for working Trilinos configurations. These configurations include GCC MPI/SERIAL, Intel MPI/SERIAL, and CLANG MPI/SERIAL. The MPI jobs will share one common set of arguments, as will the SERIAL jobs. The variation will come from loading the applicable set of modules from the SEMS NFS mount, and related environment variables.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/258KokkosAlgorithms_UnitTest_MPI_1 Fails on POWER8 with GCC 4.9 and CUDA 7.52016-04-27T15:23:12ZJames WillenbringKokkosAlgorithms_UnitTest_MPI_1 Fails on POWER8 with GCC 4.9 and CUDA 7.5*Created by: nmhamster*
@crtrott @hcedwar I am getting a test failure in Kokkos Algorithms XOR Random on POWER8 with GCC4.9 and CUDA 7.5 builds.
```
$ ctest -VV -R KokkosAlgorithms_UnitTest_MPI_1
UpdateCTestConfiguration from :/asclda...*Created by: nmhamster*
@crtrott @hcedwar I am getting a test failure in Kokkos Algorithms XOR Random on POWER8 with GCC4.9 and CUDA 7.5 builds.
```
$ ctest -VV -R KokkosAlgorithms_UnitTest_MPI_1
UpdateCTestConfiguration from :/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/DartConfiguration.tcl
Parse Config file:/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/DartConfiguration.tcl
Add coverage exclude regular expressions.
SetCTestConfiguration:CMakeCommand:/home/projects/pwr8-rhel72/cmake/3.4.3/bin/cmake
UpdateCTestConfiguration from :/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/DartConfiguration.tcl
Parse Config file:/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/DartConfiguration.tcl
Test project /ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 10
Start 10: KokkosAlgorithms_UnitTest_MPI_1
10: Test command: /ascldap/users/projects/pwr8-rhel72/openmpi/1.10.2/gcc/4.9.2/cuda/7.5.7/bin/mpiexec "-np" "1" "/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/packages/kokkos/algorithms/unit_tests/KokkosAlgorithms_UnitTest.exe"
10: Test timeout computed to be: 1500
10: [==========] Running 9 tests from 3 test cases.
10: [----------] Global test environment set-up.
10: [----------] 3 tests from cuda
10: Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 3.5 on device with compute capability 3.7 , this will likely reduce potential performance.
10: [ RUN ] cuda.Random_XorShift64
10: Test Scalar=int
10: -- Testing randomness properties
10: Pass: 1 1 -2.82471e-06 -3.40352e-05 -0.000104384 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.43252e-07 -0.0022995 -0.00836735 || 0.051031 27960 29418 || 28740.5 28674.4 || -115.67 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 -0.0169805 0.0015321 || 0.051031 1e+64 -1e+64
10: Test Scalar=unsigned int
10: -- Testing randomness properties
10: Pass: 1 1 -3.41644e-05 7.42232e-05 -3.04288e-05 || 0.000173985
10: -- Testing 1-D histogram
10: Density 1D: 6.43252e-07 0.0193418 0.0128208 || 0.051031 28059 29295 || 28130.3 28674.4 || 177.234 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 -0.0150665 0.00138121 || 0.051031 1e+64 -1e+64
10: Test Scalar=int64_t
10: -- Testing randomness properties
10: Mean: 2.867646e+04
10: Pass: 1 1 -3.6109e-05 -8.63585e-06 4.63777e-06 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.43252e-07 0.0115433 0.00712385 || 0.051031 28067 29309 || 28347.2 28674.4 || 98.4801 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 0.0200474 0.00931228 || 0.051031 1e+64 -1e+64
10: Test Scalar=uint64_t
10: -- Testing randomness properties
10: Pass: 1 1 2.42433e-05 -4.91036e-05 -1.69406e-05 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.43252e-07 0.0161715 -0.0122164 || 0.051031 27984 29362 || 28218.1 28674.4 || -168.88 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 0.0144323 -0.00292772 || 0.051031 1e+64 -1e+64
10: Test Scalar=float
10: -- Testing randomness properties
10: Pass: 1 1 5.75692e-05 9.05598e-11 -6.08529e-05 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.68478e-07 -0.00302028 0.0181608 || 0.051031 28019 29426 || 28761.3 28674.4 || 251.054 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 7.18929e-07 0.0221568 0.00457695 || 0.051031 1e+64 -1e+64
10: Test Scalar=double
10: -- Testing randomness properties
10: Pass: 1 1 1.06213e-05 5.18704e-05 3.61577e-07 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.43252e-07 0.0144485 -0.0405824 || 0.051031 28009 29477 || 28266 28674.4 || -561.011 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 0.0018761 -0.00990339 || 0.051031 1e+64 -1e+64
10: [ OK ] cuda.Random_XorShift64 (33438 ms)
10: [ RUN ] cuda.Random_XorShift1024
10: Test Scalar=int
10: -- Testing randomness properties
10: Pass: 1 1 -3.73188e-05 8.34822e-05 -0.000165954 || 0.000276214
10: -- Testing 1-D histogram
10: Mean: 1.137778e+04
10: /ascldap/users/sdhammo/git/trilinos-github-repo/packages/kokkos/algorithms/unit_tests/TestRandom.hpp:396: Failure
10: Value of: 1
10: Expected: test_int.pass_hist1d_var
10: Which is: 0
10: [ FAILED ] cuda.Random_XorShift1024 (2396 ms)
10: [ RUN ] cuda.SortUnsigned
10: Density 1D: 2.47955e-07 -0.0611763 -0.000749968 || 0.051031 10969 11798 || 12118.3 11377 || -10.3676 -0.823045
10: -- Testing 3-D histogram
10: Density 3D: 2.47955e-07 0.0076215 -0.000964689 || 0.051031 1e+64 -1e+64
10: --------------------------------------------------------------------------
10: mpiexec noticed that process rank 0 with PID 48917 on node white26 exited on signal 11 (Segmentation fault).
10: --------------------------------------------------------------------------
1/1 Test #10: KokkosAlgorithms_UnitTest_MPI_1 ...***Failed Error regular expression found in output. Regex=[ FAILED ] 46.83 sec
0% tests passed, 1 tests failed out of 1
Label Time Summary:
Kokkos = 46.83 sec (1 test)
Total Test time (real) = 47.29 sec
The following tests FAILED:
10 - KokkosAlgorithms_UnitTest_MPI_1 (Failed)
Errors while running CTest
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/259ninja and nvcc_wrapper2016-04-01T15:53:10ZJames Willenbringninja and nvcc_wrapper*Created by: bathmatt*
This was emailed about but making a formal bug report.
@bartlettroscoe @crtrott
If one configures trilinos to use ninja and build with cuda/nvcc_wrapper the compile fails with building dependencies.
Not sure wh...*Created by: bathmatt*
This was emailed about but making a formal bug report.
@bartlettroscoe @crtrott
If one configures trilinos to use ninja and build with cuda/nvcc_wrapper the compile fails with building dependencies.
Not sure where the issue is,
/projects/install/rhel6-x86_64/sems/compiler/gcc/4.9.2/openmpi/1.10.1/bin/mpicxx -I. -Ipackages/kokkos/core/src -I/home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src -I/usr/local/cuda/include -std=c++11 -g -O0 -MMD -MT packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o -MF packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o.d -o packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o -c /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/Kokkos_Core.cpp
g++: error: packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o.d: No such file or directory
[66/4434] Building C object packages/zoltan/src/CMakeFiles/zoltan.dir/coloring/coloring.c.o
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/262panzer example confiure out of date with package split2016-04-06T17:45:27ZJames Willenbringpanzer example confiure out of date with package split*Created by: bathmatt*
@rppawlo the cmake stuff for main_driver didn't get updated correctly when panzer got split.
*Created by: bathmatt*
@rppawlo the cmake stuff for main_driver didn't get updated correctly when panzer got split.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/263Phalanx: update runtime MDField to use Kokkos::DynRankView2016-05-18T01:46:59ZJames WillenbringPhalanx: update runtime MDField to use Kokkos::DynRankView*Created by: rppawlo*
*Created by: rppawlo*
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/267Tpetra::Map: Add Kokkos::View version of getNodeElementList2016-04-04T21:38:23ZJames WillenbringTpetra::Map: Add Kokkos::View version of getNodeElementList*Created by: mhoemmen*
@trilinos/tpetra
Add Kokkos::View version of getNodeElementList (that returns a list of the global indices on the calling process).
*Created by: mhoemmen*
@trilinos/tpetra
Add Kokkos::View version of getNodeElementList (that returns a list of the global indices on the calling process).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/264'No such file: klu2_simple.h' during installation testing2016-04-05T15:07:20ZJames Willenbring'No such file: klu2_simple.h' during installation testing*Created by: bavier*
Doing installation testing via `Trilinos_INSTALLATION_DIR` for several packages leads to errors during `make` about a missing `klu2_simple.h`. For example:
```
cmake \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=NO \
-D...*Created by: bavier*
Doing installation testing via `Trilinos_INSTALLATION_DIR` for several packages leads to errors during `make` about a missing `klu2_simple.h`. For example:
```
cmake \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=NO \
-DTrilinos_INSTALLATION_DIR=/ptmp/trilinos-install \
-DTrilinos_ENABLE_Ifpack2:BOOL=YES \
-DTrilinos_ENABLE_TESTS:BOOL=YES \
-DTrilinos_ENABLE_OpenMP:BOOL=YES \
/ptmp/Trilinos
```
This seems to be because there is an executable `klu2_simple` declared in Amesos2 that is not installed, and cmake attempts to build it during installation testing, but the needed klu2_simple.h header is not installed.
Declaring `klu2_simple` as an example, which seems appropriate, fixes the issue. See the attached patch.
[0001-Amesos2-Build-klu2_simple-as-an-example.txt](https://github.com/trilinos/Trilinos/files/202983/0001-Amesos2-Build-klu2_simple-as-an-example.txt)
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/266Tpetra::Map: Add noncontiguous constructor that takes global indices as Kokko...2016-04-04T21:38:23ZJames WillenbringTpetra::Map: Add noncontiguous constructor that takes global indices as Kokkos::View*Created by: mhoemmen*
@trilinos/tpetra
Add a noncontiguous constructor to Tpetra::Map that takes global indices as (device) Kokkos::View, rather than Teuchos::ArrayView. Ideally, assume UVM as little as possible (due to the difficul...*Created by: mhoemmen*
@trilinos/tpetra
Add a noncontiguous constructor to Tpetra::Map that takes global indices as (device) Kokkos::View, rather than Teuchos::ArrayView. Ideally, assume UVM as little as possible (due to the difficulty of debugging issues like #227).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/270TpetraCore tests fails to build during installation testing2017-10-26T19:46:40ZJames WillenbringTpetraCore tests fails to build during installation testing*Created by: bavier*
Trilinos 12.6.1 configured with
```
cmake \
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=NO \
-DTrilinos_ENABLE_Tpetra:BOOL=YES \
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=NO \
-DTrilinos_ENABLE_TESTS:BO...*Created by: bavier*
Trilinos 12.6.1 configured with
```
cmake \
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=NO \
-DTrilinos_ENABLE_Tpetra:BOOL=YES \
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=NO \
-DTrilinos_ENABLE_TESTS:BOOL=NO \
-DTPL_BLAS_LIBRARIES=openblas \
-DCMAKE_INSTALL_PREFIX:PATH=/ptmp/trilinos-tpetra \
..
```
And then configuring for installation-testing with
```
cmake \
-DTrilinos_INSTALLATION_DIR=/ptmp/trilinos-tpetra \
-DTrilinos_ENABLE_Tpetra:BOOL=YES \
-DTrilinos_ENABLE_TESTS:BOOL=YES \
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=NO \
..
```
Leads to the following error from `make`:
```
In file included from /ptmp/trilinos-tpetra/lib/cmake/TpetraCore/../../../include/Tpetra_CrsMatrixSolveOp_decl.hpp:50:0,
from /ptmp/trilinos-tpetra/lib/cmake/TpetraCore/../../../include/Tpetra_CrsMatrixSolveOp.hpp:1,
from /ptmp/Trilinos/packages/tpetra/core/test/BugTests/CompilationTests.cpp:50:
/ptmp/trilinos-tpetra/lib/cmake/TpetraCore/../../../include/Tpetra_CrsMatrixSolveOp.hpp:2:43: fatal error: Tpetra_CrsMatrixSolveOp_def.hpp: No such file or directory
#include "Tpetra_CrsMatrixSolveOp_def.hpp"
^
compilation terminated.
packages/tpetra/core/test/BugTests/CMakeFiles/TpetraCore_CompilationTests.dir/build.make:62: recipe for target 'packages/tpetra/core/test/BugTests/CMakeFiles/TpetraCore_CompilationTests.dir/CompilationTests.cpp.o' failed
make[2]: *** [packages/tpetra/core/test/BugTests/CMakeFiles/TpetraCore_CompilationTests.dir/CompilationTests.cpp.o] Error 1
CMakeFiles/Makefile2:3011: recipe for target 'packages/tpetra/core/test/BugTests/CMakeFiles/TpetraCore_CompilationTests.dir/all' failed
make[1]: *** [packages/tpetra/core/test/BugTests/CMakeFiles/TpetraCore_CompilationTests.dir/all] Error 2
```
With ETI enabled, the error is a bit different
```
[ 38%] Building CXX object packages/tpetra/core/test/CrsMatrix/CMakeFiles/TpetraCore_CrsMatrix_TriSolve.dir/CrsMatrix_TriSolve.cpp.o
/ptmp/Trilinos/packages/tpetra/core/test/CrsMatrix/CrsMatrix_TriSolve.cpp:107:17: error: 'Tpetra::createCrsMatrixSolveOp' has not been declared
using Tpetra::createCrsMatrixSolveOp;
^
/ptmp/Trilinos/packages/tpetra/core/test/CrsMatrix/CrsMatrix_TriSolve.cpp: In member function 'void {anonymous}::CrsMatrix_EmptyTriSolve_UnitTest<LO, GO, Scalar, Node>::runUnitTestImpl(Teuchos::FancyOStream&, bool&) const':
/ptmp/Trilinos/packages/tpetra/core/test/CrsMatrix/CrsMatrix_TriSolve.cpp:176:19: error: 'createCrsMatrixSolveOp' was not declared in this scope
ZeroIOp = createCrsMatrixSolveOp<Scalar>(ZeroMat.getConst());
^
/ptmp/Trilinos/packages/tpetra/core/test/CrsMatrix/CrsMatrix_TriSolve.cpp:176:48: error: expected primary-expression before '>' token
ZeroIOp = createCrsMatrixSolveOp<Scalar>(ZeroMat.getConst());
^
/ptmp/Trilinos/packages/tpetra/core/test/CrsMatrix/CrsMatrix_TriSolve.cpp: In member function 'void {anonymous}::CrsMatrix_TriSolve_UnitTest<LO, GO, Scalar, Node>::runUnitTestImpl(Teuchos::FancyOStream&, bool&) const':
/ptmp/Trilinos/packages/tpetra/core/test/CrsMatrix/CrsMatrix_TriSolve.cpp:396:18: error: 'createCrsMatrixSolveOp' was not declared in this scope
AIOp = createCrsMatrixSolveOp<Scalar> (AMat.getConst ());
^
/ptmp/Trilinos/packages/tpetra/core/test/CrsMatrix/CrsMatrix_TriSolve.cpp:396:47: error: expected primary-expression before '>' token
AIOp = createCrsMatrixSolveOp<Scalar> (AMat.getConst ());
^
packages/tpetra/core/test/CrsMatrix/CMakeFiles/TpetraCore_CrsMatrix_TriSolve.dir/build.make:62: recipe for target 'packages/tpetra/core/test/CrsMatrix/CMakeFiles/TpetraCore_CrsMatrix_TriSolve.dir/CrsMatrix_TriSolve.cpp.o' failed
make[2]: *** [packages/tpetra/core/test/CrsMatrix/CMakeFiles/TpetraCore_CrsMatrix_TriSolve.dir/CrsMatrix_TriSolve.cpp.o] Error 1
CMakeFiles/Makefile2:3562: recipe for target 'packages/tpetra/core/test/CrsMatrix/CMakeFiles/TpetraCore_CrsMatrix_TriSolve.dir/all' failed
make[1]: *** [packages/tpetra/core/test/CrsMatrix/CMakeFiles/TpetraCore_CrsMatrix_TriSolve.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2
```
These errors appear to be a result of how the TRILINOS_CREATE_CLIENT_TEMPLATE_HEADERS cmake macro interacts with the tpetra/core/src/Tpetra_CrsMatrixSolveOp.hpp header. The macro generates a header of the same name in CMAKE_CURRENT_BINARY_DIR. This generated header is then installed and picked up during installation testing. The tests build without error during pre-installation testing because the original header is found in CMAKE_CURRENT_SOURCE_DIR before it is found in CMAKE_CURRENT_BINARY_DIR.
Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/269Get Trilinos building with CMake + Ninja + Fortran2016-04-06T20:29:28ZJames WillenbringGet Trilinos building with CMake + Ninja + Fortran*Created by: bartlettroscoe*
This story is to get all of Trilinos building with CMake + Ninja with Fortran turned on. There is a branch of Ninja and CMake that supports projects with Fortran code. This story will track efforts to get ...*Created by: bartlettroscoe*
This story is to get all of Trilinos building with CMake + Ninja with Fortran turned on. There is a branch of Ninja and CMake that supports projects with Fortran code. This story will track efforts to get Trilinos and CMake + Ninja + Fortran working.
CC: @nmhamster. @bathmatt
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/271Panzer: CurlLaplacianExample-ConvTest randomly failing2017-05-03T19:23:26ZJames WillenbringPanzer: CurlLaplacianExample-ConvTest randomly failing*Created by: rppawlo*
On my workstation, this test occasionally fails. Probably a race condition on the output check.*Created by: rppawlo*
On my workstation, this test occasionally fails. Probably a race condition on the output check.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/272Panzer unsafe macro definitions2016-04-23T20:47:19ZJames WillenbringPanzer unsafe macro definitions*Created by: rppawlo*
The STK adapters use some unsafe macros such as HAVE_IOSS, HAVE_TEKO, HAVE_MUELU for optional package support. These need to be protected with package specific macros: PANZER_HAVE_TEKO. This is a hold over form nam...*Created by: rppawlo*
The STK adapters use some unsafe macros such as HAVE_IOSS, HAVE_TEKO, HAVE_MUELU for optional package support. These need to be protected with package specific macros: PANZER_HAVE_TEKO. This is a hold over form naming convetions from the autotools days...
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/273Teko header Teko_DiagonalPreconditionerFactory.hpp missing include2016-04-06T21:22:57ZJames WillenbringTeko header Teko_DiagonalPreconditionerFactory.hpp missing include*Created by: bavier*
Using `#include <Teko_DiagonalPreconditionerFactory.hpp>` with an installed Trilinos, e.g. when building Teko tests via installation testing, leads to the following error:
```
[ 9%] Building CXX object packages/te...*Created by: bavier*
Using `#include <Teko_DiagonalPreconditionerFactory.hpp>` with an installed Trilinos, e.g. when building Teko tests via installation testing, leads to the following error:
```
[ 9%] Building CXX object packages/teko/tests/CMakeFiles/Teko_DiagonalPreconditionerFactory.dir/unit_tests/tDiagonalPreconditionerFactory.cpp.o
In file included from /ptmp/Trilinos/packages/teko/tests/unit_tests/tDiagonalPreconditionerFactory.cpp:67:0:
/ptmp/trilinos-install/lib/cmake/Teko/../../../include/trilinos/Teko_DiagonalPreconditionerFactory.hpp:53:41: fatal error: Tpetra/Teko_TpetraHelpers.hpp: No such file or directory
#include "Tpetra/Teko_TpetraHelpers.hpp"
^
compilation terminated.
```
I would suggest the following patch, since the declarations in Teko_TpetraHelpers.hpp are not needed in Teko_DiagonalPreconditionerFactory.hpp:
```
--- a/packages/teko/src/Teko_DiagonalPreconditionerFactory.hpp
+++ b/packages/teko/src/Teko_DiagonalPreconditionerFactory.hpp
@@ -50,7 +50,6 @@
// Teko includes
#include "Teko_PreconditionerState.hpp"
#include "Teko_PreconditionerFactory.hpp"
-#include "Tpetra/Teko_TpetraHelpers.hpp"
class EpetraExt_PointToBlockDiagPermute;
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/274Panzer: Transition to use DynRankView refactor version of Intrepid22016-11-21T21:26:33ZJames WillenbringPanzer: Transition to use DynRankView refactor version of Intrepid2*Created by: rppawlo*
In the refactor of Intrepid2, we will stop passing MDFields into Intrepid and instead use Kokkos::Views and Kokkos::DynRankViews.
This is close to being finished.
Issues to complete:
#808 [done]
#809 [done]
#810 ...*Created by: rppawlo*
In the refactor of Intrepid2, we will stop passing MDFields into Intrepid and instead use Kokkos::Views and Kokkos::DynRankViews.
This is close to being finished.
Issues to complete:
#808 [done]
#809 [done]
#810 [done]
Fix the ResponseScatterEvalautor_Probe (Intrepid2::checkPointwiseInclusion was changed). [deferred - will be added as separate ticket]
Status:
1. Panzer is completed
2. Application codes completedhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/249BUILD_SHARED option in SERIAL_RELEASE.config2016-12-01T02:42:25ZJames WillenbringBUILD_SHARED option in SERIAL_RELEASE.config*Created by: d-meiser*
The configuration templates produced by the `checkin-test.py` script (e.g. `SERIAL_RELEASE.config`) contain the option `-DBUILD_SHARED:BOOL=ON` option. I can't find that option in the cmake directory:
```
[dmeise...*Created by: d-meiser*
The configuration templates produced by the `checkin-test.py` script (e.g. `SERIAL_RELEASE.config`) contain the option `-DBUILD_SHARED:BOOL=ON` option. I can't find that option in the cmake directory:
```
[dmeiser@ivyamd cmake]$ pwd
/scr_ivyamd/dmeiser/Trilinos/cmake
[dmeiser@ivyamd cmake]$ grep -R BUILD_SHARED * | grep -v BUILD_SHARED_LIBS
TODO:- Enable BUILD_SHARED=ON if Trilinos_ENABLE_PyTrilinos=ON if BUILD_SHARED is
TODO:- Report an error if Trilinos_ENABLE_PyTrilinos=ON but BUILD_SHARED=OFF and
Binary file tribits/ci_support/CheckinTest.pyc matches
tribits/ci_support/CheckinTest.py: "#-DBUILD_SHARED:BOOL=ON\n" \
tribits/ci_support/CheckinTest.py: "#-DBUILD_SHARED:BOOL=ON\n"
```
Is this supposed to be `-DBUILD_SHARED_LIBS:BOOL=ON`?
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/278Phalanx: Add extent and extent_int methods to MDField2016-05-18T02:45:04ZJames WillenbringPhalanx: Add extent and extent_int methods to MDField*Created by: rppawlo*
The dimension() method on kokkos views is beign renamed to extent() to align with the c++ standard anming conventions. Need to add the new accessors to remain consistent on MDFields.
*Created by: rppawlo*
The dimension() method on kokkos views is beign renamed to extent() to align with the c++ standard anming conventions. Need to add the new accessors to remain consistent on MDFields.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/284Panzer: build and test parallel netcdf 4.4.0 for Drekar stack2016-04-22T02:28:59ZJames WillenbringPanzer: build and test parallel netcdf 4.4.0 for Drekar stack*Created by: rppawlo*
@trilinos/panzer Need to test out new version of netcdf. Notes from @gsjaardema and @bartlettroscoe are below:
Sorry for all the responses, but just for completeness here are the SEACAS responses to the question:...*Created by: rppawlo*
@trilinos/panzer Need to test out new version of netcdf. Notes from @gsjaardema and @bartlettroscoe are below:
Sorry for all the responses, but just for completeness here are the SEACAS responses to the question:
@bartlettroscoe wrote:
3) What TPLs should be enabled? (The SEMS Dev Env assumes blas and lapack are already on the system and provides boost, hdf5, netcdf, parmetis, scotch, qd, superlu, and zlib)
4) Of the TPLs that are enabled, what versions of TPLs should considered standard?
SEACAS uses:
NetCDF -- as new as possible. Recommend 4.4.0 however, 4.3.3.1 or later is possible. Built with --enable-netcdf4, parallel enabled, and recommend --enable-pnetcdf.
HDF5 -- needed if NetCDF has --enable-netcdf4. Recommend 1.8.15 or 1.8.16; The most recent 1.10.X series is not yet supported.
Parallel-NetCDF (PNetCDF) if NetCDF is built with --enable-pnetcdf. Note that this is a different library than NetCDF. Recommend 1.7.0, but 1.6.1 is ok.
MatIO for reading/writing MatLab files. From https://github.com/tbeu/matio.git which is the current active development fork. Version 1.5.3 or later built with support for hdf5-based files.
CGNS -- version 3.3.0 preferred. Built with ENABLE_SCOPIING and ENABLE_HDF5.
ParMetis -- used by IOSS library in a parallel build. Recommend index type set to 64-bit.
At an absolute minimum with no parallel-IO support, can use just the NetCDF library.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/282NOX add solution scaling support2016-06-09T15:24:38ZJames WillenbringNOX add solution scaling support*Created by: rppawlo*
Have always done left scaling. Now have a request for right scaling.
@trilinos/nox
*Created by: rppawlo*
Have always done left scaling. Now have a request for right scaling.
@trilinos/nox
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/286Zoltan2 Problem and EvaluatePartition segfaults with NULL parameter list2016-09-02T00:08:44ZJames WillenbringZoltan2 Problem and EvaluatePartition segfaults with NULL parameter list*Created by: kddevin*
Parameter list is passed as a pointer to Problem constructors and EvaluatePartition. It is not tested for NULL before it is used. It should be tested for NULL before being used.
Needed by May 1 if possible for ne...*Created by: kddevin*
Parameter list is passed as a pointer to Problem constructors and EvaluatePartition. It is not tested for NULL before it is used. It should be tested for NULL before being used.
Needed by May 1 if possible for next Trilinos merge.
@trilinos/zoltan2