Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2019-04-25T02:54:52Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4927Compiling issue of MueLu with Trilinos 12.14.12019-04-25T02:54:52ZJames WillenbringCompiling issue of MueLu with Trilinos 12.14.1*Created by: YingzhouLi*
I am compiling Trilinos 12.14.1 on Ubuntu 18.04 with g++ 7.3 and matlab R2018a.
cmake \
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-D Trilinos_ENABLE_Amesos:BOOL=ON \
-D Trilinos_E...*Created by: YingzhouLi*
I am compiling Trilinos 12.14.1 on Ubuntu 18.04 with g++ 7.3 and matlab R2018a.
cmake \
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-D Trilinos_ENABLE_Amesos:BOOL=ON \
-D Trilinos_ENABLE_Amesos2:BOOL=ON \
-D Amesos2_ENABLE_KLU2:BOOL=ON \
-D Trilinos_ENABLE_AztecOO:BOOL=ON \
-D Trilinos_ENABLE_Epetra:BOOL=ON \
-D Trilinos_ENABLE_EpetraExt:BOOL=ON \
-D Trilinos_ENABLE_Fortran:BOOL=OFF \
-D Trilinos_ENABLE_Ifpack:BOOL=ON \
-D Trilinos_ENABLE_Ifpack2:BOOL=ON \
-D Trilinos_ENABLE_MueLu:BOOL=ON \
-D MueLu_ENABLE_TESTS:STRING=ON \
-D MueLu_ENABLE_EXAMPLES:STRING=OFF \
-D Trilinos_ENABLE_Teuchos:BOOL=ON \
-D Trilinos_ENABLE_Tpetra:BOOL=ON \
-D Trilinos_ENABLE_Intrepid2:BOOL=ON \
-D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \
-D Teuchos_ENABLE_COMPLEX=ON \
-D TPL_ENABLE_MPI:BOOL=OFF \
-D TPL_ENABLE_MATLAB:BOOL=ON \
-D MATLAB_ROOT:STRING="/usr/local/MATLAB/R2018a/" \
-D MATLAB_ARCH:STRING="glnxa64" \
-D BUILD_SHARED_LIBS:BOOL=ON \
-D CMAKE_CXX_FLAGS="-fPIC -g -DMEX_DOUBLE_HANDLE" \
-D Trilinos_EXTRA_LINK_FLAGS="-lrt -lm -lgfortran" \
..
CMakeFiles/MueLu_UnitTests.dir/ParameterList/FactoryFactory.cpp.o: In function `MueLu::MatlabSmoother<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Setup(MueLu::Level&)':
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:97: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::MuemexData(Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >&)'
CMakeFiles/MueLu_UnitTests.dir/ParameterList/FactoryFactory.cpp.o: In function `MueLu::MatlabSmoother<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Setup(MueLu::Level&)':
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:97: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::Matrix<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::MuemexData(Teuchos::RCP<Xpetra::Matrix<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >&)'
CMakeFiles/MueLu_UnitTests.dir/ParameterList/FactoryFactory.cpp.o: In function `MueLu::MatlabSmoother<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Apply(Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >&, Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, bool) const':
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:116: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::MuemexData(Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >&)'
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:121: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::MuemexData(Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >&)'
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:122: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::MuemexData(Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >&)'
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:130: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::getData()'
CMakeFiles/MueLu_UnitTests.dir/ParameterList/FactoryFactory.cpp.o: In function `MueLu::MatlabSmoother<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Apply(Xpetra::MultiVector<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >&, Xpetra::MultiVector<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, bool) const':
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:116: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::Matrix<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::MuemexData(Teuchos::RCP<Xpetra::Matrix<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >&)'
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:121: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::MultiVector<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::MuemexData(Teuchos::RCP<Xpetra::MultiVector<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >&)'
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:122: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::MultiVector<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::MuemexData(Teuchos::RCP<Xpetra::MultiVector<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >&)'
/home/yingzhou/Documents/Trilinos-trilinos-release-12-14-1/packages/muelu/test/unit_tests/../../src/../matlab/src/MueLu_MatlabSmoother_def.hpp:130: undefined reference to `MueLu::MuemexData<Teuchos::RCP<Xpetra::MultiVector<std::complex<double>, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >::getData()'
collect2: error: ld returned 1 exit status
packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/build.make:1421: recipe for target 'packages/muelu/test/unit_tests/MueLu_UnitTests.exe' failed
make[2]: *** [packages/muelu/test/unit_tests/MueLu_UnitTests.exe] Error 1
CMakeFiles/Makefile2:11396: recipe for target 'packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/all' failed
make[1]: *** [packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/all] Error 2
Makefile:162: recipe for target 'all' failed
make: *** [all] Error 2
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4915Panzer: Link problem when Tpetra_INST_INT_INT=OFF2019-04-20T20:00:47ZJames WillenbringPanzer: Link problem when Tpetra_INST_INT_INT=OFF*Created by: kddevin*
<!---
Provide a general summary of the issue in the Title above. If this issue
pertains to a particular package in Trilinos, it's worthwhile to start the
title with "PackageName: ".
-->
<!---
Note that an...*Created by: kddevin*
<!---
Provide a general summary of the issue in the Title above. If this issue
pertains to a particular package in Trilinos, it's worthwhile to start the
title with "PackageName: ".
-->
<!---
Note that anything between these delimiters is a comment that will not appear
in the issue description once created. Click on the Preview tab to see what
everything will look like when you submit.
-->
<!---
Feel free to delete anything from this template that is not applicable to the
issue you are submitting.
-->
<!---
Replace <teamName> below with the appropriate Trilinos package/team name.
-->
@trilinos/panzer
<!---
Assignees: If you know anyone who should likely tackle this issue, select them
from the Assignees drop-down on the right.
-->
<!---
Lables: Choose any applicable package names from the Labels drop-down on the
right. Additionally, choose a label to indicate the type of issue, for
instance, bug, build, documentation, enhancement, etc.
-->
## Expectations
Panzer should build with Tpetra_INST_INT_INT=OFF
## Current Behavior
I get link errors due to use of Tpetra::MultiVector with global ordinals = int.
```
"virtual thunk to Tpetra::MultiVector<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::describe(Teuchos::basic_FancyOStream<char, std::__1::char_traits<char> >&, Teuchos::EVerbosityLevel) const", referenced from:
vtable for Tpetra::MultiVector<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > in libpanzer-dof-mgr.a(Panzer_DOFManager.cpp.o)
construction vtable for Tpetra::MultiVector<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >-in-Tpetra::Vector<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > in libpanzer-dof-mgr.a(Panzer_Filtered_UniqueGlobalIndexer.cpp.o)
vtable for Tpetra::MultiVector<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > in libpanzer-dof-mgr.a(Panzer_Filtered_UniqueGlobalIndexer.cpp.o)
"virtual thunk to Tpetra::Vector<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::describe(Teuchos::basic_FancyOStream<char, std::__1::char_traits<char> >&, Teuchos::EVerbosityLevel) const", referenced from:
vtable for Tpetra::Vector<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > in libpanzer-dof-mgr.a(Panzer_Filtered_UniqueGlobalIndexer.cpp.o)
```
## Motivation and Context
We'd like to build Trilinos with only one global ordinal type enabled.
## Definition of Done
<!---
Tell us what needs to happen. If necessary, give us a task list along the
lines of:
- [ ] First do this.
- [ ] Then do that.
- [ ] Also this other thing.
-->
Panzer builds and runs with Tpetra_INST_INT_INT=OFF.
## Possible Solution
<!---
Not obligatory, but suggest a fix for the bug or documentation, or suggest
ideas on how to implement the addition or change.
-->
I see Tpetra::MultiVector<int,int,int> in tUniqueGlobalIndexerUtilities.cpp; perhaps this is the source of the problem. One could disable this test when Tpetra_INST_INT_INT=OFF.
Or perhaps the problem is due to some Epetra vs Tpetra issue?
## Steps to Reproduce
<!---
Provide a link to a live example, or an unambiguous set of steps to reproduce
this issue. Include code to reproduce, if relevant.
1. Do this.
1. Do that.
1. Shake fist angrily at computer.
-->
```
cmake \
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-D Tpetra_INST_INT_LONG_LONG:BOOL=ON \
-D Tpetra_INST_INT_INT:BOOL=OFF \
-D CMAKE_BUILD_TYPE:STRING="DEBUG" \
-D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
\
-D TPL_ENABLE_MPI:BOOL=ON \
-D MPI_EXEC_MAX_NUMPROCS:STRING=11 \
\
-D TPL_ENABLE_BinUtils:BOOL=OFF \
-D TPL_ENABLE_Pthread:BOOL=OFF \
\
-D CMAKE_C_FLAGS:STRING="-Wall -pedantic -Wno-unknown-pragmas -Wno-narrowing -Wno-inline -Wshadow -Wdeprecated-declarations -Wempty-body -Wignored-qualifiers -Wmissing-field-initializers -Wsign-compare -Wtype-limits -Wuninitialized -Winit-self -fstrict-aliasing -Wno-long-long" \
-D CMAKE_CXX_FLAGS:STRING="-Wall -pedantic -Wno-unknown-pragmas -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-inline -Wshadow -Wdeprecated-declarations -Wempty-body -Wignored-qualifiers -Wmissing-field-initializers -Wsign-compare -Wtype-limits -Wuninitialized -Winit-self -fstrict-aliasing" \
\
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON \
-D Trilinos_ENABLE_TESTS:BOOL=ON \
-D Trilinos_ENABLE_EXAMPLES:BOOL=ON \
\
-D Trilinos_ENABLE_SHADOW_WARNINGS:BOOL=ON \
-D Trilinos_VERBOSE_CONFIGURE:BOOL=OFF \
-D Trilinos_ENABLE_Fortran:BOOL=OFF \
\
-D Tpetra_ENABLE_DEPRECATED_CODE:BOOL=OFF \
\
-D Trilinos_ENABLE_Stokhos:BOOL=ON \
-D Trilinos_ENABLE_Nox:BOOL=ON \
-D Trilinos_ENABLE_ROL:BOOL=ON \
-D Trilinos_ENABLE_MiniTensor:BOOL=OFF \
-D ROL_ENABLE_MiniTensor:BOOL=OFF \
-D Trilinos_ENABLE_Panzer:BOOL=ON \
-D Trilinos_ENABLE_PanzerAdaptersSTK:BOOL=OFF \
-D Trilinos_ENABLE_PanzerAdaptersIOSS:BOOL=OFF \
-D Trilinos_ENABLE_Thyra:BOOL=ON \
-D Trilinos_ENABLE_MueLu:BOOL=ON \
-D Trilinos_ENABLE_Anasazi:BOOL=ON \
-D Trilinos_ENABLE_Belos:BOOL=ON \
\
-D Teuchos_ENABLE_STACKTRACE:BOOL=OFF \
-D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON \
.. |& tee OUTPUT.CMAKE
make -j 8 |& tee OUTPUT.MAKE
```
## Your Environment
<!---
Include relevant details about your environment such that we can replicate this
issue.
-->
- **Relevant repo SHA1s:**
- **Relevant configure flags or configure script:**
- **Operating system and version:**
- **Compiler and TPL versions:**
## Related Issues
<!---
If applicable, let us know how this bug is related to any other open issues:
-->
* Blocks
* Is blocked by
* Follows
* Precedes
* Related to
* Part of
* Composed of
## Additional Information
<!---
Anything else that might be helpful for us to know in addressing this issue:
* Configure log file:
* Build log file:
* Test log file:
* When was the last time everything worked (date/time; SHA1s; etc.)?
* What did you do that made the bug rear its ugly head?
* Have you tried turning it off and on again?
-->
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4903Xpetra appears to require GlobalOrdinal=int even when Tpetra_INST_INT_INT=OFF2019-04-18T12:01:12ZJames WillenbringXpetra appears to require GlobalOrdinal=int even when Tpetra_INST_INT_INT=OFF*Created by: mhoemmen*
@trilinos/xpetra @trilinos/tpetra
Xpetra appears to attempt to instantiate `Tpetra::Map<int, int, ...>`, or at least to use its declarations, even when `Tpetra_INST_INT_INT=OFF`. For example, when I put a `st...*Created by: mhoemmen*
@trilinos/xpetra @trilinos/tpetra
Xpetra appears to attempt to instantiate `Tpetra::Map<int, int, ...>`, or at least to use its declarations, even when `Tpetra_INST_INT_INT=OFF`. For example, when I put a `static_assert` in `Tpetra::Map` that requires `GlobalOrdinal == long long`, I am able to build all of Tpetra, but get the following Xpetra build errors:
```
In file included from .../packages/tpetra/core/src/Tpetra_Map.hpp:1:0,
from .../Trilinos/packages/xpetra/src/Map/Xpetra_Map.hpp:61,
from .../Trilinos/packages/xpetra/src/Export/Xpetra_Export.hpp:54,
from .../Trilinos/packages/xpetra/src/Export/Xpetra_EpetraExport.hpp:51,
from .../Trilinos/packages/xpetra/src/Export/Xpetra_EpetraExport.cpp:46:
.../Trilinos/packages/tpetra/core/src/Tpetra_Map_decl.hpp: In instantiation of ‘class Tpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >’:
.../Trilinos/packages/xpetra/src/Map/Xpetra_Map.hpp:224:85: required from ‘class Xpetra::Map<>’
.../Trilinos/packages/xpetra/src/Export/Xpetra_Export.hpp:59:39: required from here
.../Trilinos/packages/tpetra/core/src/Tpetra_Map_decl.hpp:257:5: error: static assertion failed: UH OH
static_assert (std::is_same<global_ordinal_type, long long>::value, "UH OH");
^
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4902Stokhos doesn't build with GO=int disabled (GO=long long enabled) and Tpetra_...2019-04-16T14:21:51ZJames WillenbringStokhos doesn't build with GO=int disabled (GO=long long enabled) and Tpetra_ENABLE_DEPRECATED_CODE=OFF*Created by: mhoemmen*
@trilinos/stokhos @trilinos/tpetra @kddevin @william76 @etphipp
I fixed all of Stokhos' unit tests and examples not to use `GO=int`, but am still getting link errors. I will diagnose by inserting a `static_as...*Created by: mhoemmen*
@trilinos/stokhos @trilinos/tpetra @kddevin @william76 @etphipp
I fixed all of Stokhos' unit tests and examples not to use `GO=int`, but am still getting link errors. I will diagnose by inserting a `static_assert (std::is_same<global_ordinal_type, long long>::value, "UH OH");` in `Tpetra::Map` and seeing where it triggers.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4901Tempus: Add Stepper Wrapper2019-04-15T22:37:31ZJames WillenbringTempus: Add Stepper Wrapper*Created by: ccober6*
@trilinos/tempus
## Expectations
Would like to have a Stepper that just wraps some code into a Stepper. The StepperWrapper would have default implementations of everything, except takeStep() which would con...*Created by: ccober6*
@trilinos/tempus
## Expectations
Would like to have a Stepper that just wraps some code into a Stepper. The StepperWrapper would have default implementations of everything, except takeStep() which would contain the wrapped code. This would allow quick and easy creation of a Stepper that does not integrate the solution, x, but does other related work.
## Motivation and Context
EMPIRE-PIC does a particle push, which is also needed by EMPIRE-Hybrid. To eliminate code duplication and allow code flexibility, we would like this piece of code to be a StepperWrapper.
## Definition of Done
- [ ] Implement StepperWrapper.
- [ ] Add regression tests.
- [ ] Pass all tests, including new ones.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4900Tpetra/Xpetra/MueLu: Purge the Clones2019-05-01T22:49:07ZJames WillenbringTpetra/Xpetra/MueLu: Purge the Clones*Created by: csiefer2*
See discussion in #4893 *Created by: csiefer2*
See discussion in #4893 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4883Tempus: Add Ability to Integrate Auxiliary Variables2019-04-14T19:18:07ZJames WillenbringTempus: Add Ability to Integrate Auxiliary Variables*Created by: ccober6*
@trilinos/tempus
## Expectations
Have the ability to time integrate auxiliary variables.
There is a difference between the PhysicsState, _p_, and an auxiliary variable, _y_. The PhysicState is physics da...*Created by: ccober6*
@trilinos/tempus
## Expectations
Have the ability to time integrate auxiliary variables.
There is a difference between the PhysicsState, _p_, and an auxiliary variable, _y_. The PhysicState is physics data that does not need time integration. It is just data at each time step that is needed to evaluate _f(x,xDot,t,p)_. An auxiliary variable, _y_, however does need time integration, but has been separated from the solution variable, _x_, because it is a simple update or has a different time-integration.
To start with we will assume that the solution, _x_, and the auxiliary variables, _y_, are integrated with the same Stepper, but is a simple update from the SolutionState (e.g., _x__{n-1}, _x__{n}, _xDot__{n-1}, _xDot__{n}, _y__{n-1}, _yDot__{n-1}, and RK stage values) and other values available from the TimeDerivative class.
## Motivation and Context
The ability to segregate the solution, _x_, and auxiliary variables, _y_, can be a substantial computational savings because of the reduction in size of the solution vector from {_x_, _y_} to just {_x_} and simple update of {_y_}.
## Definition of Done
- [x] Add auxiliary variables, _y_, to SolutionState.
- [ ] Incorporate _y_ to TimeDerivative class.
- [x] Develop means for application to provide update function to Stepper. (This may be a prototype for a new ModelEvaluator).
- [ ] Have TempusWrapperModelEvaluator use _y_ and compute _yDot_.
- [ ] Either implement the new auxiliary variables in every Stepper, or have the Stepper error out if auxiliary variables are used.
- [ ] Add test to cover this new capability.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4867Tempus: Add Stepper Capability to Initialize Within Individual Set Calls2019-04-11T23:05:45ZJames WillenbringTempus: Add Stepper Capability to Initialize Within Individual Set Calls*Created by: ccober6*
One path for Stepper construction is to start with a default constructor and then use set* methods to specify parameters, and features of the Stepper for the current problem. After this the Stepper needs to call i...*Created by: ccober6*
One path for Stepper construction is to start with a default constructor and then use set* methods to specify parameters, and features of the Stepper for the current problem. After this the Stepper needs to call initialize() to finish the changes.
Here we would like to have each set function optional call initialize() internally, so that it is less likely to forget the initialize() call before starting the time integration.
@trilinos/tempus
## Expectations
Have the ability to call initialize within each set function, and multiple calls initialize is possible.
## Definition of Done
- [ ] Add optional calls to initialize from within set functions.
- [ ] Pass all tests.
- [ ] Add test(s) illustrating functionality.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4858ifpack2: build error with scalar=FLOAT and COMPLEX_DOUBLE enabled2019-04-11T16:10:32ZJames Willenbringifpack2: build error with scalar=FLOAT and COMPLEX_DOUBLE enabled*Created by: ajpowel*
@trilinos/ifpack2
## Current Behavior
```
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Vector_SIMD_Arith.hpp:619:5: note: template argument deducti...*Created by: ajpowel*
@trilinos/ifpack2
## Current Behavior
```
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Vector_SIMD_Arith.hpp:619:5: note: template argument deduction/substitution failed:
In file included from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Serial_Internal.hpp:12:0,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Serial_Impl.hpp:8,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_def.hpp:57,
from /scratch/ajpowel/code_032119/packages/ifpack2/src/Ifpack2_BlockTriDiContainer.hpp:2,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_ContainerFactory_def.hpp:51,
from /scratch/ajpowel/code_032119/packages/ifpack2/src/Ifpack2_ContainerFactory.hpp:2,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_BlockRelaxation_decl.hpp:52,
from /scratch/ajpowel/code_032119/packages/ifpack2/src/Ifpack2_BlockRelaxation.hpp:1,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_Details_OneLevelFactory_def.hpp:54,
from /scratch/ajpowel/code_032119/packages/ifpack2/src/Ifpack2_Details_OneLevelFactory.hpp:2,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_Details_Factory_def.hpp:46,
from /scratch/ajpowel/code_032119/packages/ifpack2/src/Ifpack2_Details_Factory.hpp:2,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_Factory_decl.hpp:48,
from /scratch/ajpowel/code_032119/packages/ifpack2/src/Ifpack2_Factory.hpp:1,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_Details_LinearSolverFactory_def.hpp:54,
from /scratch/ajpowel/code_032119/packages/ifpack2/src/Ifpack2_Details_LinearSolverFactory.hpp:2,
from /scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_Details_registerLinearSolverFactory.cpp:45:
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_InnerGemmFixC_Serial_Impl.hpp:1101:67: note: mismatched types 'Kokkos::complex<RealType1>' and 'double'
C[0*_cs0+0*_cs1] += alpha * c_00; C[0*_cs0+1*_cs1] += alpha * c_01;
~~~~~~^~~~~~
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_InnerGemmFixC_Serial_Impl.hpp: In instantiation of 'int KokkosBatched::Experimental::InnerGemmFixC<mb, nb>::serial_invoke(ScalarType, const ValueType*, const ValueType*, int, ValueType*) [with ScalarType = double; ValueType = KokkosBatched::Experimental::Vector<KokkosBatched::Experimental::SIMD<float>, 16>; int mb = 1; int nb = 1]':
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_InnerGemmFixC_Serial_Impl.hpp:1285:80: required from 'int KokkosBatched::Experimental::InnerGemmFixC<mb, nb>::serial_invoke(ScalarType, const ValueType*, const ValueType*, int, int, int, ValueType*) [with ScalarType = double; ValueType = KokkosBatched::Experimental::Vector<KokkosBatched::Experimental::SIMD<float>, 16>; int mb = 2; int nb = 2]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_InnerGemmFixC_Serial_Impl.hpp:1259:71: required from 'int KokkosBatched::Experimental::InnerGemmFixC<mb, nb>::serial_invoke(ScalarType, const ValueType*, const ValueType*, int, int, int, ValueType*) [with ScalarType = double; ValueType = KokkosBatched::Experimental::Vector<KokkosBatched::Experimental::SIMD<float>, 16>; int mb = 3; int nb = 3]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_InnerGemmFixC_Serial_Impl.hpp:1230:71: required from 'int KokkosBatched::Experimental::InnerGemmFixC<mb, nb>::serial_invoke(ScalarType, const ValueType*, const ValueType*, int, int, int, ValueType*) [with ScalarType = double; ValueType = KokkosBatched::Experimental::Vector<KokkosBatched::Experimental::SIMD<float>, 16>; int mb = 4; int nb = 4]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Team_Internal.hpp:139:13: required from 'KokkosBatched::Experimental::TeamGemmInternal<ArgAlgo>::invoke(const MemberType&, int, int, int, ScalarType, const ValueType*, int, int, const ValueType*, int, int, ScalarType, ValueType*, int, int) [with MemberType = MemberType; ScalarType = ScalarType; ValueType = ValueType; ArgAlgo = KokkosBatched::Experimental::Algo::Level3::Blocked]::<lambda(int, int, int, const ValueType*, const ValueType*, ValueType*)>::<lambda(const int&)> [with MemberType = Kokkos::Impl::HostThreadTeamMember<Kokkos::Serial>; ScalarType = double; ValueType = KokkosBatched::Experimental::Vector<KokkosBatched::Experimental::SIMD<float>, 16>]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Team_Internal.hpp:144:44: required from 'struct KokkosBatched::Experimental::TeamGemmInternal<ArgAlgo>::invoke(const MemberType&, int, int, int, ScalarType, const ValueType*, int, int, const ValueType*, int, int, ScalarType, ValueType*, int, int) [with MemberType = MemberType; ScalarType = ScalarType; ValueType = ValueType; ArgAlgo = KokkosBatched::Experimental::Algo::Level3::Blocked]::<lambda(int, int, int, const ValueType*, const ValueType*, ValueType*)> [with MemberType = Kokkos::Impl::HostThreadTeamMember<Kokkos::Serial>; ScalarType = double; ValueType = KokkosBatched::Experimental::Vector<KokkosBatched::Experimental::SIMD<float>, 16>]::<lambda(const int&)>'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_Gemm_Team_Internal.hpp:130:11: [ skipping 12 instantiation contexts, use -ftemplate-backtrace-limit=0 to disable ]
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos/core/src/Kokkos_Parallel.hpp:191:4: required from 'void Kokkos::parallel_for(const ExecPolicy&, const FunctorType&, const string&, typename Kokkos::Impl::enable_if<Kokkos::is_execution_policy<ExecPolicy>::value>::type*) [with ExecPolicy = Kokkos::TeamPolicy<Kokkos::Serial, Ifpack2::BlockTriDiContainerDetails::ExtractAndFactorizeTridiags<Tpetra::RowMatrix<float, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >::ExtractAndFactorizeTag>; FunctorType = Ifpack2::BlockTriDiContainerDetails::ExtractAndFactorizeTridiags<Tpetra::RowMatrix<float, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >; std::__cxx11::string = std::__cxx11::basic_string<char>; typename Kokkos::Impl::enable_if<Kokkos::is_execution_policy<ExecPolicy>::value>::type = void]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos/core/src/Kokkos_Parallel.hpp:244:25: required from 'void Kokkos::parallel_for(const string&, const ExecPolicy&, const FunctorType&) [with ExecPolicy = Kokkos::TeamPolicy<Kokkos::Serial, Ifpack2::BlockTriDiContainerDetails::ExtractAndFactorizeTridiags<Tpetra::RowMatrix<float, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >::ExtractAndFactorizeTag>; FunctorType = Ifpack2::BlockTriDiContainerDetails::ExtractAndFactorizeTridiags<Tpetra::RowMatrix<float, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > >; std::__cxx11::string = std::__cxx11::basic_string<char>]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_impl.hpp:1826:29: required from 'void Ifpack2::BlockTriDiContainerDetails::ExtractAndFactorizeTridiags<MatrixType>::run() [with MatrixType = Tpetra::RowMatrix<float, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_impl.hpp:1849:7: required from 'void Ifpack2::BlockTriDiContainerDetails::performNumericPhase(const Teuchos::RCP<const typename Ifpack2::BlockTriDiContainerDetails::ImplType<MatrixType>::tpetra_block_crs_matrix_type>&, const Ifpack2::BlockTriDiContainerDetails::PartInterface<MatrixType>&, Ifpack2::BlockTriDiContainerDetails::BlockTridiags<MatrixType>&, typename Ifpack2::BlockTriDiContainerDetails::ImplType<MatrixType>::magnitude_type) [with MatrixType = Tpetra::RowMatrix<float, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >; typename Ifpack2::BlockTriDiContainerDetails::ImplType<MatrixType>::tpetra_block_crs_matrix_type = Tpetra::Experimental::BlockCrsMatrix<float, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >; typename Ifpack2::BlockTriDiContainerDetails::ImplType<MatrixType>::magnitude_type = float]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_BlockTriDiContainer_def.hpp:235:9: required from 'void Ifpack2::BlockTriDiContainer<MatrixType, Ifpack2::BlockTriDiContainerDetails::ImplSimdTag>::compute() [with MatrixType = Tpetra::RowMatrix<float, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >]'
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/ifpack2/src/Ifpack2_Details_registerLinearSolverFactory.cpp:76:1: required from here
/scratch/ajpowel/code_032119/TPLs_src/Trilinos/packages/kokkos-kernels/src/batched/KokkosBatched_InnerGemmFixC_Serial_Impl.hpp:1139:33: error: no match for 'operator*' (operand types are 'const double' and 'KokkosBatched::Experimental::Vector<KokkosBatched::Experimental::SIMD<float>, 16>')
C[0*_cs0+0*_cs1] += alpha * c_00;
~~~~~~^~~~~~
```
## Steps to Reproduce
0) Comment out line 986 of $PROJECT/packages/tpetra/CMakeLists.txt (suppressing fail message to prevent possible Thyra build failure)
1) Configure Trilinos packages:
```
cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_Fortran_COMPILER=mpifort -DTrilinos_ENABLE_Tpetra=ON -DTrilinos_ENABLE_COMPLEX_DOUBLE=ON -DTrilinos_ENABLE_FLOAT=ON -DTrilinos_ENABLE_Teuchos=ON -DTrilinos_ENABLE_Teko=ON /scratch/ajpowel/code_032119/TPLs_src/Trilinos
```
2) Attempt to build ifpack2:
```
cd $PROJECT/packages/ifpack2
make -j 64
```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4842MueLu: update CUDA drivers on geminga2019-04-09T17:37:38ZJames WillenbringMueLu: update CUDA drivers on geminga*Created by: jhux2*
@trilinos/muelu
So we don't forget about this.*Created by: jhux2*
@trilinos/muelu
So we don't forget about this.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4833Amesos2: Build error when compiling with SuperLU 5.2.12019-04-23T16:27:51ZJames WillenbringAmesos2: Build error when compiling with SuperLU 5.2.1*Created by: Filipe-Cumaru*
<!---
Provide a general summary of the issue in the Title above. If this issue
pertains to a particular package in Trilinos, it's worthwhile to start the
title with "PackageName: ".
-->
<!---
Note t...*Created by: Filipe-Cumaru*
<!---
Provide a general summary of the issue in the Title above. If this issue
pertains to a particular package in Trilinos, it's worthwhile to start the
title with "PackageName: ".
-->
<!---
Note that anything between these delimiters is a comment that will not appear
in the issue description once created. Click on the Preview tab to see what
everything will look like when you submit.
-->
<!---
Feel free to delete anything from this template that is not applicable to the
issue you are submitting.
-->
<!---
Replace <teamName> below with the appropriate Trilinos package/team name.
-->
@trilinos/Amesos
@trilinos/Amesos2
<!---
Assignees: If you know anyone who should likely tackle this issue, select them
from the Assignees drop-down on the right.
-->
<!---
Lables: Choose any applicable package names from the Labels drop-down on the
right. Additionally, choose a label to indicate the type of issue, for
instance, bug, build, documentation, enhancement, etc.
-->
## Expectations
<!---
Tell us what you think should happen, how you think things should work, what
you would like to see in the documentation, etc.
-->
Amesos2 works with SuperLU and SuperLU-dist 5.
## Current Behavior
<!---
Tell us how the current behavior fails to meet your expectations in some way.
-->
I've got some compilation errors when trying to build Amesos2 with SuperLU support. Some of those are shown above. I've put the full list of errors in the attached file to avoid flooding this report.
[amesos_compilation_errors.txt](https://github.com/trilinos/Trilinos/files/3055731/amesos_compilation_errors.txt)
```
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superludist_FunctionMap.hpp:378:37: error: invalid conversion from ‘SLUD::int_t {aka int}’ to ‘SLUD::D::LUstruct_t*’ [-fpermissive]
SLUD::D::LUstructInit(m, n, lu);
^
In file included from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superludist_TypeMap.hpp:87:0,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superludist_FunctionMap.hpp:63,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superludist_decl.hpp:58,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superludist.hpp:47,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Factory.hpp:108,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Details_LinearSolverFactory_def.hpp:52,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Details_LinearSolverFactory.hpp:49,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Details_registerLinearSolverFactory.cpp:45:
/usr/include/superlu-dist/superlu_ddefs.h:262:13: note: declared here
extern void LUstructInit(const int_t, LUstruct_t *);
^~~~~~~~~~~~
In file included from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_decl.hpp:58:0,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu.hpp:47,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Factory.hpp:124,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Details_LinearSolverFactory_def.hpp:52,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Details_LinearSolverFactory.hpp:49,
from /home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Details_registerLinearSolverFactory.cpp:45:
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp: At global scope:
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp:108:19: error: variable or field ‘sgssvx’ declared void
sgssvx(SLU::superlu_options_t *, SLU::SuperMatrix *, int *, int *, int *,
^~~~~~~~~~~~~~~~~
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp:108:19: error: ‘superlu_options_t’ is not a member of ‘SLU’
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp:108:38: error: expected primary-expression before ‘,’ token
sgssvx(SLU::superlu_options_t *, SLU::SuperMatrix *, int *, int *, int *,
^
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp:115:19: error: ‘mem_usage_t’ is not a member of ‘SLU’
SLU::mem_usage_t *, SLU::SuperLUStat_t *, int *);
^~~~~~~~~~~
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp:115:32: error: expected primary-expression before ‘,’ token
SLU::mem_usage_t *, SLU::SuperLUStat_t *, int *);
^
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp:115:39: error: ‘SuperLUStat_t’ is not a member of ‘SLU’
SLU::mem_usage_t *, SLU::SuperLUStat_t *, int *);
^~~~~~~~~~~~~
/home/facsa/Downloads/Trilinos-master/packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp:459:15: error: ‘sgssvx’ is not a member of ‘SLU::S’
SLU::S::sgssvx(options, A, perm_c, perm_r, etree, equed, R, C, L, U, work,
^~~~~~
```
## Steps to Reproduce
<!---
Provide a link to a live example, or an unambiguous set of steps to reproduce
this issue. Include code to reproduce, if relevant.
1. Do this.
1. Do that.
1. Shake fist angrily at computer.
-->
I've configured Trilinos using cmake with the following options:
```
cmake \
-D CMAKE_INSTALL_PREFIX:PATH=/home/facsa/Trilinos-SuperLU \
\
-D MPI_BASE_DIR:PATH=/usr \
\
-D CMAKE_BUILD_TYPE:STRING=DEBUG \
-D CMAKE_Fortran_COMPILER:FILEPATH=/usr/bin/mpif90 \
-D CMAKE_CXX_FLAGS:STRING="-std=c++11 -O3" \
-D BUILD_SHARED_LIBS:BOOL=ON \
-D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
-D PYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3 \
\
-D Trilinos_ENABLE_CXX11=ON \
-D Trilinos_CXX11_FLAGS="-std=c++11" \
-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_Teuchos:BOOL=ON \
-D Trilinos_ENABLE_Epetra:BOOL=ON \
-D Trilinos_ENABLE_EpetraExt:BOOL=ON \
-D Trilinos_ENABLE_AztecOO:BOOL=ON \
-D Trilinos_ENABLE_Amesos2:BOOL=ON \
-D Trilinos_ENABLE_ML:BOOL=ON \
-D Trilinos_ENABLE_PyTrilinos:BOOL=OFF \
\
-D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \
-D Trilinos_ENABLE_TESTS:BOOL=OFF \
\
-D TPL_ENABLE_MATLAB:BOOL=OFF \
-D TPL_ENABLE_Matio:BOOL=OFF \
-D TPL_ENABLE_MPI:BOOL=ON \
-D TPL_ENABLE_BLAS:BOOL=ON \
-D TPL_ENABLE_LAPACK:BOOL=ON \
-D TPL_ENABLE_QT:BOOL=OFF \
-D TPL_ENABLE_X11:BOOL=OFF \
-D TPL_ENABLE_SuperLU:BOOL=ON \
-D TPL_SuperLU_LIBRARIES:PATH=/usr/lib/x86_64-linux-gnu \
-D TPL_SuperLU_INCLUDE_DIRS:PATH=/usr/include/superlu \
-D TPL_ENABLE_SuperLUDist:BOOL=ON \
-D TPL_SuperLUDist_INCLUDE_DIRS:PATH=/usr/include/superlu-dist \
-D TPL_SuperLUDist_LIBRARIES:PATH=/usr/lib/x86_64-linux-gnu \
-D TPL_ENABLE_ParMETIS:BOOL=ON \
\
-D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
-D Trilinos_VERBOSE_CONFIGURE:BOOL=OFF \
..
```
and then tried to build with ```make```. I've tried to build master branch and the [latest release](https://github.com/trilinos/Trilinos/releases/tag/trilinos-release-12-14-1). Both failed with the same errors.
## Your Environment
<!---
Include relevant details about your environment such that we can replicate this
issue.
-->
I'm trying to build Trilinos on Ubuntu 18.04 (Bionic Beaver). The SuperLU libraries were installed using ```apt-get install libsuperlu-dev libsuperlu-dist-dev```.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4810spack: Does a spack Trilinos variant exist for enabling CUDA support?2019-04-05T12:52:18ZJames Willenbringspack: Does a spack Trilinos variant exist for enabling CUDA support?*Created by: cwsmith*
I'd like to install Kokkos with the CUDA backend, via Trilinos, with spack. We use CMake for our projects that depend on Kokkos, hence the install via Trilinos.
Is there a branch of spack with a variant that en...*Created by: cwsmith*
I'd like to install Kokkos with the CUDA backend, via Trilinos, with spack. We use CMake for our projects that depend on Kokkos, hence the install via Trilinos.
Is there a branch of spack with a variant that enables CUDA in Trilinos/Kokkos?
Thank-you,
Cameronhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4807Percept: integrate as a new package into trilinos2019-04-04T16:56:32ZJames WillenbringPercept: integrate as a new package into trilinos*Created by: rppawlo*
Percept used to be a subpackage in STK during the "classic stk" era. When STK was refactored, percept was no longer ported over. Since percept is tied strongly to STK and the two must be in sync, it makes sense to ...*Created by: rppawlo*
Percept used to be a subpackage in STK during the "classic stk" era. When STK was refactored, percept was no longer ported over. Since percept is tied strongly to STK and the two must be in sync, it makes sense to start snapshotting percept into trilinos along with stk once again. This issue is to set up percept build as a tribits package.
Percept is developed in sierra and is thoroughly test with sierra. The tests will not be ported into trilinos - only a few small acceptance tests and some panzer tests will be in trilinos. This will prevent a large size increase in the Trilinos code base from binary mesh files.
1. [DONE] Set up snapshot tool. Port src directory only.
2. [DONE] Set up config file to mirror compile time defines.
3. Hide dependencies on cgns and open nurbs behind ifdefs so we can compile without those TPL requirements.
4. Setup test scripts for Brian and Byron to build against.
5. Coordinate with STK team on snapshots.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4802MueLu: adding test for structured line detection2019-04-03T16:17:34ZJames WillenbringMueLu: adding test for structured line detection*Created by: lucbv*
@trilinos/muelu
## Expectations
The structured line detection should be tested more thoroughly to make sure it is usable for applications such as SPARC
## Current Behavior
Currently only a unit-test checks t...*Created by: lucbv*
@trilinos/muelu
## Expectations
The structured line detection should be tested more thoroughly to make sure it is usable for applications such as SPARC
## Current Behavior
Currently only a unit-test checks that the structured line detection works but a full example is more appropriate to make sure the code behaves as expected in actual runs.
## Motivation and Context
This provides an example to user on how an xml file using structured line detection looks like, this will eventually be used to update the MueLu User Guide.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4779Teko_testdriver_tpetra_MPI_1 randomly failing in ATDM waterman build2019-04-01T16:37:24ZJames WillenbringTeko_testdriver_tpetra_MPI_1 randomly failing in ATDM waterman build*Created by: fryeguy52*
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https:/...*Created by: fryeguy52*
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-release-debug&field2=testname&compare2=61&value2=Teko_testdriver_tpetra_MPI_1&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=2019-04-01&field5=buildstarttime&compare5=83&value5=2019-02-28) the test:
* Teko_testdriver_tpetra_MPI_1
looks to be randomly failing in the build:
* Trilinos-atdm-waterman-cuda-9.2-release-debug
It has failed 5 times in the last month each time with:
```
terminate called after throwing an instance of 'std::runtime_error'
what(): cudaGetLastError() error( cudaErrorAssert): device-side assert triggered /home/jenkins/waterman/workspace/Trilinos-atdm-waterman-cuda-9.2-release-debug/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaExec.hpp:401
Traceback functionality not available
```
full output from a failed run can be found [here](https://testing.sandia.gov/cdash/testDetails.php?test=72920917&build=4811399)
## Current Status on CDash
current 4 week history can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-release-debug&field2=testname&compare2=61&value2=Teko_testdriver_tpetra_MPI_1&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=4%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4775Trilinos PR testing missed Ifpack2 compile issue2019-04-01T16:23:52ZJames WillenbringTrilinos PR testing missed Ifpack2 compile issue*Created by: prwolfe*
From issue #4742
> Hmmm, this is short on details, but I just ran into an issue when using a recent master checkout of Trilinos.
>
> Basically lines 2472 and 2478 in the file packages/ifpack2/src/Ifpack2_Blo...*Created by: prwolfe*
From issue #4742
> Hmmm, this is short on details, but I just ran into an issue when using a recent master checkout of Trilinos.
>
> Basically lines 2472 and 2478 in the file packages/ifpack2/src/Ifpack2_BlockTriDiContainer_impl.hpp are using a const instance of a class (Kokkos::TeamPolicy) to call a const function for a non-const instance. Being named a "set" function I tried removing the "const" from the call in Ifpack2_BlockTriDiContainer_impl.hpp and got a clean build.
>
> The $6 question is how did this get through Trilinos testing and how do we fix that. This is a very simple issue which should have been caught early.
>
> _Originally posted by @prwolfe in https://github.com/trilinos/Trilinos/issues/4742#issuecomment-478109186_
>
This issue is to resolve what is missing in the testing and how to resolve that.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4774EpetraExt unit test error for missing HDF52019-04-01T07:03:36ZJames WillenbringEpetraExt unit test error for missing HDF5*Created by: MarDiehl*
When building Trilinos 12.14.1, I get the error
`error: ‘HDF5’ was not declared in this scope`.
in `Trilinos-trilinos-release-12-14-1/packages/epetraext/test/inout/EpetraExt_HDF5_UnitTest.cpp`.
The full log...*Created by: MarDiehl*
When building Trilinos 12.14.1, I get the error
`error: ‘HDF5’ was not declared in this scope`.
in `Trilinos-trilinos-release-12-14-1/packages/epetraext/test/inout/EpetraExt_HDF5_UnitTest.cpp`.
The full log is here [Error.log](https://github.com/trilinos/Trilinos/files/3028041/Error.log)
I have configures Trilinos with
`cmake .. -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON \
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=ON \
-DTrilinos_ENABLE_Gtest:BOOL=ON \
-DTrilinos_ENABLE_TESTS=ON \
-DTPL_ENABLE_gtest:BOOL=ON \
-DTPL_ENABLE_MPI:BOOL=ON \
-DPYTHON_EXECUTABLE:PATH=/usr/bin/python2 \
-DCMAKE_INSTALL_PREFIX:PATH=/usr \
-DBUILD_SHARED_LIBS:BOOL=ON
make VERBOSE=1`
and HDF5 is not added automatically, see [Configure.log](https://github.com/trilinos/Trilinos/files/3028040/Configure.log)
I'll now try to build explicitly with HDF5.Still, I think that the EpetraEx HDF5 unit test should be disabled if HDF5 is not added as a third partly library
## Teams
I don't know how to mention the EpetraExt team
## Labels
I don't see a drop down list for labels
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4762Segfault with Spectrum MPI when TPETRA_ASSUME_CUDA_AWARE_MPI=12019-03-29T22:25:11ZJames WillenbringSegfault with Spectrum MPI when TPETRA_ASSUME_CUDA_AWARE_MPI=1*Created by: jhux2*
I am running the mini-app "MiniEM" in Panzer on the LLNL machine rzansel, which is using IBM Spectrum MPI. When I set `export TPETRA_ASSUME_CUDA_AWARE_MPI=1`, the job crashes. When this variable is not set, the job...*Created by: jhux2*
I am running the mini-app "MiniEM" in Panzer on the LLNL machine rzansel, which is using IBM Spectrum MPI. When I set `export TPETRA_ASSUME_CUDA_AWARE_MPI=1`, the job crashes. When this variable is not set, the job runs to completion.
The version of spectrum mpi is `10.2.0.11rtm2`. Spectrum MPI is based on OpenMPI, and there are known problems with certain versions of OpenMPI, see #3405.
<details>
<summary>error log</summary>
```
[rzansel46:108415] *** Process received signal ***
[rzansel46:108415] Signal: Segmentation fault (11)
[rzansel46:108415] Signal code: Address not mapped (1)
[rzansel46:108415] Failing at address: 0x20007f210280
[rzansel46:108415] [ 0] [0x2000000504d8]
[rzansel46:108415] [ 1] [0x0]
[rzansel46:108415] [ 2] /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/pami_port/libpami.so.3(_ZN4PAMI4Fifo8WrapFifoINS0_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAtomicEEELj256EE18producePacket_implINS_6Device5Shmem17PacketIovecWriterILj2EEEEEbRT_+0xb8)[0x200018db0d18]
[rzansel46:108415] [ 3] /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/pami_port/libpami.so.3(_ZN4PAMI8Protocol4Send11EagerSimpleINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEELNS1_15configuration_tE1EE11simple_implEP11pami_send_t+0x498)[0x200018db1258]
[rzansel46:108415] [ 4] /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/pami_port/libpami.so.3(_ZN4PAMI8Protocol4Send5EagerINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEENS3_3IBV11PacketModelINSN_6DeviceELb1EEEE9EagerImplILNS1_15configuration_tE1ELb1EE6simpleEP11pami_send_t+0x2c)[0x200018db16fc]
[rzansel46:108415] [ 5] /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/pami_port/libpami.so.3(PAMI_Send+0x58)[0x200018ce1478]
[rzansel46:108415] [ 6] /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/spectrum_mpi/mca_pml_pami.so(pml_pami_send+0x628)[0x200013e3ec98]
[rzansel46:108415] [ 7] /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/spectrum_mpi/mca_pml_pami.so(mca_pml_pami_send+0x568)[0x200013e3fe58]
[rzansel46:108415] [ 8] /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/libmpi_ibm.so.3(PMPI_Send+0x148)[0x200011a5f918]
[rzansel46:108415] [ 9] ./PanzerMiniEM_BlockPrec.exe[0x3504ec34]
[rzansel46:108415] [10] ./PanzerMiniEM_BlockPrec.exe[0x3504eef8]
[rzansel46:108415] [11] ./PanzerMiniEM_BlockPrec.exe[0x33bff3b0]
[rzansel46:108415] [12] ./PanzerMiniEM_BlockPrec.exe[0x33c02558]
[rzansel46:108415] [13] ./PanzerMiniEM_BlockPrec.exe[0x33c048a8]
[rzansel46:108415] [14] ./PanzerMiniEM_BlockPrec.exe[0x33c86d2c]
[rzansel46:108415] [15] ./PanzerMiniEM_BlockPrec.exe[0x33c9b86c]
[rzansel46:108415] [16] ./PanzerMiniEM_BlockPrec.exe[0x33cb9314]
[rzansel46:108415] [17] ./PanzerMiniEM_BlockPrec.exe[0x180c028c]
[rzansel46:108415] [18] ./PanzerMiniEM_BlockPrec.exe[0x180fce30]
[rzansel46:108415] [19] ./PanzerMiniEM_BlockPrec.exe[0x180584e8]
[rzansel46:108415] [20] ./PanzerMiniEM_BlockPrec.exe[0x18058b34]
[rzansel46:108415] [21] ./PanzerMiniEM_BlockPrec.exe[0x17ffa384]
[rzansel46:108415] [22] ./PanzerMiniEM_BlockPrec.exe[0x14e8e80c]
[rzansel46:108415] [23] ./PanzerMiniEM_BlockPrec.exe[0x11660714]
[rzansel46:108415] [24] ./PanzerMiniEM_BlockPrec.exe[0x117ec69c]
[rzansel46:108415] [25] /lib64/libc.so.6(+0x25100)[0x200011da5100]
[rzansel46:108415] [26] /lib64/libc.so.6(__libc_start_main+0xc4)[0x200011da52f4]
[rzansel46:108415] *** End of error message ***
ERROR: One or more process (first noticed rank 3) terminated with signal 11 (core dumped)
```
</details>
This issue is meant to record the problem and see if there are any known fixes.
Module environment
```
Currently Loaded Modules:
1) cuda/9.2.148
2) StdEnv
3) xl/2018.11.26
4) spectrum-mpi/rolling-release
5) lapack/3.8.0-P9-xl-2018.08.24
6) cmake/3.12.1
7) hdf5-parallel/1.10.4
```
@trilinos/tpetra @nmhamster https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4757Tpetra::DistObject: Evaluate host-pinned buffers for CUDA + MPI2019-04-29T18:30:27ZJames WillenbringTpetra::DistObject: Evaluate host-pinned buffers for CUDA + MPI*Created by: mhoemmen*
@trilinos/tpetra
`Tpetra::DistObject` currently offers two options for CUDA + MPI:
1. Give CudaSpace buffers to MPI
2. Give HostSpace buffers to MPI
(DistObject lets subclasses pack and unpack where...*Created by: mhoemmen*
@trilinos/tpetra
`Tpetra::DistObject` currently offers two options for CUDA + MPI:
1. Give CudaSpace buffers to MPI
2. Give HostSpace buffers to MPI
(DistObject lets subclasses pack and unpack wherever they like, as long as they update the sync state correctly.)
We want to evaluate a third option: Give `CudaHostPinnedSpace` buffers to MPI. Host-pinned memory behaves like CudaUVMSpace, in the sense that both host and device can access it. However, MPI sees host-pinned memory as host memory. This means two things:
1. MPI need not be CUDA aware in order to access host-pinned memory, yet we can still pack and unpack on device.
2. We don't have to worry about CUDA-aware MPI being slow.
The latter is important, since we've observed some "CUDA-aware MPI" implementations being slow in practice.
Host-pinned memory has a high allocation cost. It may make sense to start with the static View allocation functions in PR #4734. We can't use those without further work, because each DistObject instance will need its own pack and unpack buffers. This may call for a simple memory pool, and for changes to DistObject and/or subclasses so that they only hold buffer allocations while communication is active.
## Definition of Done
- [x] Write a Tpetra test that prototypes use of CudaHostPinnedSpace for communication.
- [ ] Write a benchmark to compare performance of point-to-point communication with CudaHostPinnedSpace vs. CudaSpace communication buffers.
- [ ] If CudaHostPinnedSpace pays off, change DistObject to use it for communication buffers.
- [ ] Evaluate performance with a Tpetra benchmark.
- [ ] If performance is good, deploy solution in Tpetra.
## Related Issues
* Is blocked by #4734 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4755Tpetra: remove DynamicProfile code paths2019-03-28T18:26:16ZJames WillenbringTpetra: remove DynamicProfile code paths*Created by: tjfulle*
@trilinos/tpetra
Remove all `DynamicProfile` code paths from Tpetra. This may mean first putting all the `DynamicProfile` specific code behind compile guards that are enabled with deprecated code, or just remo...*Created by: tjfulle*
@trilinos/tpetra
Remove all `DynamicProfile` code paths from Tpetra. This may mean first putting all the `DynamicProfile` specific code behind compile guards that are enabled with deprecated code, or just removing the code in one fell swoop.Tpetra: Deprecate DynamicProfile