Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2019-05-07T01:09:59Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5002MueLu: MueLu_FixedMatrixPattern-Tpetra_MPI_4 randomly timing out in ATDM wat...2019-05-07T01:09:59ZJames WillenbringMueLu: MueLu_FixedMatrixPattern-Tpetra_MPI_4 randomly timing out in ATDM waterman build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this q...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-debug&field2=testname&compare2=61&value2=MueLu_FixedMatrixPattern-Tpetra_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=2019-04-23T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-24T00%3A00%3A00) the test:
* MueLu_FixedMatrixPattern-Tpetra_MPI_4
is randomly timing out in the build:
* Trilinos-atdm-waterman-cuda-9.2-debug
This test usually passes in about 6.5 seconds but has timed out (10 minutes) 5 times in the last 30 days
## Current Status on CDash
[Current 2 week history on CDash](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-debug&field2=testname&compare2=61&value2=MueLu_FixedMatrixPattern-Tpetra_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=2%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2506FEI_fei_ubase_MPI_4 failed due to jumbled output breaking pass/fail grep2019-05-06T23:01:28ZJames WillenbringFEI_fei_ubase_MPI_4 failed due to jumbled output breaking pass/fail grep*Created by: bartlettroscoe*
**CC:** @trilinos/fei, @trilinos/framework
The test `FEI_fei_ubase_MPI_4` failed in the standard CI build just now s shown at:
* https://testing.sandia.gov/cdash/testDetails.php?test=46227718&build=34...*Created by: bartlettroscoe*
**CC:** @trilinos/fei, @trilinos/framework
The test `FEI_fei_ubase_MPI_4` failed in the standard CI build just now s shown at:
* https://testing.sandia.gov/cdash/testDetails.php?test=46227718&build=3482611
which showed:
```
Exit Value: 0
Required regular expression not found.Regex=[End Result: TEST PASSED<br /> ]
...
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1113 and rank 0!
***
*** Unit test suite ...
***
Sorting tests by group name then by the order they were added ... (time = 0.000279)
Running unit tests ...
[...]
Total Time: 0.00781 sec
Summary: total = 54, run = 54, passed = 54, failed = 0
End ReTeuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1113 and rank 1!
skipping test of fei::DirichletBCManager::finalizeBCEqn, which only runs on 1 proc.
test Eqns_unit.feiInitSlave only runs on 2 procs. returning.
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1113 and rank 2!
skipping test of fei::DirichletBCManager::finalizeBCEqn, which only runs on 1 proc.
test Eqns_unit.feiInitSlave only runs on 2 procs. returning.
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1113 and rank 3!
skipping test of fei::DirichletBCManager::finalizeBCEqn, which only runs on 1 proc.
test Eqns_unit.feiInitSlave only runs on 2 procs. returning.
sult: TEST PASSED
```
The problem is that the startup banner from the non-root processes got in the way of `End Result: TEST PASSED`.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4525Tpetra: Increasing number of threads leads to decrease of performance2019-05-06T22:45:30ZJames WillenbringTpetra: Increasing number of threads leads to decrease of performance*Created by: freaklovesmango*
That post refers a little to that one: #3827
I wanted to change Epetra to Tpetra, expecting a shorter execution time by increasing the number of threads while using just one MPI process, thus using share...*Created by: freaklovesmango*
That post refers a little to that one: #3827
I wanted to change Epetra to Tpetra, expecting a shorter execution time by increasing the number of threads while using just one MPI process, thus using shared memory functions via Tpetra/Kokkos.
Unfortunately, the results get worse by increasing the number of threads... and again, I wonder why.
I am using Belos with Tpetra as the underlying library on a x86-64 system.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/5027Framework: Installation Testing2019-05-06T16:59:22ZJames WillenbringFramework: Installation Testing*Created by: william76*
## Enhancement
@trilinos/framework
@bartlettroscoe
@jwillenbring
As part of the roadmap towards implementing a backwards compatibility test to the Trilinos testing suite we need to first get Installation...*Created by: william76*
## Enhancement
@trilinos/framework
@bartlettroscoe
@jwillenbring
As part of the roadmap towards implementing a backwards compatibility test to the Trilinos testing suite we need to first get Installation Testing up and running.
This will require some updates to [TriBiTS][1] to support some of the installation testing capabilities that we will need. @bartlettroscoe can fill in more details on what we're needing to do there.
[1]: https://tribits.org/Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4951Tpetra: link errors in Norms with deprecated code OFF2019-05-03T23:00:55ZJames WillenbringTpetra: link errors in Norms with deprecated code OFF*Created by: kddevin*
## Bug Report
@trilinos/tpetra
### Description
Using the develop branch with Tpetra_ENABLE_DEPRECATED_CODE=OFF, I see the following link errors:
```
[ 50%] Linking CXX executable TpetraCore_BlockExpNamespac...*Created by: kddevin*
## Bug Report
@trilinos/tpetra
### Description
Using the develop branch with Tpetra_ENABLE_DEPRECATED_CODE=OFF, I see the following link errors:
```
[ 50%] Linking CXX executable TpetraCore_BlockExpNamespace.exe
Undefined symbols for architecture x86_64:
"void Tpetra::Details::normImpl<int, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, int>(int*, Kokkos::View<int const**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> > const&, Tpetra::Details::EWhichNorm, Teuchos::ArrayView<unsigned long const> const&, bool, bool, Teuchos::Comm<int> const*)", referenced from:
void (anonymous namespace)::multiVectorNormImpl<int, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Tpetra::MultiVector<int, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::mag_type*, Tpetra::MultiVector<int, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >&, Tpetra::Details::EWhichNorm) in libtpetra.a(Tpetra_MultiVector_INT_INT_LONG_LONG_SERIAL.cpp.o)
ld: symbol(s) not found for architecture x86_64
```
### Steps to Reproduce
```
cmake \
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-D CMAKE_BUILD_TYPE:STRING="DEBUG" \
-D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
\
-D MPI_BIN_DIR:PATH="/Users/kddevin/InstalledSoftware/openmpi-1.8.3_clang/bin" \
-D TPL_ENABLE_MPI:BOOL=ON \
-D MPI_EXEC_MAX_NUMPROCS:STRING=11 \
\
-D TPL_ENABLE_BinUtils:BOOL=OFF \
-D TPL_ENABLE_Pthread:BOOL=OFF \
\
-D CMAKE_C_FLAGS:STRING="-Wall -pedantic -Wno-unknown-pragmas -Wno-narrowing -Wno-inline -Wshadow -Wdeprecated-declarations -Wempty-body -Wignored-qualifiers -Wmissing-field-initializers -Wsign-compare -Wtype-limits -Wuninitialized -Winit-self -fstrict-aliasing -Wno-long-long" \
-D CMAKE_CXX_FLAGS:STRING="-Wall -pedantic -Wno-unknown-pragmas -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-inline -Wshadow -Wdeprecated-declarations -Wempty-body -Wignored-qualifiers -Wmissing-field-initializers -Wsign-compare -Wtype-limits -Wuninitialized -Winit-self -fstrict-aliasing" \
\
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_TESTS:BOOL=OFF \
-D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \
\
-D Trilinos_ENABLE_SHADOW_WARNINGS:BOOL=ON \
-D Trilinos_VERBOSE_CONFIGURE:BOOL=OFF \
-D Trilinos_ENABLE_Fortran:BOOL=OFF \
\
-D Trilinos_ENABLE_Tpetra:BOOL=ON \
-D Tpetra_ENABLE_TESTS:BOOL=ON \
-D Tpetra_ENABLE_EXAMPLES:BOOL=ON \
\
-D Tpetra_ENABLE_DEPRECATED_CODE:BOOL=OFF \
-D Tpetra_INST_INT_LONG_LONG:BOOL=ON \
-D Tpetra_INST_INT_INT:BOOL=OFF \
\
-D Teuchos_ENABLE_STACKTRACE:BOOL=OFF \
-D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON \
..
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3137MueLu: Static library is HUGE; splitting by GlobalOrdinal etc. won't help2019-05-03T22:19:12ZJames WillenbringMueLu: Static library is HUGE; splitting by GlobalOrdinal etc. won't help*Created by: mhoemmen*
@trilinos/muelu @micahahoward @tcfisher
SPARC wasn't actually able to build MueLu before on Intel 17, static debug, because the library was too huge for the linker. PR #3100 fixes this for ATDM Dashboard buil...*Created by: mhoemmen*
@trilinos/muelu @micahahoward @tcfisher
SPARC wasn't actually able to build MueLu before on Intel 17, static debug, because the library was too huge for the linker. PR #3100 fixes this for ATDM Dashboard builds by using a new BinUtils module, and thus a new linker. I fixed this for SPARC by setting `Tpetra_INST_INT_INT=OFF` and `Amesos2_ENABLE_Epetra=OFF`.
While I'm able to build Trilinos now, `libmuelu.a` is still 3.8G. This is with only one Scalar type (`double`), one GlobalOrdinal type (`long long`), and one Node type (`OpenMP`) enabled. That suggests that splitting MueLu's library by GlobalOrdinal and/or Node won't actually help shrink the library. Instead, if we want to split it, we'll need to split it by topic.
## Expectations
Libraries shouldn't be so huge that they require 64-bit linkers.
## Current Behavior
`libmuelu.a` is 3.8G when I have only one Scalar, GlobalOrdinal, and Node type combination enabled.
## Possible Solution
Split `libmuelu` by topic, e.g., smoothers, aggregation, etc.
## Steps to Reproduce
Intel 17 static debug build, ATDM libraries.
## Related Issues
* Follows #3069
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3178Pull request testing should set -Werror2019-05-02T22:07:04ZJames WillenbringPull request testing should set -Werror*Created by: mhoemmen*
@vbrunini asks whether pull request testing could set `-Werror`, so as to avoid issues like #3177.
@trilinos/framework @khpierson
## Expectations
Trilinos -- at least the library, not necessarily tests a...*Created by: mhoemmen*
@vbrunini asks whether pull request testing could set `-Werror`, so as to avoid issues like #3177.
@trilinos/framework @khpierson
## Expectations
Trilinos -- at least the library, not necessarily tests and examples -- should build warning-free.
## Current Behavior
See #3177. There is an issue that it's impossible to fix warnings in some packages.
## Motivation and Context
Sierra builds with warnings as errors, so they want Trilinos to build warning-free.
## Possible Solution
Exclude legacy packages like ML. Fix warnings. Add `-Werror` to at least one PR build.
## Related Issues
* Related to #3177 Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5033MueLu: Tests failing on ATDM cuda 10 build2019-05-02T19:11:33ZJames WillenbringMueLu: Tests failing on ATDM cuda 10 build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this q...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=MueLu_&field3=site&compare3=61&value3=white&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=84&value5=2019-04-29T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-30T00%3A00%3A00) the tests:
* MueLu_Maxwell3D-Epetra_MPI_4
* MueLu_ImportPerformance_Epetra_MPI_4
* MueLu_ImportPerformance_Tpetra_MPI_4
are failing in the build:
* Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
this is common in the output:
```
MueLu_ImportPerformance.exe: sys/memtype_cache.c:90: ucs_memtype_cache_delete: Assertion `pgt_region != ((void *)0)' failed.
```
## Current Status on CDash
[Current failing MueLu tests on this build](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=MueLu_&field3=site&compare3=61&value3=white&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=84&value5=today&field6=buildstarttime&compare6=83&value6=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3958Trilinos needs automated coverage testsing2019-05-02T17:46:20ZJames WillenbringTrilinos needs automated coverage testsing*Created by: bartlettroscoe*
@trilinos/framework
Trilinos currently has no coverage testing. The last coverage build posted to CDash was for MueLu back on 8/20/2018 as shown in [this query](https://testing.sandia.gov/cdash-dev-view...*Created by: bartlettroscoe*
@trilinos/framework
Trilinos currently has no coverage testing. The last coverage build posted to CDash was for MueLu back on 8/20/2018 as shown in [this query](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&date=2018-11-28&filtercount=2&showfilters=1&filtercombine=and&field1=hascoverage&compare1=1&value1=&field2=buildstarttime&compare2=84&value2=now).
We need regular automated coverage builds posting to CDash.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5036Zoltan2: Tests failing on ATDM cuda 10 build2019-05-02T17:33:07ZJames WillenbringZoltan2: Tests failing on ATDM cuda 10 build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/zoltan2, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this q...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/zoltan2, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Zoltan2&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=2019-04-29T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-30T00%3A00%3A00) the tests:
* Zoltan2_directoryTest_Kokkos_MPI_4
* Zoltan2_directoryTest_KokkosSimple_MPI_4
* Zoltan2_directoryTest_findUniqueGids.cpp_MPI_4
are failing in the build:
* Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
## Current Status on CDash
[Failing zoltan2 tests for the current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Zoltan2&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=today&field6=buildstarttime&compare6=83&value6=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Zoltan2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5035Teko: Tests failing on ATDM cuda 10 build2019-05-02T17:17:38ZJames WillenbringTeko: Tests failing on ATDM cuda 10 build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this qu...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Teko_&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=2019-04-29T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-30T00%3A00%3A00) the tests:
* Teko_testdriver_MPI_1
* Teko_testdriver_MPI_4
* Teko_testdriver_tpetra_MPI_1
* Teko_testdriver_tpetra_MPI_4 |
are failing in the build:
* Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
## Current Status on CDash
[Failing Teko tests on this build for current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Teko_&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=today&field6=buildstarttime&compare6=83&value6=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3276Trilinos auto PR tester stability issues2019-05-02T13:20:11ZJames WillenbringTrilinos auto PR tester stability issues*Created by: bartlettroscoe*
@trilinos/framework
## Description
Over the last few weeks and months, the Trilinos auto PR tester has seen several cases where one or more PR builds for a given PR testing iteration failed to produce ...*Created by: bartlettroscoe*
@trilinos/framework
## Description
Over the last few weeks and months, the Trilinos auto PR tester has seen several cases where one or more PR builds for a given PR testing iteration failed to produce results on CDash or showed build or test failures that were not related to the changes on that particular PR.
This Story is to log these fails and keep track of them in order to provide some statistics about these cases in order to inform how to address them. This should replace making comments in individual PRs that exhibit these types of problems like #3260 and #3213.
## PR Builds Showing Random Failures
Below are a few examples of the stability problems (but are not all of the problems).
| PR ID | Num PR Builds to reach passing | First test trigger | Start first test| Passing test | Merge PR |
| --: | --: | --: | --: | --: | --: |
| #3258 | 2 | [8/8/2018 2:35 PM ET](https://github.com/trilinos/Trilinos/pull/3258#issue-207098955) | [8/8/2018 2:44 PM](https://github.com/trilinos/Trilinos/pull/3258#issuecomment-411510956) | [8/8/2018 9:15 PM ET]() | Not merged |
| #3260 | 4 | [8/8/2018 5:22 PM ET](https://github.com/trilinos/Trilinos/pull/3260#issue-207141537) | [8/8/2018 6:31 PM ET](https://github.com/trilinos/Trilinos/pull/3260#issuecomment-411574370) | [8/10/2018 4:13 AM ET](https://github.com/trilinos/Trilinos/pull/3260#issuecomment-412010497) | [8/10/2018 8:25 AM](https://github.com/trilinos/Trilinos/pull/3260#event-1782381644) |
| #3213 | 3 | [7/31/2018 4:30 PM ET](https://github.com/trilinos/Trilinos/pull/3213#issue-205233060) | [7/31/2018 4:57 PM ET](https://github.com/trilinos/Trilinos/pull/3213#issuecomment-409365522) | [8/1/2018 9:48 AM ET](https://github.com/trilinos/Trilinos/pull/3213#issuecomment-409580677) | [8/1/2018 9:53 AM ET](https://github.com/trilinos/Trilinos/pull/3213#event-1765281809) |
| #3098 | 4 | [7/12/2018 12:52 PM ET](https://github.com/trilinos/Trilinos/pull/3098#issue-201063953) | [7/12/2018 1:07 PM ET](https://github.com/trilinos/Trilinos/pull/3098#issuecomment-404582631) | [7/13/2018 11:12 PM ET](https://github.com/trilinos/Trilinos/pull/3098#issuecomment-404994581) | [7/14/2018 10:59 PM ET](https://github.com/trilinos/Trilinos/pull/3098#event-1733896640) |
| #3369 | 6 | [8/29/2018 9:08 AM ET](https://github.com/trilinos/Trilinos/pull/3369#issue-211746901) | [8/29/2018 9:16 AM ET](https://github.com/trilinos/Trilinos/pull/3369#issuecomment-416948915) | [8/31/2018 6:09 AM ET](https://github.com/trilinos/Trilinos/pull/3369#issuecomment-417618824) | [8/31/2018 8:33 AM ET](https://github.com/trilinos/Trilinos/pull/3369#event-1820478271) |
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4900Tpetra/Xpetra/MueLu: Purge the Clones2019-05-01T22:49:07ZJames WillenbringTpetra/Xpetra/MueLu: Purge the Clones*Created by: csiefer2*
See discussion in #4893 *Created by: csiefer2*
See discussion in #4893 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4988STK: STKUnit_tests_stk_topology_unit_tests_MPI_1 failing in ATDM build cuda-...2019-05-01T22:37:42ZJames WillenbringSTK: STKUnit_tests_stk_topology_unit_tests_MPI_1 failing in ATDM build cuda-9.2-gnu-7.2.0-rdc-release-debug-pt on 'white' and 'ride'*Created by: fryeguy52*
## Bug Report
CC: @trilinos/stk, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packag...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/stk, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt&field2=testname&compare2=61&value2=STKUnit_tests_stk_topology_unit_tests_MPI_1&field3=site&compare3=61&value3=ride&field4=buildstarttime&compare4=84&value4=2019-04-22T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-23T00%3A00%3A00) the test:
* STKUnit_tests_stk_topology_unit_tests_MPI_1
is failing since 2019-04-09 in the build:
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
<more-details>
## Current Status on CDash
[The current status of this test](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt&field2=testname&compare2=61&value2=STKUnit_tests_stk_topology_unit_tests_MPI_1&field3=site&compare3=61&value3=ride&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_STK=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4986Xpetra: implement ETI system2019-05-01T17:33:51ZJames WillenbringXpetra: implement ETI system*Created by: jhux2*
In order to decrease downstream package's library sizes, such as MueLu's, Xpetra needs to allow for ETI (explicit template instantiation).
@trilinos/xpetra @cgcgcg @lucbv
Related to #4984, #3137.*Created by: jhux2*
In order to decrease downstream package's library sizes, such as MueLu's, Xpetra needs to allow for ETI (explicit template instantiation).
@trilinos/xpetra @cgcgcg @lucbv
Related to #4984, #3137.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4952Framework: Does ATDM testing set MPI_EXEC correctly?2019-04-30T17:47:48ZJames WillenbringFramework: Does ATDM testing set MPI_EXEC correctly?*Created by: kddevin*
## Bug Report
@trilinos/framework @bartlettroscoe
### Description
I am trying to use ATDM testing as instructed in #4904
The compilation works fine, but the tests all fail. I suspect the problem is that ct...*Created by: kddevin*
## Bug Report
@trilinos/framework @bartlettroscoe
### Description
I am trying to use ATDM testing as instructed in #4904
The compilation works fine, but the tests all fail. I suspect the problem is that ctest is using an incompatible version of mpiexec. Should the atdm scripts set MPI_EXEC as they set the CMAKE_*_COMPILER flags?
```
-- MPI_USE_COMPILER_WRAPPERS='ON'
-- Leaving current CMAKE_C_COMPILER=/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/7.2.0/openmpi/1.10.1/bin/mpicc since it is already set!
-- Leaving current CMAKE_CXX_COMPILER=/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/7.2.0/openmpi/1.10.1/bin/mpicxx since it is already set!
-- MPI_EXEC='mpiexec'
-- MPI_EXEC='/usr/local/bin/mpiexec'
```
### Steps to Reproduce
```
[vesper Obj_atdm]$ source ../cmake/std/atdm/load-env.sh gnu-7.2.0-openmp-release-debug
Hostname 'vesper.sandia.gov' matches known ATDM host 'sems-rhel6' and system 'sems-rhel6'
Setting compiler and build options for buld name 'gnu-7.2.0-openmp-release-debug'
Using SEMS RHEL6 compiler stack GNU-7.2.0 to build RELEASE-DEBUG code with Kokkos node type OPENMP
[vesper Obj_atdm]$ which mpiexec
/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/7.2.0/openmpi/1.10.1/bin/mpiexec
[vesper Obj_atdm]$ cmake \
> -GNinja \
> -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
> -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Piro=ON \
> .. >& output.txt
[vesper Obj_atdm]$ grep -i mpiexec output.txt
-- MPI_EXEC='mpiexec'
-- MPI_EXEC='/usr/local/bin/mpiexec
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/5042Tpetra: fix deprecated case in RowMatrix::add2019-04-30T03:44:47ZJames WillenbringTpetra: fix deprecated case in RowMatrix::add*Created by: kddevin*
## Bug Report
@trilinos/tpetra
relate to #4701
### Description
One case of RowMatrix::add is not handled correctly in the deprecation of DynamicProfile. This case is rarely, if ever, used. we should ei...*Created by: kddevin*
## Bug Report
@trilinos/tpetra
relate to #4701
### Description
One case of RowMatrix::add is not handled correctly in the deprecation of DynamicProfile. This case is rarely, if ever, used. we should either fix it and test it so it works, or throw an error to state we don't support it. (I prefer the first.)
for details and a proposed solution, see https://github.com/trilinos/Trilinos/pull/4701/files/cfd349400d2faa8009822abf165b7c7cfcfa7e74#diff-c109d71d09ef95780e788d789a64f2d9
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/5034Framework: apparent problem during PR cmake configure for CUDA2019-04-29T19:00:46ZJames WillenbringFramework: apparent problem during PR cmake configure for CUDA*Created by: jhux2*
@trilinos/framework The [results](https://testing.sandia.gov/cdash/index.php?project=Trilinos&filtercount=1&showfilters=1&field1=buildname&compare1=65&value1=PR-5028) for PR #5028 are showing a lot of tests not run f...*Created by: jhux2*
@trilinos/framework The [results](https://testing.sandia.gov/cdash/index.php?project=Trilinos&filtercount=1&showfilters=1&field1=buildname&compare1=65&value1=PR-5028) for PR #5028 are showing a lot of tests not run for CUDA. The cmake log has the following. Is this perhaps a bug in the configure script itself?
```
Finished configuring Trilinos!
Total time to configure Trilinos: 1m53.234s
-- Configuring done
CMake Warning:
Value of Trilinos_ENABLE_TESTS contained a newline; truncating
-- Generating done
CMake Warning:
Value of Trilinos_ENABLE_TESTS contained a newline; truncating
-- Build files have been written to: /home/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_cuda_9.2/pull_request_test
```
@trilinos/framework @bartlettroscoe @william76 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4757Tpetra::DistObject: Evaluate host-pinned buffers for CUDA + MPI2019-04-29T18:30:27ZJames WillenbringTpetra::DistObject: Evaluate host-pinned buffers for CUDA + MPI*Created by: mhoemmen*
@trilinos/tpetra
`Tpetra::DistObject` currently offers two options for CUDA + MPI:
1. Give CudaSpace buffers to MPI
2. Give HostSpace buffers to MPI
(DistObject lets subclasses pack and unpack where...*Created by: mhoemmen*
@trilinos/tpetra
`Tpetra::DistObject` currently offers two options for CUDA + MPI:
1. Give CudaSpace buffers to MPI
2. Give HostSpace buffers to MPI
(DistObject lets subclasses pack and unpack wherever they like, as long as they update the sync state correctly.)
We want to evaluate a third option: Give `CudaHostPinnedSpace` buffers to MPI. Host-pinned memory behaves like CudaUVMSpace, in the sense that both host and device can access it. However, MPI sees host-pinned memory as host memory. This means two things:
1. MPI need not be CUDA aware in order to access host-pinned memory, yet we can still pack and unpack on device.
2. We don't have to worry about CUDA-aware MPI being slow.
The latter is important, since we've observed some "CUDA-aware MPI" implementations being slow in practice.
Host-pinned memory has a high allocation cost. It may make sense to start with the static View allocation functions in PR #4734. We can't use those without further work, because each DistObject instance will need its own pack and unpack buffers. This may call for a simple memory pool, and for changes to DistObject and/or subclasses so that they only hold buffer allocations while communication is active.
## Definition of Done
- [x] Write a Tpetra test that prototypes use of CudaHostPinnedSpace for communication.
- [ ] Write a benchmark to compare performance of point-to-point communication with CudaHostPinnedSpace vs. CudaSpace communication buffers.
- [ ] If CudaHostPinnedSpace pays off, change DistObject to use it for communication buffers.
- [ ] Evaluate performance with a Tpetra benchmark.
- [ ] If performance is good, deploy solution in Tpetra.
## Related Issues
* Is blocked by #4734 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/5031Framework: PR Reproduction Instructions Failing2019-04-29T17:24:35ZJames WillenbringFramework: PR Reproduction Instructions Failing*Created by: csiefer2*
Finished configuring Trilinos!
-- Configuring incomplete, errors occurred!
With no errors showing up in the cmake output (joy!).
GCC 4.9.3, as it turns out.*Created by: csiefer2*
Finished configuring Trilinos!
-- Configuring incomplete, errors occurred!
With no errors showing up in the cmake output (joy!).
GCC 4.9.3, as it turns out.