Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2019-06-08T15:27:26Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5065Tpetra: Deprecated DynamicProfile used without deprecation guards2019-06-08T15:27:26ZJames WillenbringTpetra: Deprecated DynamicProfile used without deprecation guards*Created by: kddevin*
## Bug Report
@trilinos/tpetra
### Description
DynamicProfile is a deprecated feature.
It is used in Tpetra_CrsMatrix_decl.hpp without deprecation guards.
https://github.com/trilinos/Trilinos/blob/ec7ddad27...*Created by: kddevin*
## Bug Report
@trilinos/tpetra
### Description
DynamicProfile is a deprecated feature.
It is used in Tpetra_CrsMatrix_decl.hpp without deprecation guards.
https://github.com/trilinos/Trilinos/blob/ec7ddad27cbfca383fb4c70f8beeba6193739853/packages/tpetra/core/src/Tpetra_CrsMatrix_decl.hpp#L5039
Thus, building with TPETRA_ENABLE_DEPRECATED_CODE=OFF fails.
Related to #4701
### Steps to Reproduce
Build Tpetra with TPETRA_ENABLE_DEPRECATED_CODE=OFF.
I am building with many other packages; I don't know if they are needed to see the error.
```
-D Tpetra_ENABLE_DEPRECATED_CODE:BOOL=OFF \
-D Teuchos_HIDE_DEPRECATED_CODE:BOOL=ON \
\
-D Trilinos_ENABLE_Stokhos:BOOL=ON \
-D Trilinos_ENABLE_Nox:BOOL=ON \
-D Trilinos_ENABLE_ROL:BOOL=ON \
-D ROL_ENABLE_EXAMPLES:BOOL=OFF \
-D Trilinos_ENABLE_MiniTensor:BOOL=OFF \
-D ROL_ENABLE_MiniTensor:BOOL=OFF \
-D Trilinos_ENABLE_Panzer:BOOL=ON \
-D Trilinos_ENABLE_PanzerAdaptersSTK:BOOL=OFF \
-D Trilinos_ENABLE_PanzerAdaptersIOSS:BOOL=OFF \
-D Trilinos_ENABLE_Thyra:BOOL=ON \
-D Trilinos_ENABLE_MueLu:BOOL=ON \
-D Trilinos_ENABLE_Anasazi:BOOL=ON \
-D Trilinos_ENABLE_Belos:BOOL=ON \
-D Trilinos_ENABLE_TrilinosCouplings:BOOL=ON \
-D Trilinos_ENABLE_STKIO:BOOL=OFF \
-D Trilinos_ENABLE_STKUtil:BOOL=OFF \
```
Tpetra: Deprecate DynamicProfilehttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5027Framework: Installation Testing2019-05-06T16:59:22ZJames WillenbringFramework: Installation Testing*Created by: william76*
## Enhancement
@trilinos/framework
@bartlettroscoe
@jwillenbring
As part of the roadmap towards implementing a backwards compatibility test to the Trilinos testing suite we need to first get Installation...*Created by: william76*
## Enhancement
@trilinos/framework
@bartlettroscoe
@jwillenbring
As part of the roadmap towards implementing a backwards compatibility test to the Trilinos testing suite we need to first get Installation Testing up and running.
This will require some updates to [TriBiTS][1] to support some of the installation testing capabilities that we will need. @bartlettroscoe can fill in more details on what we're needing to do there.
[1]: https://tribits.org/Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5036Zoltan2: Tests failing on ATDM cuda 10 build2019-05-02T17:33:07ZJames WillenbringZoltan2: Tests failing on ATDM cuda 10 build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/zoltan2, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this q...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/zoltan2, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Zoltan2&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=2019-04-29T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-30T00%3A00%3A00) the tests:
* Zoltan2_directoryTest_Kokkos_MPI_4
* Zoltan2_directoryTest_KokkosSimple_MPI_4
* Zoltan2_directoryTest_findUniqueGids.cpp_MPI_4
are failing in the build:
* Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
## Current Status on CDash
[Failing zoltan2 tests for the current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Zoltan2&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=today&field6=buildstarttime&compare6=83&value6=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Zoltan2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5035Teko: Tests failing on ATDM cuda 10 build2019-05-02T17:17:38ZJames WillenbringTeko: Tests failing on ATDM cuda 10 build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this qu...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Teko_&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=2019-04-29T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-30T00%3A00%3A00) the tests:
* Teko_testdriver_MPI_1
* Teko_testdriver_MPI_4
* Teko_testdriver_tpetra_MPI_1
* Teko_testdriver_tpetra_MPI_4 |
are failing in the build:
* Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
## Current Status on CDash
[Failing Teko tests on this build for current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Teko_&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=today&field6=buildstarttime&compare6=83&value6=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5033MueLu: Tests failing on ATDM cuda 10 build2019-05-02T19:11:33ZJames WillenbringMueLu: Tests failing on ATDM cuda 10 build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this q...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=MueLu_&field3=site&compare3=61&value3=white&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=84&value5=2019-04-29T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-30T00%3A00%3A00) the tests:
* MueLu_Maxwell3D-Epetra_MPI_4
* MueLu_ImportPerformance_Epetra_MPI_4
* MueLu_ImportPerformance_Tpetra_MPI_4
are failing in the build:
* Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
this is common in the output:
```
MueLu_ImportPerformance.exe: sys/memtype_cache.c:90: ucs_memtype_cache_delete: Assertion `pgt_region != ((void *)0)' failed.
```
## Current Status on CDash
[Current failing MueLu tests on this build](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=MueLu_&field3=site&compare3=61&value3=white&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=84&value5=today&field6=buildstarttime&compare6=83&value6=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5014KokkosKernels: KokkosKernels_sparse_ tests timing out on ATDM complex build2019-04-25T18:53:43ZJames WillenbringKokkosKernels: KokkosKernels_sparse_ tests timing out on ATDM complex build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/kokkoskernels, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/kokkoskernels, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug&field2=testname&compare2=65&value2=KokkosKernels_sparse_&field3=buildstarttime&compare3=84&value3=2019-04-25T00%3A00%3A00&field4=buildstarttime&compare4=83&value4=2019-03-26T00%3A00%3A00) the tests:
* KokkosKernels_sparse_openmp_MPI_1
* KokkosKernels_sparse_serial_MPI_1
are failing timing out in the build:
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug
## Current Status on CDash
[The status of these tests for the current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug&field2=testname&compare2=65&value2=KokkosKernels_sparse_&field3=buildstarttime&compare3=84&value3=today&field4=buildstarttime&compare4=83&value4=2%20days%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_KokkosKernels=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5006Ifpack2: Ifpack2_MTSGS_belos_MPI_1 randomly failing in ATDM cee rhel6 intel ...2019-04-24T23:12:57ZJames WillenbringIfpack2: Ifpack2_MTSGS_belos_MPI_1 randomly failing in ATDM cee rhel6 intel build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [thi...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=testname&compare2=61&value2=Ifpack2_MTSGS_belos_MPI_1&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=84&value4=2019-04-24T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-25T00%3A00%3A00) the test:
* Ifpack2_MTSGS_belos_MPI_1
is failing in the build:
* Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
This has failed 9 times in the last 4 weeks with something similar to:
```
Achieved tolerance: 6.44895e-10
Actual iters(20) > expected number of iterations (19), or resid-norm(0) >= 1.e-7
proc 0 total program time: 0.0258739
End Result: TEST FAILED
```
## Current Status on CDash
[Current 2 week history on CDash](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=testname&compare2=61&value2=Ifpack2_MTSGS_belos_MPI_1&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=2%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5002MueLu: MueLu_FixedMatrixPattern-Tpetra_MPI_4 randomly timing out in ATDM wat...2019-05-07T01:09:59ZJames WillenbringMueLu: MueLu_FixedMatrixPattern-Tpetra_MPI_4 randomly timing out in ATDM waterman build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this q...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-debug&field2=testname&compare2=61&value2=MueLu_FixedMatrixPattern-Tpetra_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=2019-04-23T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-24T00%3A00%3A00) the test:
* MueLu_FixedMatrixPattern-Tpetra_MPI_4
is randomly timing out in the build:
* Trilinos-atdm-waterman-cuda-9.2-debug
This test usually passes in about 6.5 seconds but has timed out (10 minutes) 5 times in the last 30 days
## Current Status on CDash
[Current 2 week history on CDash](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-debug&field2=testname&compare2=61&value2=MueLu_FixedMatrixPattern-Tpetra_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=2%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4989Muelu: MueLu_Maxwell3D-Tpetra_MPI_4 failing on atdm complex build2019-06-08T15:27:25ZJames WillenbringMuelu: MueLu_Maxwell3D-Tpetra_MPI_4 failing on atdm complex build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected pa...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug&field2=testname&compare2=61&value2=MueLu_Maxwell3D-Tpetra_MPI_4&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=84&value4=2019-04-22T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-23T00%3A00%3A00) the test:
* MueLu_Maxwell3D-Tpetra_MPI_4
is failing in the build:
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug
## Current Status on CDash
[Test results last 5 days](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug&field2=testname&compare2=61&value2=MueLu_Maxwell3D-Tpetra_MPI_4&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=83&value4=5%20days%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4982MueLu: MueLu_Helmholtz2DParallel_MPI_4 failing on ATDM complex builds2019-06-08T15:27:25ZJames WillenbringMueLu: MueLu_Helmholtz2DParallel_MPI_4 failing on ATDM complex builds*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=-complex-&field3=testname&compare3=65&value3=MueLu_Helmholtz2DParallel_MPI_4&field4=buildstarttime&compare4=84&value4=2019-04-22T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-23T00%3A00%3A00) the test:
* MueLu_Helmholtz2DParallel_MPI_4
has been failing since 2019-01-11 in the builds:
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug
* Trilinos-atdm-sems-rhel7-gnu-7.2.0-openmp-complex-shared-release-debug
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug
* Trilinos-atdm-sems-rhel7-clang-3.9.0-openmp-complex-shared-release-debug
new commits on 2019-04-11 can be found [here](https://testing.sandia.gov/cdash/viewNotes.php?buildid=4867040#!#note2)
## Current Status on CDash
[results for the current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=complex&field3=testname&compare3=65&value3=MueLu_Helmholtz2DParallel_MPI_4&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel6-gnu-7.2.0-openmp-complex-shared-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4988STK: STKUnit_tests_stk_topology_unit_tests_MPI_1 failing in ATDM build cuda-...2019-05-01T22:37:42ZJames WillenbringSTK: STKUnit_tests_stk_topology_unit_tests_MPI_1 failing in ATDM build cuda-9.2-gnu-7.2.0-rdc-release-debug-pt on 'white' and 'ride'*Created by: fryeguy52*
## Bug Report
CC: @trilinos/stk, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packag...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/stk, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt&field2=testname&compare2=61&value2=STKUnit_tests_stk_topology_unit_tests_MPI_1&field3=site&compare3=61&value3=ride&field4=buildstarttime&compare4=84&value4=2019-04-22T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-23T00%3A00%3A00) the test:
* STKUnit_tests_stk_topology_unit_tests_MPI_1
is failing since 2019-04-09 in the build:
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
<more-details>
## Current Status on CDash
[The current status of this test](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt&field2=testname&compare2=61&value2=STKUnit_tests_stk_topology_unit_tests_MPI_1&field3=site&compare3=61&value3=ride&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_STK=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4963Intrepid2 and Panzer build errors in new cuda-10.1 builds on 'white'2020-07-22T01:04:27ZJames WillenbringIntrepid2 and Panzer build errors in new cuda-10.1 builds on 'white'*Created by: bartlettroscoe*
CC: @trilinos/intrepid2, @trilinos/panzer @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (...*Created by: bartlettroscoe*
CC: @trilinos/intrepid2, @trilinos/panzer @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could be other "ATDM Sev: XXX")>
<???: Add label "type: bug"?>
<???: Add label for affected packages (e.g. "pkg: MueLu", "pkg: Tpetra", "pkg: Kokkos", etc.)>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
<???: Add milestone "Initial cleanup of new ATDM ..." or "Keep promoted ATDM ...">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
## Next Action Status
<status-and-or-first-action>
## Description
As shown [here on CDash](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&parentid=4917892), there are many build failures in Intrepid2 and Panzer shown in the new build:
* `Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug`
that as been set up on 'white' (see [TRIL-245](https://sems-atlassian-son.sandia.gov/jira/browse/TRIL-245)).
This starts with build errors in Intrepid2 as shown [here](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?buildid=4917897) that look like:
```
/home/jenkins/white/workspace/Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug/SRC_AND_BUILD/Trilinos/packages/intrepid2/src/Discretization/Basis/Intrepid2_HCURL_TET_In_FEMDef.hpp:451:64: error: invalid conversion from ‘Intrepid2::ordinal_type* {aka int*}’ to ‘Intrepid2::ordinal_type {aka int}’ [-fpermissive]
tags[i_card+1][0] = 2; // face dof
```
The build errors in Panzer look to be these same errors coming from Intrepid2.
## Current Status on CDash
* [Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug Intrepid2 build summary last 5 days](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&date=2019-04-19&filtercount=3&showfilters=1&filtercombine=and&field1=subprojects&compare1=93&value1=Intrepid2&field2=buildname&compare2=61&value2=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field3=buildstarttime&compare3=83&value3=5%20days%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on the machine 'white' as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system 'white' are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands on 'white' to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh \
Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Intrepid2=ON \
$TRILINOS_DIR
$ make NP=16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4960Ifpack2_Relaxation_def.hpp variable template C++14 extension warning breaking...2019-04-23T23:01:23ZJames WillenbringIfpack2_Relaxation_def.hpp variable template C++14 extension warning breaking SPARC Trilinos Integration builds starting 4/18/2019*Created by: bartlettroscoe*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but coul...*Created by: bartlettroscoe*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could be other "ATDM Sev: XXX")>
<???: Add label "type: bug"?>
<???: Add label for affected packages (e.g. "pkg: MueLu", "pkg: Tpetra", "pkg: Kokkos", etc.)>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
<???: Add milestone "Initial cleanup of new ATDM ..." or "Keep promoted ATDM ...">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
## Next Action Status
<status-and-or-first-action>
## Description
The new warning elevated to an error:
```
Ifpack2_Relaxation_def.hpp:147:6: error: variable templates are a C++14 extension [-Werror,-Wc++14-extensions]
void Relaxation::updateCachedMultiVector(const Teuchos::RCP > & map, size_t numVecs) const{
^
```
is breaking the SPARC Trilinos integration builds as shown [here](http://compsim-dashboard.sandia.gov/cdash/index.php?project=SPARC&date=2019-04-18&filtercount=1&showfilters=1&field1=buildname&compare1=66&value1=-trildev) with that warning being shown [here](http://compsim-dashboard.sandia.gov/cdash/viewBuildError.php?buildid=108404)
## Current Status on CDash
* [sparc-alltpls_cee-cpu_clang-5.0.1_openmpi-1.10.2_static_opt-trildev builds over last 5 days](http://compsim-dashboard.sandia.gov/cdash/index.php?project=SPARC&date=2019-04-18&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=sparc-alltpls_cee-cpu_clang-5.0.1_openmpi-1.10.2_static_opt-trildev&field2=buildstarttime&compare2=83&value2=5%20days%20ago)
## Steps to Reproduce
I can't see this warning being generated in the [Ifack2 package build itself for the build Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?type=1&buildid=4910167) so I am not sure one can reproduce this just with Trilinos. (Does this suggest a lack of test coverage for Ifpack2?)
But if one can get on the CEE LAN and can clone the SPARC repos, then one can reproduce using the scripts described [here](https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/Building+ATDM+APPs+Against+Local+Installs+of+Trilinos#BuildingATDMAPPsAgainstLocalInstallsofTrilinos-BuildingandTestingSPARCAgainstLocalTrilinosInstallation). After getting Trilinos on to the 'develop' branch as described [here](https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/Building+ATDM+APPs+Against+Local+Installs+of+Trilinos#BuildingATDMAPPsAgainstLocalInstallsofTrilinos-BuildingagainstTrilinos'develop'usingtheATDMTrilinosconfiguration), one can reproduce this using the command:
```
$ cd sparc/
$ env \
ATDM_TRIL_SPARC_BUILDS_LIST=cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt \
ATDM_TRIL_SPARC_SKIP_NATIVE_BUILD=1 \
./sparc-tril-dev-scripts/run_builds_and_tests.sh
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4929Link problems with libmuelu breaking most ATDM Trilinos builds starting 4/17/...2020-07-22T01:04:27ZJames WillenbringLink problems with libmuelu breaking most ATDM Trilinos builds starting 4/17/2019*Created by: bartlettroscoe*
CC: @trilinos/muelu , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could ...*Created by: bartlettroscoe*
CC: @trilinos/muelu , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could be other "ATDM Sev: XXX")>
<???: Add label "type: bug"?>
<???: Add label for affected packages (e.g. "pkg: MueLu", "pkg: Tpetra", "pkg: Kokkos", etc.)>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
<???: Add milestone "Initial cleanup of new ATDM ..." or "Keep promoted ATDM ...">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&date=2019-04-17&filtercount=1&showfilters=1&field1=buildname&compare1=65&value1=Trilinos-atdm-) there are link errors related to the muelu library. For example, as shown [here](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?buildid=4904861) it shows link errors like:
```
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu7LWGraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu7LWGraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::LWGraph<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu5GraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu5GraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::Graph<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu7LWGraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu7LWGraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::LWGraph<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu5GraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu5GraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::Graph<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
collect2: error: ld returned 1 exit status
```
## Steps to Reproduce
One should be able to reproduce this failure on many of the systems as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4904Piro compilation error due to missing Optipack_Config.h file2020-07-22T01:04:27ZJames WillenbringPiro compilation error due to missing Optipack_Config.h file*Created by: ikalash*
Trilinos failed to compile in the Albany nightlies due to the following error in Piro:
```
repos/Trilinos/packages/piro/src/Piro_PerformAnalysis.hpp:53:10: fatal error: OptiPack_Config.h: No such file or direct...*Created by: ikalash*
Trilinos failed to compile in the Albany nightlies due to the following error in Piro:
```
repos/Trilinos/packages/piro/src/Piro_PerformAnalysis.hpp:53:10: fatal error: OptiPack_Config.h: No such file or directory
```
Did OptiPack_Config.h get removed? Is it still needed? Here is the more detailed error message:
http://cdash.sandia.gov/CDash-2-3-0/viewBuildError.php?buildid=83699
@trilinos/piro
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4891Unused var isInitializedLocalMap_ warning in Xpetra_EpetraMap.hpp triggering ...2020-07-22T01:04:27ZJames WillenbringUnused var isInitializedLocalMap_ warning in Xpetra_EpetraMap.hpp triggering build error in SPARC Trilinos Integration builds with -Werror*Created by: bartlettroscoe*
CC: @trilinos/xpetra , @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could b...*Created by: bartlettroscoe*
CC: @trilinos/xpetra , @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could be other "ATDM Sev: XXX")>
<???: Add label "type: bug"?>
<???: Add label for affected packages (e.g. "pkg: MueLu", "pkg: Tpetra", "pkg: Kokkos", etc.)>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
<???: Add milestone "Initial cleanup of new ATDM ..." or "Keep promoted ATDM ...">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
## Next Action Status
<status-and-or-first-action>
## Description
The new unused var warning shown, for example, [here](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?type=1&buildid=4878569) in the build:
* `Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt`
in the build of MueLu which shows the warning:
```
[CTest: warning matched] /scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt/SRC_AND_BUILD/Trilinos/packages/xpetra/src/Map/Xpetra_EpetraMap.hpp:699:18: warning: private field 'isInitializedLocalMap_' is not used [-Wunused-private-field]
mutable bool isInitializedLocalMap_ = false; // It's OK to use C++11 when Tpetra is enabled
```
is triggering build errors for the `clang-5.0.1` and `gnu-7.2.0` SPARC Trilinos Integration builds that have `-Werror` set. One can see these build errors, for example, [here](http://compsim-dashboard.sandia.gov/cdash/viewBuildError.php?buildid=107482).
## Current Status on CDash
The status of these warnings can be seen in:
* [MueLu warning in Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt build over last 5 days](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&date=2019-04-13&filtercount=3&showfilters=1&filtercombine=and&field1=subprojects&compare1=93&value1=MueLu&field2=buildname&compare2=61&value2=Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt&field3=buildstarttime&compare3=83&value3=5%20days%20ago).
Just click on the number (e.g. '5') under "Build | Warn" to see the warnings.
## Steps to Reproduce
One should be able to reproduce this failure on any CEE LAN RHEL6 machine as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system <supported-atdm-system> are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh \
Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
```
and one should see the warnings.
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4847KokkosKernels_graph tests timing out in ATDM builds2019-05-01T19:20:49ZJames WillenbringKokkosKernels_graph tests timing out in ATDM builds*Created by: fryeguy52*
CC: @trilinos/kokkoskernels, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e...*Created by: fryeguy52*
CC: @trilinos/kokkoskernels, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=65&value2=KokkosKernels_graph_&field3=testname&compare3=66&value3=_MPI_1&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=83&value5=2019-04-07&field6=buildstarttime&compare6=84&value6=2019-04-10) the tests:
* KokkosKernels_graph_openmp_MPI_1
* KokkosKernels_graph_serial_MPI_1
started timing out on 2019-04-07 in the builds:
* Trilinos-atdm-hansen-shiller-gnu-debug-serial
* Trilinos-atdm-waterman-cuda-9.2-debug
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug
* Trilinos-atdm-white-ride-gnu-7.2.0-openmp-debug
## Current Status on CDash
Failures for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=65&value2=KokkosKernels_graph_&field3=testname&compare3=66&value3=_MPI_1&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=83&value5=yesterday&field6=buildstarttime&compare6=84&value6=today)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-gnu-7.2.0-openmp-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_KokkosKernels=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4801Many PanzerAdaptersSTK (random?) failures in ATDM sems-rhel7+cuda+complex builds2019-04-20T20:12:51ZJames WillenbringMany PanzerAdaptersSTK (random?) failures in ATDM sems-rhel7+cuda+complex builds*Created by: fryeguy52*
CC: @trilinos/panzer, @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "...*Created by: fryeguy52*
CC: @trilinos/panzer, @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-&field2=testname&compare2=65&value2=PanzerAdaptersSTK_&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=84&value4=2019-04-03T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-21T00%3A00%3A00&field6=status&compare6=61&value6=Failed) these tests:
* PanzerAdaptersSTK_CurlLaplacianExample
* PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1
* PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2
* PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3
* PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4
* PanzerAdaptersSTK_CurlLaplacianMultiblockExample-ConvTest-Quad-Order-1
* PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1
* PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2
* PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3
* PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1
* PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2
* PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1
* PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-1
* PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-2
* PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-3
* PanzerAdaptersSTK_MixedPoissonMultiblockExample-ConvTest-Hex-Order-1
* PanzerAdaptersSTK_PoissonInterfaceExample_2d_diffsideids_MPI_1
have failed at least once on one of these builds since 2019-03-21:
* Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
* Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug
## Current Status on CDash
PanzerAdaptersSTK tests that failed on those builds for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-&field2=testname&compare2=65&value2=PanzerAdaptersSTK_&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=yesterday&field6=status&compare6=61&value6=Failed)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Panzer=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4790TpetraCore tests failing in ATDM sems-rhel7+cuda+complex build2019-04-20T20:21:19ZJames WillenbringTpetraCore tests failing in ATDM sems-rhel7+cuda+complex build*Created by: fryeguy52*
CC: @trilinos/tpetra, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "Mu...*Created by: fryeguy52*
CC: @trilinos/tpetra, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug&field2=testname&compare2=65&value2=TpetraCore&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=84&value4=2019-04-02T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-25T00%3A00%3A00&field6=status&compare6=61&value6=Failed) the build:
* Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
has several failing TpetraCore tests. The following are failing everyday ([CDash](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug&field2=testname&compare2=65&value2=TpetraCore_gemm_m_eq_&field3=testname&compare3=66&value3=_MPI_1&field4=site&compare4=61&value4=sems-rhel7&field5=buildstarttime&compare5=84&value5=2019-04-02T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-03T00%3A00%3A00))
* TpetraCore_gemm_m_eq_1_MPI_1
* TpetraCore_gemm_m_eq_1_MPI_2
* TpetraCore_gemm_m_eq_1_MPI_5
* TpetraCore_gemm_m_eq_1_MPI_13
While these have failed randomly over the last couple weeks ([CDash](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=7&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug&field2=testname&compare2=65&value2=TpetraCore&field3=testname&compare3=64&value3=_gemm_m_eq_&field4=site&compare4=61&value4=sems-rhel7&field5=buildstarttime&compare5=84&value5=2019-04-02T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-19T00%3A00%3A00&field7=status&compare7=61&value7=Failed)):
* TpetraCore_MultiVector_MicroBenchmark_MPI_1
* TpetraCore_Map_Bug5822_2_MPI_2
* TpetraCore_getEntryOnHost_MPI_1
* TpetraCore_gemv_MPI_1
* TpetraCore_deep_copy_MultiVector_to_SerialDenseMatrix_MPI_1
* TpetraCore_createMirrorView_MPI_1
several more tests have failed over the last 2 weeks in similar complex shared build shown [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug&field2=testname&compare2=65&value2=TpetraCore&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=2%20weeks%20ago&field6=status&compare6=61&value6=failed):
* Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug
Those tests are:
* TpetraCore_MatrixMatrix_UnitTests_MPI_4
* TpetraCore_Issue_607_MPI_4
* TpetraCore_Issue_114_MPI_4
* TpetraCore_Issue601_MPI_4
* TpetraCore_ImportExport2_UnitTests_MPI_4
* TpetraCore_ImportBug5430_MPI_4
* TpetraCore_Import_Union_MPI_4
* TpetraCore_CrsMatrix_NonlocalSumInto_Ignore_MPI_4
* TpetraCore_CrsMatrix_gaussSeidel_MPI_4
* TpetraCore_CrsGraph_getNumDiags_MPI_4
* TpetraCore_AddProfiling_UnitTests_MPI_4
<more-details>
## Current Status on CDash
[Failed TpetraCore tests for the current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug&field2=testname&compare2=65&value2=TpetraCore&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=yesterday&field6=status&compare6=61&value6=failed)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Tpetra=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4779Teko_testdriver_tpetra_MPI_1 randomly failing in ATDM waterman build2019-04-01T16:37:24ZJames WillenbringTeko_testdriver_tpetra_MPI_1 randomly failing in ATDM waterman build*Created by: fryeguy52*
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https:/...*Created by: fryeguy52*
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-release-debug&field2=testname&compare2=61&value2=Teko_testdriver_tpetra_MPI_1&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=2019-04-01&field5=buildstarttime&compare5=83&value5=2019-02-28) the test:
* Teko_testdriver_tpetra_MPI_1
looks to be randomly failing in the build:
* Trilinos-atdm-waterman-cuda-9.2-release-debug
It has failed 5 times in the last month each time with:
```
terminate called after throwing an instance of 'std::runtime_error'
what(): cudaGetLastError() error( cudaErrorAssert): device-side assert triggered /home/jenkins/waterman/workspace/Trilinos-atdm-waterman-cuda-9.2-release-debug/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaExec.hpp:401
Traceback functionality not available
```
full output from a failed run can be found [here](https://testing.sandia.gov/cdash/testDetails.php?test=72920917&build=4811399)
## Current Status on CDash
current 4 week history can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-release-debug&field2=testname&compare2=61&value2=Teko_testdriver_tpetra_MPI_1&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=4%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos clean