Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2019-05-02T17:17:38Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5035Teko: Tests failing on ATDM cuda 10 build2019-05-02T17:17:38ZJames WillenbringTeko: Tests failing on ATDM cuda 10 build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this qu...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Teko_&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=2019-04-29T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-30T00%3A00%3A00) the tests:
* Teko_testdriver_MPI_1
* Teko_testdriver_MPI_4
* Teko_testdriver_tpetra_MPI_1
* Teko_testdriver_tpetra_MPI_4 |
are failing in the build:
* Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
## Current Status on CDash
[Failing Teko tests on this build for current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=Teko_&field3=status&compare3=61&value3=Failed&field4=site&compare4=61&value4=white&field5=buildstarttime&compare5=84&value5=today&field6=buildstarttime&compare6=83&value6=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5033MueLu: Tests failing on ATDM cuda 10 build2019-05-02T19:11:33ZJames WillenbringMueLu: Tests failing on ATDM cuda 10 build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this q...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=MueLu_&field3=site&compare3=61&value3=white&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=84&value5=2019-04-29T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2019-03-30T00%3A00%3A00) the tests:
* MueLu_Maxwell3D-Epetra_MPI_4
* MueLu_ImportPerformance_Epetra_MPI_4
* MueLu_ImportPerformance_Tpetra_MPI_4
are failing in the build:
* Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
this is common in the output:
```
MueLu_ImportPerformance.exe: sys/memtype_cache.c:90: ucs_memtype_cache_delete: Assertion `pgt_region != ((void *)0)' failed.
```
## Current Status on CDash
[Current failing MueLu tests on this build](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug&field2=testname&compare2=65&value2=MueLu_&field3=site&compare3=61&value3=white&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=84&value5=today&field6=buildstarttime&compare6=83&value6=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-10.1-gnu-7.2.0-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5006Ifpack2: Ifpack2_MTSGS_belos_MPI_1 randomly failing in ATDM cee rhel6 intel ...2019-04-24T23:12:57ZJames WillenbringIfpack2: Ifpack2_MTSGS_belos_MPI_1 randomly failing in ATDM cee rhel6 intel build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [thi...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=testname&compare2=61&value2=Ifpack2_MTSGS_belos_MPI_1&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=84&value4=2019-04-24T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-25T00%3A00%3A00) the test:
* Ifpack2_MTSGS_belos_MPI_1
is failing in the build:
* Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
This has failed 9 times in the last 4 weeks with something similar to:
```
Achieved tolerance: 6.44895e-10
Actual iters(20) > expected number of iterations (19), or resid-norm(0) >= 1.e-7
proc 0 total program time: 0.0258739
End Result: TEST FAILED
```
## Current Status on CDash
[Current 2 week history on CDash](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=testname&compare2=61&value2=Ifpack2_MTSGS_belos_MPI_1&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=2%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/5002MueLu: MueLu_FixedMatrixPattern-Tpetra_MPI_4 randomly timing out in ATDM wat...2019-05-07T01:09:59ZJames WillenbringMueLu: MueLu_FixedMatrixPattern-Tpetra_MPI_4 randomly timing out in ATDM waterman build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this q...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-debug&field2=testname&compare2=61&value2=MueLu_FixedMatrixPattern-Tpetra_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=2019-04-23T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-24T00%3A00%3A00) the test:
* MueLu_FixedMatrixPattern-Tpetra_MPI_4
is randomly timing out in the build:
* Trilinos-atdm-waterman-cuda-9.2-debug
This test usually passes in about 6.5 seconds but has timed out (10 minutes) 5 times in the last 30 days
## Current Status on CDash
[Current 2 week history on CDash](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-debug&field2=testname&compare2=61&value2=MueLu_FixedMatrixPattern-Tpetra_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=2%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4989Muelu: MueLu_Maxwell3D-Tpetra_MPI_4 failing on atdm complex build2019-06-08T15:27:25ZJames WillenbringMuelu: MueLu_Maxwell3D-Tpetra_MPI_4 failing on atdm complex build*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected pa...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug&field2=testname&compare2=61&value2=MueLu_Maxwell3D-Tpetra_MPI_4&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=84&value4=2019-04-22T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-23T00%3A00%3A00) the test:
* MueLu_Maxwell3D-Tpetra_MPI_4
is failing in the build:
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug
## Current Status on CDash
[Test results last 5 days](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug&field2=testname&compare2=61&value2=MueLu_Maxwell3D-Tpetra_MPI_4&field3=site&compare3=61&value3=sems-rhel7&field4=buildstarttime&compare4=83&value4=5%20days%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4982MueLu: MueLu_Helmholtz2DParallel_MPI_4 failing on ATDM complex builds2019-06-08T15:27:25ZJames WillenbringMueLu: MueLu_Helmholtz2DParallel_MPI_4 failing on ATDM complex builds*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=-complex-&field3=testname&compare3=65&value3=MueLu_Helmholtz2DParallel_MPI_4&field4=buildstarttime&compare4=84&value4=2019-04-22T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-23T00%3A00%3A00) the test:
* MueLu_Helmholtz2DParallel_MPI_4
has been failing since 2019-01-11 in the builds:
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug
* Trilinos-atdm-sems-rhel7-gnu-7.2.0-openmp-complex-shared-release-debug
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug
* Trilinos-atdm-sems-rhel7-clang-3.9.0-openmp-complex-shared-release-debug
new commits on 2019-04-11 can be found [here](https://testing.sandia.gov/cdash/viewNotes.php?buildid=4867040#!#note2)
## Current Status on CDash
[results for the current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=complex&field3=testname&compare3=65&value3=MueLu_Helmholtz2DParallel_MPI_4&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel6-gnu-7.2.0-openmp-complex-shared-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4960Ifpack2_Relaxation_def.hpp variable template C++14 extension warning breaking...2019-04-23T23:01:23ZJames WillenbringIfpack2_Relaxation_def.hpp variable template C++14 extension warning breaking SPARC Trilinos Integration builds starting 4/18/2019*Created by: bartlettroscoe*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but coul...*Created by: bartlettroscoe*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could be other "ATDM Sev: XXX")>
<???: Add label "type: bug"?>
<???: Add label for affected packages (e.g. "pkg: MueLu", "pkg: Tpetra", "pkg: Kokkos", etc.)>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
<???: Add milestone "Initial cleanup of new ATDM ..." or "Keep promoted ATDM ...">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
## Next Action Status
<status-and-or-first-action>
## Description
The new warning elevated to an error:
```
Ifpack2_Relaxation_def.hpp:147:6: error: variable templates are a C++14 extension [-Werror,-Wc++14-extensions]
void Relaxation::updateCachedMultiVector(const Teuchos::RCP > & map, size_t numVecs) const{
^
```
is breaking the SPARC Trilinos integration builds as shown [here](http://compsim-dashboard.sandia.gov/cdash/index.php?project=SPARC&date=2019-04-18&filtercount=1&showfilters=1&field1=buildname&compare1=66&value1=-trildev) with that warning being shown [here](http://compsim-dashboard.sandia.gov/cdash/viewBuildError.php?buildid=108404)
## Current Status on CDash
* [sparc-alltpls_cee-cpu_clang-5.0.1_openmpi-1.10.2_static_opt-trildev builds over last 5 days](http://compsim-dashboard.sandia.gov/cdash/index.php?project=SPARC&date=2019-04-18&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=sparc-alltpls_cee-cpu_clang-5.0.1_openmpi-1.10.2_static_opt-trildev&field2=buildstarttime&compare2=83&value2=5%20days%20ago)
## Steps to Reproduce
I can't see this warning being generated in the [Ifack2 package build itself for the build Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?type=1&buildid=4910167) so I am not sure one can reproduce this just with Trilinos. (Does this suggest a lack of test coverage for Ifpack2?)
But if one can get on the CEE LAN and can clone the SPARC repos, then one can reproduce using the scripts described [here](https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/Building+ATDM+APPs+Against+Local+Installs+of+Trilinos#BuildingATDMAPPsAgainstLocalInstallsofTrilinos-BuildingandTestingSPARCAgainstLocalTrilinosInstallation). After getting Trilinos on to the 'develop' branch as described [here](https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/Building+ATDM+APPs+Against+Local+Installs+of+Trilinos#BuildingATDMAPPsAgainstLocalInstallsofTrilinos-BuildingagainstTrilinos'develop'usingtheATDMTrilinosconfiguration), one can reproduce this using the command:
```
$ cd sparc/
$ env \
ATDM_TRIL_SPARC_BUILDS_LIST=cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt \
ATDM_TRIL_SPARC_SKIP_NATIVE_BUILD=1 \
./sparc-tril-dev-scripts/run_builds_and_tests.sh
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4929Link problems with libmuelu breaking most ATDM Trilinos builds starting 4/17/...2020-07-22T01:04:27ZJames WillenbringLink problems with libmuelu breaking most ATDM Trilinos builds starting 4/17/2019*Created by: bartlettroscoe*
CC: @trilinos/muelu , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could ...*Created by: bartlettroscoe*
CC: @trilinos/muelu , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could be other "ATDM Sev: XXX")>
<???: Add label "type: bug"?>
<???: Add label for affected packages (e.g. "pkg: MueLu", "pkg: Tpetra", "pkg: Kokkos", etc.)>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
<???: Add milestone "Initial cleanup of new ATDM ..." or "Keep promoted ATDM ...">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&date=2019-04-17&filtercount=1&showfilters=1&field1=buildname&compare1=65&value1=Trilinos-atdm-) there are link errors related to the muelu library. For example, as shown [here](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?buildid=4904861) it shows link errors like:
```
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu7LWGraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu7LWGraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::LWGraph<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu5GraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu5GraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::Graph<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu7LWGraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu7LWGraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::LWGraph<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu5GraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu5GraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::Graph<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
collect2: error: ld returned 1 exit status
```
## Steps to Reproduce
One should be able to reproduce this failure on many of the systems as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4779Teko_testdriver_tpetra_MPI_1 randomly failing in ATDM waterman build2019-04-01T16:37:24ZJames WillenbringTeko_testdriver_tpetra_MPI_1 randomly failing in ATDM waterman build*Created by: fryeguy52*
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https:/...*Created by: fryeguy52*
CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-release-debug&field2=testname&compare2=61&value2=Teko_testdriver_tpetra_MPI_1&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=2019-04-01&field5=buildstarttime&compare5=83&value5=2019-02-28) the test:
* Teko_testdriver_tpetra_MPI_1
looks to be randomly failing in the build:
* Trilinos-atdm-waterman-cuda-9.2-release-debug
It has failed 5 times in the last month each time with:
```
terminate called after throwing an instance of 'std::runtime_error'
what(): cudaGetLastError() error( cudaErrorAssert): device-side assert triggered /home/jenkins/waterman/workspace/Trilinos-atdm-waterman-cuda-9.2-release-debug/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaExec.hpp:401
Traceback functionality not available
```
full output from a failed run can be found [here](https://testing.sandia.gov/cdash/testDetails.php?test=72920917&build=4811399)
## Current Status on CDash
current 4 week history can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-release-debug&field2=testname&compare2=61&value2=Teko_testdriver_tpetra_MPI_1&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=4%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4678Stratimikos and Rythmos tests failing on many ATDM builds2019-03-26T15:04:05ZJames WillenbringStratimikos and Rythmos tests failing on many ATDM builds*Created by: fryeguy52*
CC: @trilinos/stratimikos, @srajama1 (Trilinos Linear Solvers Product Lead), @rppawlo (Trilinos Nonlinear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add lab...*Created by: fryeguy52*
CC: @trilinos/stratimikos, @srajama1 (Trilinos Linear Solvers Product Lead), @rppawlo (Trilinos Nonlinear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-20&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=status&compare2=61&value2=Failed&field3=testname&compare3=62&value3=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field4=buildstarttime&compare4=83&value4=2019-03-20&field5=buildstarttime&compare5=84&value5=2019-03-21) the tests:
* Stratimikos_test_single_stratimikos_solver_driver_belos_np_MPI_1
* Stratimikos_test_single_stratimikos_solver_driver_belos_ml_MPI_1
* Stratimikos_test_single_stratimikos_solver_driver_belos_ifpack_MPI_1
* Rythmos_timeDiscretizedBackwardEuler_amesos_MPI_1
are failing in many ATDM builds.
[new commits when these started failing](https://testing.sandia.gov/cdash/viewNotes.php?buildid=4754139#!#note4)
## Current Status on CDash
currently failing tests in ATDM builds can be seen [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-20&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=status&compare2=61&value2=Failed&field3=buildstarttime&compare3=83&value3=today&field4=buildstarttime&compare4=84&value4=tomorrow)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel6-gnu-7.2.0-openmp-release
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Stratimikos=ON \
-DTrilinos_ENABLE_Rythmos=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4646Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 timing out on waterman cuda...2019-04-21T01:44:26ZJames WillenbringIfpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 timing out on waterman cuda builds*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https:/...*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-17&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=testname&compare2=61&value2=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildname&compare4=63&value4=cuda&field5=buildstarttime&compare5=83&value5=2019-03-15&field6=buildstarttime&compare6=84&value6=today) the tests:
* Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4
are timing out in the builds:
* Trilinos-atdm-waterman-cuda-9.2-opt
* Trilinos-atdm-waterman-cuda-9.2-debug
* Trilinos-atdm-waterman-cuda-9.2-release-debug
the same test in the white cuda builds finished in about 30 seconds shown [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-17&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=testname&compare2=61&value2=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field3=site&compare3=61&value3=white&field4=buildname&compare4=63&value4=cuda&field5=buildstarttime&compare5=83&value5=2019-03-15&field6=buildstarttime&compare6=84&value6=today)
## Current Status on CDash
[Current status](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-17&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=testname&compare2=61&value2=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildname&compare4=63&value4=cuda&field5=buildstarttime&compare5=83&value5=yesterday&field6=buildstarttime&compare6=84&value6=today)
## Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4622Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 failing in ATDM cuda builds2019-03-18T16:12:43ZJames WillenbringIfpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 failing in ATDM cuda builds*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/que...*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-14&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=64&value2=-rdc-&field3=testname&compare3=61&value3=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=83&value5=2019-03-14&field6=buildstarttime&compare6=84&value6=2019-03-15) the tests:
* Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4
are failing in the builds:
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-shared-release-debug
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug
* Trilinos-atdm-waterman-cuda-9.2-release-debug
* Trilinos-atdm-waterman-cuda-9.2-rdc-shared-release-debug
* Trilinos-atdm-waterman-cuda-9.2-rdc-release-debug
* Trilinos-atdm-waterman-cuda-9.2-opt
* Trilinos-atdm-waterman-cuda-9.2-debug
* Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
* Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug
## Current Status on CDash
[Currently Status on CDash for all ATDM builds](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field3=buildstarttime&compare3=84&value3=today&field4=buildstarttime&compare4=83&value4=yesterday)
## Steps to Reproduce on waterman
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
## Steps to Reproduce on white/ride
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4599MueLu build failures in new ATDM Trilinos sems-rhel7+cuda+complex builds2019-04-10T17:41:55ZJames WillenbringMueLu build failures in new ATDM Trilinos sems-rhel7+cuda+complex builds*Created by: bartlettroscoe*
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https...*Created by: bartlettroscoe*
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&date=2019-03-11&filtercount=2&showfilters=1&filtercombine=and&field1=subprojects&compare1=93&value1=MueLu&field2=buildname&compare2=65&value2=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-), MueLu has build errors in library code in the new cuda+complex builds:
* `Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug`
* `Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug`
using the 'sems-rhel7' env.
The build errors shown [here](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?buildid=4695056) and [here](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?buildid=4695082) show errors building the source files **`ExplicitInstantiation/MueLu_TentativePFactory_kokkos.cpp`** showing errors like:
* `Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Kokkos_View.hpp(816): error: calling a constexpr __host__ function("std::real<double> ") from a __device__ function("Kokkos::Impl::ParallelFor< ::, ::Kokkos::RangePolicy<int, ::Kokkos::Cuda > , ::Kokkos::Cuda> ::operator () const") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.`
and **`ExplicitInstantiation/MueLu_TentativePFactory_kokkos.cpp`** showing errors like:
* `Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Kokkos_View.hpp(971): error: calling a constexpr __host__ function("std::complex<double> ::complex") from a __device__ function("Kokkos::Impl::ParallelFor< ::, ::Kokkos::RangePolicy<int, ::Kokkos::Cuda > , ::Kokkos::Cuda> ::operator () const") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.`
## Current Status on CDash
The current status of these builds over the last 7 days can be see in [this query](https://testing.sandia.gov/cdash/index.php?project=Trilinos&date=2019-03-11&filtercount=3&showfilters=1&filtercombine=and&field1=subprojects&compare1=93&value1=MueLu&field2=buildname&compare2=65&value2=Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-&field3=buildstarttime&compare3=83&value3=7%20days%20ago).
## Steps to Reproduce
These builds are from the CEE LAN machine 'ascicgpu14' and someone with access to the CEE LAN should be able to log onto 'ascicgpu15' and reproduce these failures in as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system `sems-rhel7' are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel7-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh \
sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ ninja -j16
```
Since some developers do not have access to the SRN CEE LAN, it is likely that these build errors can also be produce on other machines that have a CUDA build. For example, one can likely reproduce these build errors on the SON machine 'white' as described at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
using the commands:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-9.2-complex-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ ninja -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4353Ifpack2_unit_tests_MPI_4 randomly failing on ATDM waterman build2019-04-21T01:32:25ZJames WillenbringIfpack2_unit_tests_MPI_4 randomly failing on ATDM waterman build*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. ...*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-opt&field2=testname&compare2=61&value2=Ifpack2_unit_tests_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=2019-02-08T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-12-27T00%3A00%3A00) the test:
* Ifpack2_unit_tests_MPI_4
is randomly failing in the buils:
* Trilinos-atdm-waterman-cuda-9.2-opt
It has failed roughly 6 times in the last month. Here are some examples of the output when it fails:
```
Error, relErr(Y.get1dView ()[9932],Z.get1dView ()[9932]) = relErr(29832,0) = 1 <= tol = 2.22045e-12: failed!
```
```
p=0 | The following tests FAILED:
p=0 | 48. Ifpack2OverlappingRowMatrix_default_scalar_type_default_local_ordinal_type_default_global_ordinal_type_Test0_UnitTest ...
p=0 |
p=0 | Total Time: 6.49 sec
p=0 |
p=1 | Summary: total = 82, run = 82, passed = 81, failed = 1
p=1 |
p=1 | End Result: TEST FAILED
```
## Current Status on CDash
[2 Week history of this test](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-opt&field2=testname&compare2=61&value2=Ifpack2_unit_tests_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=tomorrow&field5=buildstarttime&compare5=83&value5=2%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce the build on waterman where this test is randomly failing as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4260Belos tests timing out on ATDM intel-18 mpich build2019-03-27T20:41:42ZJames WillenbringBelos tests timing out on ATDM intel-18 mpich build*Created by: fryeguy52*
CC: @trilinos/belos, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in the links below the t...*Created by: fryeguy52*
CC: @trilinos/belos, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in the links below the tests:
* [Belos_rcg_hb_MPI_4](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=buildstarttime&compare2=84&value2=2019-01-25&field3=buildstarttime&compare3=83&value3=2019-01-01&field4=testname&compare4=61&value4=Belos_rcg_hb_MPI_4)
* [Belos_gcrodr_hb_MPI_4](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=buildstarttime&compare2=84&value2=2019-01-25&field3=buildstarttime&compare3=83&value3=2019-01-01&field4=testname&compare4=61&value4=Belos_gcrodr_hb_MPI_4)
are randomly timing out in the build:
* Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
## Current Status on CDash
The current status of the Belos tests on this build for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=buildstarttime&compare2=84&value2=today&field3=buildstarttime&compare3=83&value3=yesterday&field4=testname&compare4=65&value4=Belos_)
## Steps to Reproduce
One should be able to reproduce a build where this is randomly failing on a machine with a cee rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce the build where this issue is randomly occurring should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Belos=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4219Elevate ShyLU_Node from ST to PT since it is being used by SPARC?2019-01-26T04:05:03ZJames WillenbringElevate ShyLU_Node from ST to PT since it is being used by SPARC?*Created by: bartlettroscoe*
**CC:** @trilinos/framework, @trilinos/shylu, @srajama1 (Trilinos Linear Solvers Product Area Lead)
**Blocking:** #2597
## Description
The current SPARC Trilinos configuration explicitly enables `S...*Created by: bartlettroscoe*
**CC:** @trilinos/framework, @trilinos/shylu, @srajama1 (Trilinos Linear Solvers Product Area Lead)
**Blocking:** #2597
## Description
The current SPARC Trilinos configuration explicitly enables `ShyLU_Node` subpackage `ShyLU_NodeHTS` (see [here](https://sems-atlassian-son.sandia.gov/jira/browse/TRIL-212?focusedCommentId=25503&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-25503). Therefore, ATDM Trilinos builds supporting SPARC are enabling ShyLU_Node, for example, as shown [here](https://testing.sandia.gov/cdash-dev-view/viewConfigure.php?buildid=4427483) showing:
```
...
-- Setting Trilinos_ENABLE_ShyLU_NodeHTS=ON
-- Setting Trilinos_ENABLE_ShyLU_NodeTacho=ON
-- Setting Trilinos_ENABLE_ShyLU_Node=ON
...
Final set of enabled packages: ... ShyLU_Node ... 41
Final set of enabled SE packages: ... ShyLU_NodeHTS ShyLU_NodeTacho ShyLU_Node ... 112
```
So it looks like `ShyLU_NodeTacho` may be getting implicitly enabled by accident. (We will need to see if SPARC actually using `ShyLU_NodeTacho` or not.) But `ShyLU_NodeTacho` is already declared to be `PT` (Primary Tested) but `ShyLU_NodeHTS` is currently declared to be `ST` (Secondary Tested).
In any case, since an important internal Trilinos customer (i.e SPARC) is using `ShyLU_NodeHTS`, [by definition](http://trac.trilinos.org/wiki/TribitsLifecycleModelOverview#test_categories), it needs to be elevated from Secondary Tested (ST) to Primary Tested (PT). Otherwise, `ShyLU_NodeHTS` will not get enabled in Trilinos PR builds and therefore will not protect SPARC (see #2597).
## Proposed Solution
Update the line:
```
HTS hts ST OPTIONAL
```
to be
```
HTS hts PT OPTIONAL
```
in the file
* Trilinos/packages/shylu/shylu_node/cmake/Dependencies.cmake
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/4159Belos_Tpetra_HybridGMRES_hb_test_* randomly failing in many trilinos builds2019-04-26T14:20:53ZJames WillenbringBelos_Tpetra_HybridGMRES_hb_test_* randomly failing in many trilinos builds*Created by: fryeguy52*
CC: @trilinos/belos, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
PR #4229 that may fix this was merged to 'develop' on 1/22/2019. Next: Watch for a...*Created by: fryeguy52*
CC: @trilinos/belos, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
PR #4229 that may fix this was merged to 'develop' on 1/22/2019. Next: Watch for any more random failures and if no new failures by 2/22/2019 then we can close ...
## Description
As shown in [this query](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm&field2=testname&compare2=65&value2=Belos_Tpetra_HybridGMRES_hb_test_&field3=status&compare3=62&value3=passed&field4=status&compare4=62&value4=notrun&field5=buildstarttime&compare5=83&value5=2018-12-01) the tests:
* Belos_Tpetra_HybridGMRES_hb_test_1_MPI_4
* Belos_Tpetra_HybridGMRES_hb_test_0_MPI_4
have failed 11 total times since 2018-12-01 in the following ATDM builds:
* Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt
* Trilinos-atdm-cee-rhel6-gnu-4.9.3-openmpi-1.10.2-serial-static-opt
* Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt
* Trilinos-atdm-mutrino-intel-opt-openmp-HSW
* Trilinos-atdm-sems-rhel6-gnu-debug-openmp
* Trilinos-atdm-sems-rhel6-gnu-opt-openmp
* Trilinos-atdm-sems-rhel6-gnu-opt-serial
[This query](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&filtercount=4&showfilters=1&filtercombine=and&field1=testname&compare1=65&value1=Belos_Tpetra_HybridGMRES_hb_test_&field2=status&compare2=62&value2=passed&field3=status&compare3=62&value3=notrun&field4=buildstarttime&compare4=83&value4=2018-12-01) shows that `Belos_Tpetra_HybridGMRES_hb_test_*` tests have been failing in other trilinos builds as well during that same time period.
Here is some typical output from a failure:
```
Belos Version 1.3d - 9/17/2008
Dimension of matrix: 1806
Number of right-hand sides: 1
Block size used by solver: 1
Max number of Gmres iterations: 1805
Relative residual tolerance: 1e-05
Failed.......OR Combination ->
OK...........Number of Iterations = 800 < 1805
Unconverged..(2-Norm Res Vec) / (2-Norm Prec Res0)
residual [ 0 ] = 0.0224497 > 1e-05
========================================================================================================================
TimeMonitor results over 4 processors
Timer Name MinOverProcs MeanOverProcs MaxOverProcs MeanOverCallCounts
------------------------------------------------------------------------------------------------------------------------
Belos: BlockGmresSolMgr total solve time 0.5308 (1) 0.5308 (1) 0.5308 (1) 0.5308 (1)
Belos: DGKS[2]: Ortho (Inner Product) 0.03627 (1370) 0.03643 (1370) 0.03654 (1370) 2.659e-05 (1370)
Belos: DGKS[2]: Ortho (Norm) 0.01371 (2416) 0.01547 (2416) 0.01742 (2416) 6.402e-06 (2416)
Belos: DGKS[2]: Ortho (Update) 0.02398 (1370) 0.02443 (1370) 0.02485 (1370) 1.783e-05 (1370)
Belos: DGKS[2]: Orthogonalization 0.08255 (816) 0.08426 (816) 0.08634 (816) 0.0001033 (816)
Belos: GmresPolyOp creation time 0.001283 (1) 0.001308 (1) 0.001326 (1) 0.001308 (1)
Belos: Hybrid Gmres: Operation Op*x 0.03555 (815) 0.03593 (815) 0.03648 (815) 4.408e-05 (815)
Belos: Hybrid Gmres: Operation Prec*x 0.3885 (816) 0.3908 (816) 0.3932 (816) 0.0004789 (816)
Belos: Operation Op*x 0.382 (8986) 0.3853 (8986) 0.3886 (8986) 4.288e-05 (8986)
Belos: Operation Prec*x 0 (0) 0 (0) 0 (0) 0 (0)
========================================================================================================================
---------- Actual Residuals (normalized) ----------
Problem 0 : 0.0224497
End Result: TEST FAILED
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[58035,1],2]
Exit code: 1
--------------------------------------------------------------------------
```
## Current Status on CDash
* [Current status and recent history of failures of test Belos_Tpetra_HybridGMRES_hb_test_* on CDash](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=buildname&compare2=65&value2=Trilinos-atdm-&field3=testname&compare3=65&value3=Belos_Tpetra_HybridGMRES_hb_test_&field4=status&compare4=61&value4=failed&field5=buildstarttime&compare5=83&value5=4%20weeks%20ago)
* [Recent history of test Belos_Tpetra_HybridGMRES_hb_test_1_MPI_4 in build](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-chama-intel-opt-openmp&field2=testname&compare2=61&value2=Belos_Tpetra_HybridGMRES_hb_test_1_MPI_4&field3=site&compare3=61&value3=chama&field4=buildstarttime&compare4=83&value4=4%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce a build where this random failure has a chance of occurring with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce a build where this random failure has a chance of occurring should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel6-gnu-opt-openmp
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Belos=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3994MueLu_Maxwell3D- tests not run due to build failure in ATDM build2018-12-21T02:48:28ZJames WillenbringMueLu_Maxwell3D- tests not run due to build failure in ATDM build*Created by: fryeguy52*
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
Merge of PR #3993 on 12/4/2018 resulted in passing build on [12/5/2018](https://test...*Created by: fryeguy52*
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
Merge of PR #3993 on 12/4/2018 resulted in passing build on [12/5/2018](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&parentid=4253654).
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=65&value2=MueLu_Maxwell3D-&field3=testname&compare3=66&value3=_MPI_4&field4=site&compare4=61&value4=cee-rhel6&field5=buildstarttime&compare5=84&value5=2018-12-04T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2018-11-04T00%3A00%3A00) the following tests are not being run due to a [build failure](https://testing.sandia.gov/cdash/viewBuildError.php?buildid=4245202) that started on 12/01/2018:
* MueLu_Maxwell3D-Epetra_MPI_4
* MueLu_Maxwell3D-Tpetra-Stratimikos_MPI_4
* MueLu_Maxwell3D-Tpetra_MPI_4
in the build:
* Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt
The error occurs when building `packages/muelu/test/maxwell/CMakeFiles/MueLu_Maxwell3D.dir/Maxwell3D.cpp.o`
Standard error:
```
/scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt/SRC_AND_BUILD/Trilinos/packages/muelu/test/maxwell/Maxwell3D.cpp:262:11: error: no viable overloaded '='
tm2 = Teuchos::null;
~~~ ^ ~~~~~~~~~~~~~
/scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt/SRC_AND_BUILD/Trilinos/packages/teuchos/comm/src/Teuchos_TimeMonitor.hpp:178:34: note: candidate function (the implicit copy assignment operator) not viable: no known conversion from 'Teuchos::ENull' to 'const Teuchos::TimeMonitor' for 1st argument
class TEUCHOSCOMM_LIB_DLL_EXPORT TimeMonitor :
^
/scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt/SRC_AND_BUILD/Trilinos/packages/muelu/test/maxwell/Maxwell3D.cpp:274:11: error: no viable overloaded '='
tm3 = Teuchos::null;
~~~ ^ ~~~~~~~~~~~~~
/scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt/SRC_AND_BUILD/Trilinos/packages/teuchos/comm/src/Teuchos_TimeMonitor.hpp:178:34: note: candidate function (the implicit copy assignment operator) not viable: no known conversion from 'Teuchos::ENull' to 'const Teuchos::TimeMonitor' for 1st argument
class TEUCHOSCOMM_LIB_DLL_EXPORT TimeMonitor :
^
2 errors generated.
```
## Current Status on CDash
The current status of these tests/builds for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=65&value2=MueLu_Maxwell3D-&field3=testname&compare3=66&value3=_MPI_4&field4=site&compare4=61&value4=cee-rhel6)
## Steps to Reproduce
One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3992Anasazi_Epetra_BKS_norestart_test_MPI_4 failing in seveal ATDM builds.2018-12-20T18:04:13ZJames WillenbringAnasazi_Epetra_BKS_norestart_test_MPI_4 failing in seveal ATDM builds.*Created by: fryeguy52*
CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
Triggered by the PR #3951 merged to 'develop' on 10/28/2018 that worked around Int...*Created by: fryeguy52*
CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
Triggered by the PR #3951 merged to 'develop' on 10/28/2018 that worked around Intel 18.0.2 MKL GEEV defect. Next: Try updated Intel MKL 18.0.5 on 'mutrino' (with local revert of #3951) and see all of these failures go away (@fryeguy52) ...
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=buildstarttime&compare3=83&value3=2018-11-04T00%3A00%3A00&field4=status&compare4=61&value4=Failed) the test:
* Anasazi_Epetra_BKS_norestart_test_MPI_4
is failing in the builds:
* Trilinos-atdm-mutrino-intel-opt-openmp-HSW (since ???)
* Trilinos-atdm-mutrino-intel-opt-openmp-KNL (since ???)
* Trilinos-atdm-cee-rhel6-intel-17.0.1-intelmpi-5.1.2-serial-static-opt (since 11/30/2018)
* Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt (11/29/2018 & 12/1/2018)
* Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt (on 12/2/2018)
* Trilinos-atdm-cee-rhel6-gnu-4.9.3-openmpi-1.10.2-serial-static-opt (on 12/10/2018)
<more-details>
Looks like some of these failures are random like shown for the build [Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=83&value4=2018-11-11T00%3A00%3A00) and the build [Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=84&value4=2018-12-11T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-11-11T00%3A00%3A00).
The errors look like [here](https://testing.sandia.gov/cdash/testDetails.php?test=61150478&build=4276066) for example:
```
Number of iterations performed in BlockKrylovSchur_test.exe: 30
Direct residual norms computed in BlockKrylovSchur_test.exe
Eigenvalue Residual
----------------------------------------
1.199112e+05 1.296543e-07
1.196455e+05 1.185550e-07
1.192047e+05 4.530562e-04
1.185918e+05 1.497329e-04
1.178109e+05 4.552932e-04
End Result: TEST FAILED
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[25128,1],1]
Exit code: 255
--------------------------------------------------------------------------
...
```
## Current Status on CDash
The current status of these tests/builds for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=62&value2=Trilinos-atdm-cee-rhel6-intel-18.0.2-mpich2-3.2-serial-static-opt&field3=testname&compare3=61&value3=Anasazi_Epetra_BKS_norestart_test_MPI_4&field4=buildstarttime&compare4=83&value4=1%20day%20ago&field5=status&compare5=61&value5=Failed&field6=site&compare6=62&value6=mutrino)
## Steps to Reproduce
One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6-intel-17.0.1-intelmpi-5.1.2-serial-static-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Anasazi=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3991MueLu: MueLu hangs when try to "export data" such as matrices after repartiti...2019-01-23T22:50:02ZJames WillenbringMueLu: MueLu hangs when try to "export data" such as matrices after repartitioning has occurred*Created by: pwxy*
MueLu hangs when try to "export data" such as matrices after repartitioning has occurred.
The MPI processes that have dropped out after repartitioning will throw and the run hangs:
```
p=3: *** Caught standard st...*Created by: pwxy*
MueLu hangs when try to "export data" such as matrices after repartitioning has occurred.
The MPI processes that have dropped out after repartitioning will throw and the run hangs:
```
p=3: *** Caught standard std::exception of type 'Teuchos::bad_any_cast' :
../../packages/muelu/src/Interface/../MueCentral/MueLu_VariableContainer.hpp:103:
Throw number = 17
Throw test that evaluated to true: data_->type() != typeid(T)
Error, cast to type Data<Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >> failed since the actual underlying type is 'Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >!
This is develop Trilinos cloned this morning (Dec 4, 2018), SHA1 573e3290b0500eee45e582cb8fcee0b1c6476cec
Example MueLu_Driver.exe run that exhibits this issue:
mpirun -n 4 MueLu_Driver.exe --matrixType=Laplace3D --nx=50 --ny=50 --nz=4 --mx=2 --my=2 --mz=1
[ptlin@ceerws3709 scaling]$ cat scaling.xml
<ParameterList name="MueLu">
<!--
For a generic symmetric scalar problem, these are the recommended settings for MueLu.
-->
<!-- =========== GENERAL ================ -->
<Parameter name="verbosity" type="string" value="high"/>
<Parameter name="coarse: max size" type="int" value="1000"/>
<Parameter name="multigrid algorithm" type="string" value="sa"/>
<!-- reduces setup cost for symmetric problems -->
<Parameter name="transpose: use implicit" type="bool" value="true"/>
<!-- start of default values for general options (can be omitted) -->
<Parameter name="max levels" type="int" value="10"/>
<Parameter name="number of equations" type="int" value="1"/>
<Parameter name="sa: use filtered matrix" type="bool" value="true"/>
<!-- end of default values -->
<!-- =========== AGGREGATION =========== -->
<Parameter name="aggregation: type" type="string" value="uncoupled"/>
<Parameter name="aggregation: drop scheme" type="string" value="classical"/>
<!-- Uncomment the next line to enable dropping of weak connections, which can help AMG convergence
for anisotropic problems. The exact value is problem dependent. -->
<!-- <Parameter name="aggregation: drop tol" type="double" value="0.02"/> -->
<!-- =========== SMOOTHING =========== -->
<Parameter name="smoother: type" type="string" value="CHEBYSHEV"/>
<ParameterList name="smoother: params">
<Parameter name="chebyshev: degree" type="int" value="2"/>>
<Parameter name="chebyshev: ratio eigenvalue" type="double" value="7"/>
<Parameter name="chebyshev: min eigenvalue" type="double" value="1.0"/>
<Parameter name="chebyshev: zero starting solution" type="bool" value="true"/>
</ParameterList>
<!-- =========== REPARTITIONING =========== -->
<Parameter name="repartition: enable" type="bool" value="true"/>
<Parameter name="repartition: partitioner" type="string" value="zoltan2"/>
<Parameter name="repartition: start level" type="int" value="2"/>
<Parameter name="repartition: min rows per proc" type="int" value="800"/>
<Parameter name="repartition: max imbalance" type="double" value="1.1"/>
<Parameter name="repartition: remap parts" type="bool" value="false"/>
<!-- start of default values for repartitioning (can be omitted) -->
<Parameter name="repartition: remap parts" type="bool" value="true"/>
<Parameter name="repartition: rebalance P and R" type="bool" value="false"/>
<ParameterList name="repartition: params">
<Parameter name="algorithm" type="string" value="multijagged"/>
</ParameterList>
<!-- end of default values -->
<ParameterList name="export data">
<Parameter name="A" type="string" value="{2}"/>
</ParameterList>
</ParameterList>
[ptlin@ceerws3709 scaling]$
```