Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2018-11-30T03:13:48Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2466Address failing testing test Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4 in the ...2018-11-30T03:13:48ZJames WillenbringAddress failing testing test Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4 in the debug builds on Power8 white and ride and Power9 waterman*Created by: bartlettroscoe*
**CC:** @trilinos/belos
## Next Action Status
Since test was disabled in commit a68547f, no recent signs of this test failure.
## Description
As shown at:
* https://testing.sandia.gov/cdash/q...*Created by: bartlettroscoe*
**CC:** @trilinos/belos
## Next Action Status
Since test was disabled in commit a68547f, no recent signs of this test failure.
## Description
As shown at:
* https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-27&limit=0&filtercount=1&showfilters=1&field1=testname&compare1=61&value1=Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4
the test `Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4` fails in the builds:
* `Trilinos-atdm-white-ride-cuda-debug`
* `Trilinos-atdm-white-ride-gnu-debug-openmp`
run on `white` and `ride` and passes in every other build of Trilinos, including, ironically, the `opt` builds on `white` and `ride` which otherwise show a lot of failing Belos tests as described in #2454. This failing test for the `cuda-debug` build shows a setfault:
```
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name white24 and rank 3!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name white24 and rank 1!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name white24 and rank 2!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name white24 and rank 0!
[white24:56629] *** Process received signal ***
[white24:56629] Signal: Segmentation fault (11)
[white24:56629] Signal code: Invalid permissions (2)
[white24:56629] Failing at address: 0x3fffd33fb038
...
```
and for the `gnu-debug-openmp` build shows:
```
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name white24 and rank 3!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name white24 and rank 1!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name white24 and rank 2!
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name white24 and rank 0!
[white24:56629] *** Process received signal ***
[white24:56629] Signal: Segmentation fault (11)
[white24:56629] Signal code: Invalid permissions (2)
[white24:56629] Failing at address: 0x3fffd33fb038
...
```
## Related Issues:
* Related to: #2454, #2464
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2866 Amesos2 test file amesos2/test/adapters/Tpetra_CrsMatrix_Adapter_UnitTests.c...2018-11-30T03:21:23ZJames Willenbring Amesos2 test file amesos2/test/adapters/Tpetra_CrsMatrix_Adapter_UnitTests.cpp build failure for ATDM Trilinos CUDA 9.0 builds on 'hansen/'shiller'*Created by: bartlettroscoe*
CC: @trilinos/amesos2 (Package Team), @srajama1 (Product Lead), @fryeguy52
## Next Action Status
Build error was fixed in merged PR #2876 and the build failure went away and all Amesos2 tests passed i...*Created by: bartlettroscoe*
CC: @trilinos/amesos2 (Package Team), @srajama1 (Product Lead), @fryeguy52
## Next Action Status
Build error was fixed in merged PR #2876 and the build failure went away and all Amesos2 tests passed in CUDA 9.0 builds on 6/5/2018.
## Description
As shown in the query:
* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&date=2018-05-31&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-hansen-shiller-cuda-9.0-&field2=subprojects&compare2=93&value2=Amesos2&field3=buildstarttime&compare3=84&value3=2018-06-02
the Amesos2 has had a single build failure in both of the CUDA 9.0 builds on 'hansen'/'shiller':
* `Trilinos-atdm-hansen-shiller-cuda-9.0-debug`
* `Trilinos-atdm-hansen-shiller-cuda-9.0-opt`
since this CUDA 9.0 build was first set up (see #2706).
The build failure is for the file `packages/amesos2/test/adapters/Tpetra_CrsMatrix_Adapter_UnitTests.cpp` and is shown, for example, at:
* https://testing-vm.sandia.gov/cdash/viewBuildError.php?buildid=3564003
and shows:
```
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-9.0-debug/SRC_AND_BUILD/Trilinos/packages/amesos2/test/adapters/Tpetra_CrsMatrix_Adapter_UnitTests.cpp(106): error: member "<unnamed>::test_traits<Scalar>::test_mat [with Scalar=double]" was referenced but not defined
1 error detected in the compilation of "/tmp/tmpxft_00007af7_00000000-4_Tpetra_CrsMatrix_Adapter_UnitTests.cpp4.ii".
```
This results in the "Not Run" test `Amesos2_Tpetra_CrsMatrix_Adapter_UnitTests_MPI_4` as shown, for example, at:
* https://testing-vm.sandia.gov/cdash/viewTest.php?onlynotrun&buildid=3564003
The only other failures in these CUDA 9.0 builds are failures do to the Kokos update described in #2728 which also impact the CUDA 8.0 builds. I fully expect those to go away once those issues are fixed in #2728.
## Steps to Reproduce
Following the instructions in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#shillerhansen
which is linked to from:
* https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/ATDM+Builds+of+Trilinos
one should be able to figure out how to reproduce this.
But to be specific, the exact instructions to reproduce this build failure are:
1. Log onto 'hansen' (SON) or 'shiller' (SON)
2. Clone the Trilinos repo (pointed to by `$TRILINOS_DIR` below) and get on the 'develop' branch
3. Create `<some_build_dir>` and do:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-9.0-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Amesos2=ON \
$TRILINOS_DIR
$ make NP=16
```
That should reproduce the build error.
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3992Anasazi_Epetra_BKS_norestart_test_MPI_4 failing in seveal ATDM builds.2018-12-20T18:04:13ZJames WillenbringAnasazi_Epetra_BKS_norestart_test_MPI_4 failing in seveal ATDM builds.*Created by: fryeguy52*
CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
Triggered by the PR #3951 merged to 'develop' on 10/28/2018 that worked around Int...*Created by: fryeguy52*
CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
Triggered by the PR #3951 merged to 'develop' on 10/28/2018 that worked around Intel 18.0.2 MKL GEEV defect. Next: Try updated Intel MKL 18.0.5 on 'mutrino' (with local revert of #3951) and see all of these failures go away (@fryeguy52) ...
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=buildstarttime&compare3=83&value3=2018-11-04T00%3A00%3A00&field4=status&compare4=61&value4=Failed) the test:
* Anasazi_Epetra_BKS_norestart_test_MPI_4
is failing in the builds:
* Trilinos-atdm-mutrino-intel-opt-openmp-HSW (since ???)
* Trilinos-atdm-mutrino-intel-opt-openmp-KNL (since ???)
* Trilinos-atdm-cee-rhel6-intel-17.0.1-intelmpi-5.1.2-serial-static-opt (since 11/30/2018)
* Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt (11/29/2018 & 12/1/2018)
* Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt (on 12/2/2018)
* Trilinos-atdm-cee-rhel6-gnu-4.9.3-openmpi-1.10.2-serial-static-opt (on 12/10/2018)
<more-details>
Looks like some of these failures are random like shown for the build [Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=83&value4=2018-11-11T00%3A00%3A00) and the build [Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=84&value4=2018-12-11T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-11-11T00%3A00%3A00).
The errors look like [here](https://testing.sandia.gov/cdash/testDetails.php?test=61150478&build=4276066) for example:
```
Number of iterations performed in BlockKrylovSchur_test.exe: 30
Direct residual norms computed in BlockKrylovSchur_test.exe
Eigenvalue Residual
----------------------------------------
1.199112e+05 1.296543e-07
1.196455e+05 1.185550e-07
1.192047e+05 4.530562e-04
1.185918e+05 1.497329e-04
1.178109e+05 4.552932e-04
End Result: TEST FAILED
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[25128,1],1]
Exit code: 255
--------------------------------------------------------------------------
...
```
## Current Status on CDash
The current status of these tests/builds for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=62&value2=Trilinos-atdm-cee-rhel6-intel-18.0.2-mpich2-3.2-serial-static-opt&field3=testname&compare3=61&value3=Anasazi_Epetra_BKS_norestart_test_MPI_4&field4=buildstarttime&compare4=83&value4=1%20day%20ago&field5=status&compare5=61&value5=Failed&field6=site&compare6=62&value6=mutrino)
## Steps to Reproduce
One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6-intel-17.0.1-intelmpi-5.1.2-serial-static-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Anasazi=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3989Anasazi_Epetra_GeneralizedDavidson_nh_test_MPI_4 in many ATDM builds2018-12-20T17:28:41ZJames WillenbringAnasazi_Epetra_GeneralizedDavidson_nh_test_MPI_4 in many ATDM builds*Created by: fryeguy52*
CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
The merge of PR #4031 to 'develop' on 12/13/2018 seems to have resulted in the ...*Created by: fryeguy52*
CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
The merge of PR #4031 to 'develop' on 12/13/2018 seems to have resulted in the test `Anasazi_Epetra_GeneralizedDavidson_nh_test_MPI_4` passing in all ATDM Trilinos builds. It passed in all 41 ATDM Trilinos builds on 2018-12-19 as shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&date=2018-12-19&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Anasazi_Epetra_GeneralizedDavidson_nh_test_MPI_4) (and there were no missing builds for testing day 2018-12-19 so this should be complete test results).
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Anasazi_Epetra_GeneralizedDavidson_nh_test_MPI_4&field3=buildstarttime&compare3=84&value3=2018-12-04T00%3A00%3A00&field4=buildstarttime&compare4=83&value4=2018-11-04T00%3A00%3A00&field5=status&compare5=61&value5=Failed) the test: `Anasazi_Epetra_GeneralizedDavidson_nh_test_MPI_4` is has failed in many ATDM builds since 11/24/2018 all the builds where this has failed in that time are are:
* Trilinos-atdm-sems-rhel6-intel-opt-openmp
* Trilinos-atdm-mutrino-intel-opt-openmp-KNL
* Trilinos-atdm-mutrino-intel-opt-openmp-HSW
* Trilinos-atdm-chama-intel-opt-openmp
* Trilinos-atdm-chama-intel-debug-openmp
* Trilinos-atdm-cee-rhel6-intel-17.0.1-intelmpi-5.1.2-serial-static-opt
* Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt
* Trilinos-atdm-cee-rhel6-gnu-4.9.3-openmpi-1.10.2-serial-static-opt
* Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt
The test has been failing everyday since 11/29/2018 in the builds:
* Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt
* Trilinos-atdm-cee-rhel6-gnu-4.9.3-openmpi-1.10.2-serial-static-opt
* Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt
the test output looks like this in these cases:
```
Building Map
Setting up info for filling matrix
Creating matrix
Filling matrix
Calling FillComplete on matrix
Setting Anasazi parameters
Creating initial vector for solver
Creating eigenproblem
Creating eigensolver (GeneralizedDavidsonSolMgr)
Solving eigenproblem
[ceerws1113:51638] *** An error occurred in MPI_Allreduce
[ceerws1113:51638] *** reported by process [999489537,2]
[ceerws1113:51638] *** on communicator MPI_COMM_WORLD
[ceerws1113:51638] *** MPI_ERR_IN_STATUS: error code in status
[ceerws1113:51638] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[ceerws1113:51638] *** and potentially your MPI job)
[ceerws1113:51629] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[ceerws1113:51629] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
```
## Current Status on CDash
The current status of this test on all ATDM builds can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Anasazi_Epetra_GeneralizedDavidson_nh_test_MPI_4)
History for the last week on ATDM builds can be seen [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Anasazi_Epetra_GeneralizedDavidson_nh_test_MPI_4&field3=buildstarttime&compare3=83&value3=7%20days%20ago)
## Steps to Reproduce on CEE RHEL6
One should be able to reproduce this failure on a machine with a cee rhel6 environment because it has been failing there everyday. The process is described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6-gnu-4.9.3-openmpi-1.10.2-serial-static-op
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Anasazi=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3344Anasazi_Tpetra_MVOPTester_MPI_4 failing in ATDM cuda 9 builds on waterman 2018-12-07T15:10:59ZJames WillenbringAnasazi_Tpetra_MVOPTester_MPI_4 failing in ATDM cuda 9 builds on waterman *Created by: fryeguy52*
CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe
## Next Action Status
Downgrade from OpenMPI 3.1.0 to OpenMPI 2.1.2 fixed the problem (as it fixed failing tests ...*Created by: fryeguy52*
CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe
## Next Action Status
Downgrade from OpenMPI 3.1.0 to OpenMPI 2.1.2 fixed the problem (as it fixed failing tests in other packages as well).
## Description
As shown in [this query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-08-21&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-waterman-cuda&field2=testname&compare2=61&value2=Anasazi_Tpetra_MVOPTester_MPI_4&field3=buildstarttime&compare3=84&value3=now) the test:
* Anasazi_Tpetra_MVOPTester_MPI_4
is failing in the builds:
* Trilinos-atdm-waterman-cuda-9.2-opt
* Trilinos-atdm-waterman-cuda-9.2-debug
test output
```
The following tests FAILED:
7. MultiVector_int_longlong_double_OPTestLocal_UnitTest ...
Total Time: 7.62 sec
Summary: total = 8, run = 8, passed = 7, failed = 1
End Result: TEST FAILED
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[50974,1],2]
Exit code: 1
```
## Steps to Reproduce
One should be able to reproduce this failure on the machine waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Anasazi=ON \
$TRILINOS_DIR
$ make NP=20
$ bsub -x -Is -n 20 ctest -j20
```Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/994Belos: Change GMRES default orthogonalizer from DGKS to ICGS 2-pass2019-03-22T09:43:04ZJames WillenbringBelos: Change GMRES default orthogonalizer from DGKS to ICGS 2-pass*Created by: mhoemmen*
@trilinos/belos @jjellio @hkthorn *Created by: mhoemmen*
@trilinos/belos @jjellio @hkthorn Tpetra-backloghttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3497Belos_gcrodr_hb_MPI_4 failing in ATDM builds on mutrino2018-12-12T21:22:57ZJames WillenbringBelos_gcrodr_hb_MPI_4 failing in ATDM builds on mutrino*Created by: fryeguy52*
CC: @trilinos/belos , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe
## Next Action Status
PR #3951 merged to 'develop' on 11/28/2018 resulted in this test passing in the Intel 18.0.2 ...*Created by: fryeguy52*
CC: @trilinos/belos , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe
## Next Action Status
PR #3951 merged to 'develop' on 11/28/2018 resulted in this test passing in the Intel 18.0.2 builds on 'mutrino' and the 'cee-rhel6' builds on 12/1/2018 and in all builds for several days as of 12/3/2018.
## Description
As shown in [this query](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&date=2018-09-24&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=site&compare2=61&value2=mutrino&field3=status&compare3=62&value3=passed&field4=buildstarttime&compare4=83&value4=2018-09-01&field5=testname&compare5=63&value5=Belos) the test:
* Belos_gcrodr_hb_MPI_4
is failing in the builds:
* Trilinos-atdm-mutrino-intel-opt-openmp-HSW
* Trilinos-atdm-mutrino-intel-opt-openmp-KNL
some test output:
```
*** Error in `/lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/BUILD/packages/belos/epetra/test/GCRODR/Belos_gcrodr_hb.exe': free(): invalid pointer: 0x000001000011bba0 ***
*** Error in `/lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/BUILD/packages/belos/epetra/test/GCRODR/Belos_gcrodr_hb.exe': free(): invalid pointer: 0x00000100004b4980 ***
*** Error in `/lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/BUILD/packages/belos/epetra/test/GCRODR/Belos_gcrodr_hb.exe': free(): invalid pointer: 0x00000100004b4980 ***
*** Error in `/lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/BUILD/packages/belos/epetra/test/GCRODR/Belos_gcrodr_hb.exe': free(): invalid pointer: 0x00000100004b4980 ***
```
## Steps to Reproduce
One should be able to reproduce this failure on the machine mutrino as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system mutrino are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#mutrino
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh intel-opt-openmp-HSW
$ cmake \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make -j16
$ salloc -N 1 -p standard -J $JOB_NAME ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2920Belos_pseudo_stochastic_pcg_hb_[0,1]_MPI_4 tests failing due to max iteration...2018-12-03T20:39:41ZJames WillenbringBelos_pseudo_stochastic_pcg_hb_[0,1]_MPI_4 tests failing due to max iterations limit seemingly randomly in the `Trilinos-atdm-white-ride-cuda-debug` build on 'white'*Created by: bartlettroscoe*
CC: @trilinos/belos, @fryeguy52, @srajama1 (Linear Solvers Product Lead)
## Next Action Status
Disabled in build `Trilinos-atdm-white-ride-cuda-debug` in commit cc7fff2 pushed on 6/12/2018 and showed d...*Created by: bartlettroscoe*
CC: @trilinos/belos, @fryeguy52, @srajama1 (Linear Solvers Product Lead)
## Next Action Status
Disabled in build `Trilinos-atdm-white-ride-cuda-debug` in commit cc7fff2 pushed on 6/12/2018 and showed disabled and missing on CDash on 6/13/2018. PR #3546 merged on 10/2/2018 which re-enables tests that should be fixed from PR #3050 merged before. No new failures as of 12/3/2018!
## Description
As shown in [this rather complex query showing all failing Belos tests other than Belos_rcg_hb_MPI_4 in all promoted ATDM builds since 5/10/2018](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=17&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=62&value2=Trilinos-atdm-mutrino-intel-debug-openmp&field3=buildname&compare3=62&value3=Trilinos-atdm-mutrino-intel-opt-openmp&field4=buildname&compare4=62&value4=Trilinos-atdm-white-ride-cuda-debug-pt-all-at-once&field5=site&compare5=62&value5=ride&field6=testname&compare6=62&value6=Belos_rcg_hb_MPI_4&field7=buildstarttime&compare7=84&value7=2018-06-08&field8=buildstarttime&compare8=83&value8=2018-05-10&field9=buildname&compare9=62&value9=Trilinos-atdm-white-ride-cuda-opt&field10=buildname&compare10=62&value10=Trilinos-atdm-white-ride-gnu-opt-openmp&field11=site&compare11=62&value11=serrano&field12=site&compare12=62&value12=shiller&field13=buildname&compare13=62&value13=Trilinos-atdm-white-ride-cuda-debug-all-at-once&field14=site&compare14=62&value14=chama&field15=testname&compare15=65&value15=Belos&field16=status&compare16=62&value16=passed&field17=status&compare17=62&value17=notrun) the tests:
* Belos_pseudo_stochastic_pcg_hb_0_MPI_4
* Belos_pseudo_stochastic_pcg_hb_1_MPI_4
failed 5 times in total and appear to be randomly failing in the `Trilinos-atdm-white-ride-cuda-debug` build. (The other failing test shown was `Belos_pseudo_pcg_hb_1_MPI_4` also for the `Trilinos-atdm-white-ride-cuda-debug` build but that only failed once yesterday so we will ignore that for now.) (The test `Belos_rcg_hb_MPI_4` was excluded from the above query because it is addressed in #2919.)
Looking at the testing history for these tests `Belos_pseudo_stochastic_pcg_hb_[0,1]_MPI_4` from 5/10/2018 through today 6/8/2018 in [this less complex query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=site&compare2=62&value2=ride&field3=testname&compare3=65&value3=Belos_pseudo_stochastic_pcg_hb_&field4=buildstarttime&compare4=84&value4=2018-06-08&field5=buildstarttime&compare5=83&value5=2018-05-10&field6=buildname&compare6=61&value6=Trilinos-atdm-white-ride-cuda-debug) one can see that these tests complete in about the same time in under 2 seconds when they pass or fail.
The output when these tests fail (such as shown for the test `Belos_pseudo_stochastic_pcg_hb_1_MPI_4` yesterday on 6/7/2018 [here](https://testing-vm.sandia.gov/cdash/testDetails.php?test=48082702&build=3589607)) looks like:
```
Belos::StatusTestGeneralOutput: Passed
(Num calls,Mod test,State test): (104, 1, Passed)
Passed.......OR Combination ->
Failed.......Number of Iterations = 100 == 100
Unconverged..(2-Norm Imp Res Vec) / (2-Norm Res0)
residual [ 0 ] = 8.95881e-09 < 1e-08
residual [ 1 ] = 1.21989e-08 > 1e-08
residual [ 2 ] = 6.84374e-09 < 1e-08
residual [ 3 ] = 9.15804e-09 < 1e-08
residual [ 4 ] = 7.2567e-09 < 1e-08
Passed.......OR Combination ->
Failed.......Number of Iterations = 100 == 100
Unconverged..(2-Norm Imp Res Vec) / (2-Norm Res0)
residual [ 0 ] = 8.95881e-09 < 1e-08
residual [ 1 ] = 1.21989e-08 > 1e-08
residual [ 2 ] = 6.84374e-09 < 1e-08
residual [ 3 ] = 9.15804e-09 < 1e-08
residual [ 4 ] = 7.2567e-09 < 1e-08
==================================================================================================================================
TimeMonitor results over 4 processors
Timer Name MinOverProcs MeanOverProcs MaxOverProcs MeanOverCallCounts
----------------------------------------------------------------------------------------------------------------------------------
Belos: Operation Op*x 0.06571 (101) 0.07122 (101) 0.07694 (101) 0.0007051 (101)
Belos: Operation Prec*x 0.1014 (104) 0.108 (104) 0.1151 (104) 0.001039 (104)
Belos: PseudoBlockStochasticCGSolMgr total solve time 0.2159 (1) 0.216 (1) 0.2162 (1) 0.216 (1)
Epetra_CrsMatrix::Multiply(TransA,X,Y) 0.0665 (102) 0.07206 (102) 0.07777 (102) 0.0007065 (102)
Epetra_CrsMatrix::Solve(Upper,Trans,UnitDiag,X,Y) 0.101 (210) 0.1076 (210) 0.1147 (210) 0.0005122 (210)
==================================================================================================================================
---------- Actual Residuals (normalized) ----------
Problem 0 : 8.95881e-09
Problem 1 : 1.21989e-08
Problem 2 : 6.84374e-09
Problem 3 : 9.15804e-09
Problem 4 : 7.2567e-09
End Result: TEST FAILED
```
So this shows that the test fails due to the max iteration limit of 100 being reached before reaching the desired residual tolerance . The other failures for the tests `Belos_pseudo_stochastic_pcg_hb_0_MPI_4` and `Belos_pseudo_stochastic_pcg_hb_1_MPI_4` look to all be maxing out the number of iterations at 100.
When the test `Belos_pseudo_stochastic_pcg_hb_1_MPI_4` passed the day before on 6/6/2018 as shown [here](https://testing-vm.sandia.gov/cdash/testDetails.php?test=48012272&build=3584608) showed output like:
```
Belos::StatusTestGeneralOutput: Passed
(Num calls,Mod test,State test): (89, 1, Passed)
Passed.......OR Combination ->
OK...........Number of Iterations = 87 < 100
Converged....(2-Norm Imp Res Vec) / (2-Norm Res0)
residual [ 0 ] = 5.02551e-09 < 1e-08
residual [ 1 ] = 5.92159e-09 < 1e-08
residual [ 2 ] = 6.61897e-09 < 1e-08
residual [ 3 ] = 8.2598e-09 < 1e-08
residual [ 4 ] = 3.67011e-09 < 1e-08
Passed.......OR Combination ->
OK...........Number of Iterations = 87 < 100
Converged....(2-Norm Imp Res Vec) / (2-Norm Res0)
residual [ 0 ] = 5.02551e-09 < 1e-08
residual [ 1 ] = 5.92159e-09 < 1e-08
residual [ 2 ] = 6.61897e-09 < 1e-08
residual [ 3 ] = 8.2598e-09 < 1e-08
residual [ 4 ] = 3.67011e-09 < 1e-08
=================================================================================================================================
TimeMonitor results over 4 processors
Timer Name MinOverProcs MeanOverProcs MaxOverProcs MeanOverCallCounts
---------------------------------------------------------------------------------------------------------------------------------
Belos: Operation Op*x 0.0652 (88) 0.06892 (88) 0.07251 (88) 0.0007831 (88)
Belos: Operation Prec*x 0.09675 (89) 0.1009 (89) 0.1101 (89) 0.001134 (89)
Belos: PseudoBlockStochasticCGSolMgr total solve time 0.195 (1) 0.195 (1) 0.195 (1) 0.195 (1)
Epetra_CrsMatrix::Multiply(TransA,X,Y) 0.06596 (89) 0.06969 (89) 0.07333 (89) 0.0007831 (89)
Epetra_CrsMatrix::Solve(Upper,Trans,UnitDiag,X,Y) 0.09635 (180) 0.1006 (180) 0.1098 (180) 0.0005587 (180)
=================================================================================================================================
---------- Actual Residuals (normalized) ----------
Problem 0 : 5.02551e-09
Problem 1 : 5.92159e-09
Problem 2 : 6.61897e-09
Problem 3 : 8.2598e-09
Problem 4 : 3.67011e-09
End Result: TEST PASSED
```
which shows it converged in 87 iterations. I looked at several other instances when these tests passed and they all look to be converging in 87 iterations.
Is this non-deterministic behavior due to fact that this is "stochastic" code and therefore the behavior is truly random or is it due to the fact that the random seed is not set consistently, or is it due to non-deterministic behavior in the accumulations with the CUDA 8.0 threaded Kokkos implementation on this machine? The fact that the test seems to converge in 87 iterations when it passes suggests that this is not purposeful random behavior but is a result of some other undesired and unintended random behavior.
## Steps to reproduce
Following the instructions at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
one might be able to reproduce this behavior on 'white' or 'ride' by cloning the Trilinos github repo, getting on the 'develop' branch and then doing:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Belos=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
But given that this test looks to be randomly failing, it may be hard to reproduce this behavior locally.
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3338Belos test failing on ATDM waterman builds2018-11-30T03:13:48ZJames WillenbringBelos test failing on ATDM waterman builds*Created by: fryeguy52*
CC: @trilinos/belos , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe
## Next Action Status
PR #3363 merged to 'develop' on 8/27/2018 the test `Belos_Tpetra_MVOPTester_complex_test_MPI_...*Created by: fryeguy52*
CC: @trilinos/belos , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe
## Next Action Status
PR #3363 merged to 'develop' on 8/27/2018 the test `Belos_Tpetra_MVOPTester_complex_test_MPI_4` passed on 8/28/2018 but the test `Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4` is still failing. PR #3454 merged on 9/19/2018 disabled this test in these waterman builds in commit cb9a9c9 and this test disappeared from the `trilinos-atdm-waterman-*debug*` builds after 2018-09-21 as shown [here](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&date=2018-10-15&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-waterman-&field2=buildname&compare2=63&value2=debug&field3=testname&compare3=61&value3=Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4&field4=buildstarttime&compare4=83&value4=2018-09-15). Next: Fix these?
## Description
As shown in [this query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-08-20&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-waterman-&field2=status&compare2=62&value2=passed&field3=status&compare3=62&value3=notrun) the tests:
* Belos_Tpetra_MVOPTester_complex_test_MPI_4
* Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4
are failing in some of the `Trilinos-atdm-waterman-*` builds
The test `Belos_Tpetra_MVOPTester_complex_test_MPI_4` is failing on the two cuda-9.2 builds
* Trilinos-atdm-waterman-cuda-9.2-opt
* Trilinos-atdm-waterman-cuda-9.2-debug
The test `Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4` is failing on the two debug builds
* Trilinos-atdm-waterman-gnu-debug-openmp
* Trilinos-atdm-waterman-cuda-9.2-debug
<more-details>
## Steps to Reproduce
One should be able to reproduce this failure on the machine waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Belos=ON \
$TRILINOS_DIR
$ make NP=20
$ bsub -x -Is -n 20 ctest -j20
```Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4260Belos tests timing out on ATDM intel-18 mpich build2019-03-27T20:41:42ZJames WillenbringBelos tests timing out on ATDM intel-18 mpich build*Created by: fryeguy52*
CC: @trilinos/belos, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in the links below the t...*Created by: fryeguy52*
CC: @trilinos/belos, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in the links below the tests:
* [Belos_rcg_hb_MPI_4](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=buildstarttime&compare2=84&value2=2019-01-25&field3=buildstarttime&compare3=83&value3=2019-01-01&field4=testname&compare4=61&value4=Belos_rcg_hb_MPI_4)
* [Belos_gcrodr_hb_MPI_4](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=buildstarttime&compare2=84&value2=2019-01-25&field3=buildstarttime&compare3=83&value3=2019-01-01&field4=testname&compare4=61&value4=Belos_gcrodr_hb_MPI_4)
are randomly timing out in the build:
* Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
## Current Status on CDash
The current status of the Belos tests on this build for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt&field2=buildstarttime&compare2=84&value2=today&field3=buildstarttime&compare3=83&value3=yesterday&field4=testname&compare4=65&value4=Belos_)
## Steps to Reproduce
One should be able to reproduce a build where this is randomly failing on a machine with a cee rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce the build where this issue is randomly occurring should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Belos=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2839Failing MueLu tests on mutrino in ATDM configuration2018-11-30T03:12:09ZJames WillenbringFailing MueLu tests on mutrino in ATDM configuration*Created by: fryeguy52*
## Next action Status
PR #3736 merged to 'develop' on 10/25/2018 re-enables these tests on 'mutrino'. Newly passing tests seen on 10/26/2018 in `Trilinos-atdm-mutrino-intel-opt-openmp-HSW` build shown [here]...*Created by: fryeguy52*
## Next action Status
PR #3736 merged to 'develop' on 10/25/2018 re-enables these tests on 'mutrino'. Newly passing tests seen on 10/26/2018 in `Trilinos-atdm-mutrino-intel-opt-openmp-HSW` build shown [here](https://testing.sandia.gov/cdash/viewTest.php?onlydelta&buildid=4099480).
## Description
The following tests are failing on mutrino in the ATDM configuration nightly testing. Click links for test output.
[MueLu_CreateOperatorTpetra_MPI_1](https://testing.sandia.gov/cdash/testDetails.php?test=48648533&build=3604771)
[MueLu_CreateOperatorTpetra_MPI_4](https://testing.sandia.gov/cdash/testDetails.php?test=48648552&build=3604771)
[MueLu_ParameterListInterpreterTpetra_MPI_1](https://testing.sandia.gov/cdash/testDetails.php?test=48648532&build=3604771)
[MueLu_ParameterListInterpreterTpetra_MPI_4](https://testing.sandia.gov/cdash/testDetails.php?test=48648551&build=3604771)
[MueLu_ParameterListInterpreterTpetraHeavy_MPI_1](https://testing.sandia.gov/cdash/testDetails.php?test=48648535&build=3604771)
[MueLu_ParameterListInterpreterTpetraHeavy_MPI_4](https://testing.sandia.gov/cdash/testDetails.php?test=48648550&build=3604771)
They are failing on the following builds:
[Trilinos-atdm-mutrino-intel-opt-openmp](https://testing.sandia.gov/cdash/buildSummary.php?buildid=3604771)
[Trilinos-atdm-mutrino-intel-debug-openmp](https://testing.sandia.gov/cdash/buildSummary.php?buildid=3604985)
and have been failing since the builds were set up on 5/18/2018
@trilinos/muelu
@bartlettroscoe
## Steps to Reproduce
On mutrino clone trilinos and do the following. This will automatically set up the environment the way that jenkins is running:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh intel-opt-openmp
$ cmake \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make -j16
$ salloc -N 1 -p standard ctest -j16
```
for more information on how to reproduce see:
https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.mdInitial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4622Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 failing in ATDM cuda builds2019-03-18T16:12:43ZJames WillenbringIfpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 failing in ATDM cuda builds*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/que...*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-14&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=64&value2=-rdc-&field3=testname&compare3=61&value3=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field4=status&compare4=62&value4=Passed&field5=buildstarttime&compare5=83&value5=2019-03-14&field6=buildstarttime&compare6=84&value6=2019-03-15) the tests:
* Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4
are failing in the builds:
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-shared-release-debug
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug
* Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug
* Trilinos-atdm-waterman-cuda-9.2-release-debug
* Trilinos-atdm-waterman-cuda-9.2-rdc-shared-release-debug
* Trilinos-atdm-waterman-cuda-9.2-rdc-release-debug
* Trilinos-atdm-waterman-cuda-9.2-opt
* Trilinos-atdm-waterman-cuda-9.2-debug
* Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
* Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug
## Current Status on CDash
[Currently Status on CDash for all ATDM builds](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field3=buildstarttime&compare3=84&value3=today&field4=buildstarttime&compare4=83&value4=yesterday)
## Steps to Reproduce on waterman
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
## Steps to Reproduce on white/ride
One should be able to reproduce this failure on ride or white as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for ride or white are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3290Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 test failing on new cuda 9....2018-11-30T03:21:24ZJames WillenbringIfpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 test failing on new cuda 9.2 ATDM build on white/ride*Created by: fryeguy52*
CC: @trilinos/ifpack2 , @srajama1 (Trilinos Linear Solvers Product Lead) @bartlettroscoe
## Next Action Status
PR #3549 merged on 10/2/2018 changed from hybrid OpenMPI 2.1.2/3.1.0 env to consistent GCC 7...*Created by: fryeguy52*
CC: @trilinos/ifpack2 , @srajama1 (Trilinos Linear Solvers Product Lead) @bartlettroscoe
## Next Action Status
PR #3549 merged on 10/2/2018 changed from hybrid OpenMPI 2.1.2/3.1.0 env to consistent GCC 7.2.0 + OpenMPI 2.1.2 + CUDA 9.2 env and TPLs and appears to fix this failing Ifpack2 test on 'white' **and** 'ride' and did not seem to break other tests. As of 10/9/2018 this test has not failed on 'white' or 'ride' since 10/1/2018 and there are now additional test failures.
## Description
As shown in [this query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&date=2018-08-11&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-white-ride-cuda-9.2&field2=buildname&compare2=64&value2=Trilinos-atdm-white-ride-cuda-9.2-debug-pt&field3=testname&compare3=63&value3=Ifpack2_) the tests:
* Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4
are failing in the builds:
* Trilinos-atdm-white-ride-cuda-9.2-debug
* Trilinos-atdm-white-ride-cuda-9.2-opt
from the test output:
```
[white25:104781] mca_base_component_repository_open: unable to open mca_coll_hcoll: libsharp_coll.so.2: cannot open shared object file: No such file or directory (ignored)
[white25:104783] mca_base_component_repository_open: unable to open mca_coll_hcoll: libsharp_coll.so.2: cannot open shared object file: No such file or directory (ignored)
[white25:104785] mca_base_component_repository_open: unable to open mca_coll_hcoll: libsharp_coll.so.2: cannot open shared object file: No such file or directory (ignored)
[white25:104784] mca_base_component_repository_open: unable to open mca_coll_hcoll: libsharp_coll.so.2: cannot open shared object file: No such file or directory (ignored)
<I> nranks 4 ni 10 nj 10 nk 10 bs 5 nrhs 1 isplit 4 jsplit 1
```
## Steps to Reproduce
One should be able to reproduce this failure on the machine `white` as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for the system `white` are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-9.2-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4646Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 timing out on waterman cuda...2019-04-21T01:44:26ZJames WillenbringIfpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 timing out on waterman cuda builds*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https:/...*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-17&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=testname&compare2=61&value2=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildname&compare4=63&value4=cuda&field5=buildstarttime&compare5=83&value5=2019-03-15&field6=buildstarttime&compare6=84&value6=today) the tests:
* Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4
are timing out in the builds:
* Trilinos-atdm-waterman-cuda-9.2-opt
* Trilinos-atdm-waterman-cuda-9.2-debug
* Trilinos-atdm-waterman-cuda-9.2-release-debug
the same test in the white cuda builds finished in about 30 seconds shown [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-17&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=testname&compare2=61&value2=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field3=site&compare3=61&value3=white&field4=buildname&compare4=63&value4=cuda&field5=buildstarttime&compare5=83&value5=2019-03-15&field6=buildstarttime&compare6=84&value6=today)
## Current Status on CDash
[Current status](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2019-03-17&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=testname&compare2=61&value2=Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildname&compare4=63&value4=cuda&field5=buildstarttime&compare5=83&value5=yesterday&field6=buildstarttime&compare6=84&value6=today)
## Steps to Reproduce
One should be able to reproduce this failure on waterman as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4353Ifpack2_unit_tests_MPI_4 randomly failing on ATDM waterman build2019-04-21T01:32:25ZJames WillenbringIfpack2_unit_tests_MPI_4 randomly failing on ATDM waterman build*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. ...*Created by: fryeguy52*
CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "ATDM">
<???: Add label "bug"?>
<???: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)>
<???: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-opt&field2=testname&compare2=61&value2=Ifpack2_unit_tests_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=2019-02-08T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-12-27T00%3A00%3A00) the test:
* Ifpack2_unit_tests_MPI_4
is randomly failing in the buils:
* Trilinos-atdm-waterman-cuda-9.2-opt
It has failed roughly 6 times in the last month. Here are some examples of the output when it fails:
```
Error, relErr(Y.get1dView ()[9932],Z.get1dView ()[9932]) = relErr(29832,0) = 1 <= tol = 2.22045e-12: failed!
```
```
p=0 | The following tests FAILED:
p=0 | 48. Ifpack2OverlappingRowMatrix_default_scalar_type_default_local_ordinal_type_default_global_ordinal_type_Test0_UnitTest ...
p=0 |
p=0 | Total Time: 6.49 sec
p=0 |
p=1 | Summary: total = 82, run = 82, passed = 81, failed = 1
p=1 |
p=1 | End Result: TEST FAILED
```
## Current Status on CDash
[2 Week history of this test](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-waterman-cuda-9.2-opt&field2=testname&compare2=61&value2=Ifpack2_unit_tests_MPI_4&field3=site&compare3=61&value3=waterman&field4=buildstarttime&compare4=84&value4=tomorrow&field5=buildstarttime&compare5=83&value5=2%20weeks%20ago)
## Steps to Reproduce
One should be able to reproduce the build on waterman where this test is randomly failing as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for waterman are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4929Link problems with libmuelu breaking most ATDM Trilinos builds starting 4/17/...2020-07-22T01:04:27ZJames WillenbringLink problems with libmuelu breaking most ATDM Trilinos builds starting 4/17/2019*Created by: bartlettroscoe*
CC: @trilinos/muelu , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could ...*Created by: bartlettroscoe*
CC: @trilinos/muelu , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
<Checklist>
<???: Add label "client: ATDM">
<???: Add label "ATDM Sev: Blocker" (by default but could be other "ATDM Sev: XXX")>
<???: Add label "type: bug"?>
<???: Add label for affected packages (e.g. "pkg: MueLu", "pkg: Tpetra", "pkg: Kokkos", etc.)>
<???: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
<???: Add milestone "Initial cleanup of new ATDM ..." or "Keep promoted ATDM ...">
<???: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
## Next Action Status
<status-and-or-first-action>
## Description
As shown in [this query](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&date=2019-04-17&filtercount=1&showfilters=1&field1=buildname&compare1=65&value1=Trilinos-atdm-) there are link errors related to the muelu library. For example, as shown [here](https://testing.sandia.gov/cdash-dev-view/viewBuildError.php?buildid=4904861) it shows link errors like:
```
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu7LWGraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu7LWGraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::LWGraph<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu5GraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu5GraphIixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::Graph<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu7LWGraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu7LWGraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::LWGraph<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
packages/muelu/src/libmuelu.a(MueLu_CoalesceDropFactory.cpp.o):(.rodata._ZTVN5MueLu5GraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE[_ZTVN5MueLu5GraphIiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6OpenMPENS1_9HostSpaceEEEEE]+0xa8): undefined reference to `MueLu::Graph<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, int) const'
collect2: error: ld returned 1 exit status
```
## Steps to Reproduce
One should be able to reproduce this failure on many of the systems as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2232 MueLu build failure Trilinos-atdm-hanson-shiller-intel-opt-serial build2018-11-30T03:12:08ZJames Willenbring MueLu build failure Trilinos-atdm-hanson-shiller-intel-opt-serial build*Created by: bartlettroscoe*
**CC:** @trilinos/muelu
## Description:
The test executable target `MueLu_UnitTests` is shown to have a build failure for the new `Trilinos-atdm-hanson-shiller-intel-opt-serial` build currently being ...*Created by: bartlettroscoe*
**CC:** @trilinos/muelu
## Description:
The test executable target `MueLu_UnitTests` is shown to have a build failure for the new `Trilinos-atdm-hanson-shiller-intel-opt-serial` build currently being submitted to the `Specialized` group/track shown yesterday at:
* https://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=3379989
* https://testing.sandia.gov/cdash/viewBuildError.php?buildid=3380117
This shows link failures:
```
ackages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/UnsmooshFactory.cpp.o: In function `MueLuTests::UnsmooshFactory_UnsmooshTentativeP_UnitTest<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
UnsmooshFactory.cpp:(.text._ZNK10MueLuTests43UnsmooshFactory_UnsmooshTentativeP_UnitTestIdiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb[_ZNK10MueLuTests43UnsmooshFactory_UnsmooshTentativeP_UnitTestIdiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb]+0x4ff): undefined reference to `Teuchos::RCP<Xpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > Galeri::Xpetra::CreateMap<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Xpetra::UnderlyingLib, std::string const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::ParameterList&)'
packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/UnsmooshFactory.cpp.o: In function `MueLuTests::UnsmooshFactory_UnsmooshTentativeP_UnitTest<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
UnsmooshFactory.cpp:(.text._ZNK10MueLuTests43UnsmooshFactory_UnsmooshTentativeP_UnitTestIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb[_ZNK10MueLuTests43UnsmooshFactory_UnsmooshTentativeP_UnitTestIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb]+0x501): undefined reference to `Teuchos::RCP<Xpetra::Map<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > Galeri::Xpetra::CreateMap<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Xpetra::UnderlyingLib, std::string const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::ParameterList&)'
packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/VariableDofLaplacianFactory.cpp.o: In function `MueLuTests::VariableDofLaplacianFactory_VarLaplConstructor2_UnitTest<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
VariableDofLaplacianFactory.cpp:(.text._ZNK10MueLuTests56VariableDofLaplacianFactory_VarLaplConstructor2_UnitTestIdiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb[_ZNK10MueLuTests56VariableDofLaplacianFactory_VarLaplConstructor2_UnitTestIdiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb]+0x4f6): undefined reference to `Teuchos::RCP<Xpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > Galeri::Xpetra::CreateMap<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Xpetra::UnderlyingLib, std::string const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::ParameterList&)'
packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/VariableDofLaplacianFactory.cpp.o: In function `MueLuTests::VariableDofLaplacianFactory_VarLaplPtent_UnitTest<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
VariableDofLaplacianFactory.cpp:(.text._ZNK10MueLuTests49VariableDofLaplacianFactory_VarLaplPtent_UnitTestIdiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb[_ZNK10MueLuTests49VariableDofLaplacianFactory_VarLaplPtent_UnitTestIdiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb]+0x4fc): undefined reference to `Teuchos::RCP<Xpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > Galeri::Xpetra::CreateMap<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Xpetra::UnderlyingLib, std::string const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::ParameterList&)'
packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/VariableDofLaplacianFactory.cpp.o: In function `MueLuTests::VariableDofLaplacianFactory_VarLaplConstructor2_UnitTest<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
VariableDofLaplacianFactory.cpp:(.text._ZNK10MueLuTests56VariableDofLaplacianFactory_VarLaplConstructor2_UnitTestIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb[_ZNK10MueLuTests56VariableDofLaplacianFactory_VarLaplConstructor2_UnitTestIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb]+0x4f8): undefined reference to `Teuchos::RCP<Xpetra::Map<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > Galeri::Xpetra::CreateMap<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Xpetra::UnderlyingLib, std::string const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::ParameterList&)'
packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/VariableDofLaplacianFactory.cpp.o: In function `MueLuTests::VariableDofLaplacianFactory_VarLaplPtent_UnitTest<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
VariableDofLaplacianFactory.cpp:(.text._ZNK10MueLuTests49VariableDofLaplacianFactory_VarLaplPtent_UnitTestIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb[_ZNK10MueLuTests49VariableDofLaplacianFactory_VarLaplPtent_UnitTestIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb]+0x501): undefined reference to `Teuchos::RCP<Xpetra::Map<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > Galeri::Xpetra::CreateMap<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Xpetra::UnderlyingLib, std::string const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::ParameterList&)'
packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/Repartition.cpp.o: In function `MueLuTests::Repartition_CoordinateMap_UnitTest<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
Repartition.cpp:(.text._ZNK10MueLuTests34Repartition_CoordinateMap_UnitTestIdiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb[_ZNK10MueLuTests34Repartition_CoordinateMap_UnitTestIdiiN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb]+0x61f): undefined reference to `Teuchos::RCP<Xpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > Galeri::Xpetra::CreateMap<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Xpetra::UnderlyingLib, std::string const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::ParameterList&)'
packages/muelu/test/unit_tests/CMakeFiles/MueLu_UnitTests.dir/Repartition.cpp.o: In function `MueLuTests::Repartition_CoordinateMap_UnitTest<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
Repartition.cpp:(.text._ZNK10MueLuTests34Repartition_CoordinateMap_UnitTestIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb[_ZNK10MueLuTests34Repartition_CoordinateMap_UnitTestIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb]+0x621): undefined reference to `Teuchos::RCP<Xpetra::Map<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > Galeri::Xpetra::CreateMap<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Xpetra::UnderlyingLib, std::string const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::ParameterList&)'
```
## Steps to Reproduce
Following the instructions at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
after locally Trilinos, you can reproduce on the test bed machines `hansen` or `shiller` as follows:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh intel-opt-serial
$ cmake -GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_MueLu=ON -DTrilinos_ENABLE_TESTS=ON \
$TRILINOS_DIR
$ make NP=16 MueLu_UnitTests &> make.out
```
(Where `TRILINOS_DIR` points to the locally cloned Trilinos source repo on the `develop` branch.)
That produced the same link failure shown on CDash for me on `shiller`.
Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3655MueLu: CreatePreconditioner_PDESystem hangs/fails in fused Jacobi under CUDA2018-11-30T03:12:10ZJames WillenbringMueLu: CreatePreconditioner_PDESystem hangs/fails in fused Jacobi under CUDA*Created by: jhux2*
I first saw this on waterman, but it's also happening on geminga:
```
[snip]
Eigenvalue estimate
Calculating max eigenvalue estimate now (max iters = 10)
Prolongator damping factor = 0.68 (1.33 / 1.94)
Fused ...*Created by: jhux2*
I first saw this on waterman, but it's also happening on geminga:
```
[snip]
Eigenvalue estimate
Calculating max eigenvalue estimate now (max iters = 10)
Prolongator damping factor = 0.68 (1.33 / 1.94)
Fused (I-omega*D^{-1} A)*Ptent
[hang]
```
2018-10-17 MueLu dashboard
Linux-gcc-5.3.0-OPENMPI-1.8.7_RELEASE_KOKKOS-REFACTOR_EXPERIMENTAL_CUDA-8.0.44
[MueLu_UnitTestsTpetra_MPI_1](https://testing.sandia.gov/cdash/testDetails.php?test=56901030&build=4060598)
[MueLu_UnitTestsTpetra_MPI_4](https://testing.sandia.gov/cdash/testDetails.php?test=56901000&build=4060598)
Blocks: #2674, #3482
@trilinos/muelu @csiefer2 Initial cleanup of new ATDM builds of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3994MueLu_Maxwell3D- tests not run due to build failure in ATDM build2018-12-21T02:48:28ZJames WillenbringMueLu_Maxwell3D- tests not run due to build failure in ATDM build*Created by: fryeguy52*
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
Merge of PR #3993 on 12/4/2018 resulted in passing build on [12/5/2018](https://test...*Created by: fryeguy52*
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
Merge of PR #3993 on 12/4/2018 resulted in passing build on [12/5/2018](https://testing.sandia.gov/cdash-dev-view/index.php?project=Trilinos&parentid=4253654).
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=65&value2=MueLu_Maxwell3D-&field3=testname&compare3=66&value3=_MPI_4&field4=site&compare4=61&value4=cee-rhel6&field5=buildstarttime&compare5=84&value5=2018-12-04T00%3A00%3A00&field6=buildstarttime&compare6=83&value6=2018-11-04T00%3A00%3A00) the following tests are not being run due to a [build failure](https://testing.sandia.gov/cdash/viewBuildError.php?buildid=4245202) that started on 12/01/2018:
* MueLu_Maxwell3D-Epetra_MPI_4
* MueLu_Maxwell3D-Tpetra-Stratimikos_MPI_4
* MueLu_Maxwell3D-Tpetra_MPI_4
in the build:
* Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt
The error occurs when building `packages/muelu/test/maxwell/CMakeFiles/MueLu_Maxwell3D.dir/Maxwell3D.cpp.o`
Standard error:
```
/scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt/SRC_AND_BUILD/Trilinos/packages/muelu/test/maxwell/Maxwell3D.cpp:262:11: error: no viable overloaded '='
tm2 = Teuchos::null;
~~~ ^ ~~~~~~~~~~~~~
/scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt/SRC_AND_BUILD/Trilinos/packages/teuchos/comm/src/Teuchos_TimeMonitor.hpp:178:34: note: candidate function (the implicit copy assignment operator) not viable: no known conversion from 'Teuchos::ENull' to 'const Teuchos::TimeMonitor' for 1st argument
class TEUCHOSCOMM_LIB_DLL_EXPORT TimeMonitor :
^
/scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt/SRC_AND_BUILD/Trilinos/packages/muelu/test/maxwell/Maxwell3D.cpp:274:11: error: no viable overloaded '='
tm3 = Teuchos::null;
~~~ ^ ~~~~~~~~~~~~~
/scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt/SRC_AND_BUILD/Trilinos/packages/teuchos/comm/src/Teuchos_TimeMonitor.hpp:178:34: note: candidate function (the implicit copy assignment operator) not viable: no known conversion from 'Teuchos::ENull' to 'const Teuchos::TimeMonitor' for 1st argument
class TEUCHOSCOMM_LIB_DLL_EXPORT TimeMonitor :
^
2 errors generated.
```
## Current Status on CDash
The current status of these tests/builds for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=65&value2=MueLu_Maxwell3D-&field3=testname&compare3=66&value3=_MPI_4&field4=site&compare4=61&value4=cee-rhel6)
## Steps to Reproduce
One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for a machine with a cee rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16
```
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/4982MueLu: MueLu_Helmholtz2DParallel_MPI_4 failing on ATDM complex builds2019-06-08T15:27:25ZJames WillenbringMueLu: MueLu_Helmholtz2DParallel_MPI_4 failing on ATDM complex builds*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia...*Created by: fryeguy52*
## Bug Report
CC: @trilinos/muelu, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=-complex-&field3=testname&compare3=65&value3=MueLu_Helmholtz2DParallel_MPI_4&field4=buildstarttime&compare4=84&value4=2019-04-22T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2019-03-23T00%3A00%3A00) the test:
* MueLu_Helmholtz2DParallel_MPI_4
has been failing since 2019-01-11 in the builds:
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-debug
* Trilinos-atdm-sems-rhel7-gnu-7.2.0-openmp-complex-shared-release-debug
* Trilinos-atdm-sems-rhel7-intel-17.0.1-openmp-complex-shared-release-debug
* Trilinos-atdm-sems-rhel7-clang-3.9.0-openmp-complex-shared-release-debug
new commits on 2019-04-11 can be found [here](https://testing.sandia.gov/cdash/viewNotes.php?buildid=4867040#!#note2)
## Current Status on CDash
[results for the current testing day](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=63&value2=complex&field3=testname&compare3=65&value3=MueLu_Helmholtz2DParallel_MPI_4&field4=buildstarttime&compare4=84&value4=today&field5=buildstarttime&compare5=83&value5=yesterday)
## Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
More specifically, the commands given for with a sems rhel6 environment are provided at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#sems-rhel6-environment
The exact commands to reproduce this issue should be:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel6-gnu-7.2.0-openmp-complex-shared-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8
```
Keep promoted "ATDM" builds of Trilinos clean