New PanzerMiniEM test failures in ATDM Trilinos builds starting 7/21/2018
Created by: bartlettroscoe
CC: @trilinos/panzer, @trilinos/muelu, @cgcgcg (pushed breaking commits?), @rppawlo (Trilinos Nonlinear Solvers Product Lead)
Next Action Status
Commit 65a9d011 from PR #3205 merged on 7/30/2015 disabled all of the PanzerMiniEM_MiniEM-BlockPrec_XXX
tests in all of the ATDM builds (no MueLu Epetra support enabled in ATDM Trilinos builds). The existing Tpetra-based PanzerMiniEM tests are still running and passing which is all that is needed to protect EMPIRE.
Description
As shown in this query, the tests:
- PanzerMiniEM_MiniEM-BlockPrec_Augmentation_Epetra_MPI_1
- PanzerMiniEM_MiniEM-BlockPrec_Augmentation_Epetra_MPI_4
- PanzerMiniEM_MiniEM-BlockPrec_RefMaxwell_Epetra_MPI_1
- PanzerMiniEM_MiniEM-BlockPrec_RefMaxwell_MPI_1
- PanzerMiniEM_MiniEM-BlockPrec_RefMaxwell_MPI_4
started failing consistently on in the builds:
- Trilinos-atdm-hansen-shiller-cuda-8.0-debug
- Trilinos-atdm-hansen-shiller-cuda-8.0-opt
- Trilinos-atdm-hansen-shiller-cuda-9.0-debug
- Trilinos-atdm-hansen-shiller-cuda-9.0-opt
- Trilinos-atdm-hansen-shiller-gnu-debug-serial
- Trilinos-atdm-hansen-shiller-gnu-opt-serial
- Trilinos-atdm-hansen-shiller-intel-debug-serial
- Trilinos-atdm-hansen-shiller-intel-opt-serial
- Trilinos-atdm-rhel6-gnu-debug-serial
- Trilinos-atdm-rhel6-gnu-opt-serial
starting on 7/21/2018.
NOTES on other failing tests shown in that above CDash query:
-
The test failure
PanzerAdaptersSTK_model_evaluator_mass_check_MPI_1
in the buildTrilinos-atdm-rhel6-gnu-debug-serial
on 'sems-rhel6' at 2018-07-20 09:35:22 with results shown here was due to a freak build failure for the Panzer executablePanzerAdaptersSTK_model_evaluator_mass_check
wn here. -
The test timeout at 10 min
PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-3
in the buildTrilinos-atdm-hansen-shiller-gnu-debug-serial
on 'hansen' on 2018-07-13T06:11:37 UTC shownn here looks to be a freak timeout if you look at this query and not related to this. -
The sites 'chama' and 'mutrino' were filtered out since the have different issues causing failures.
If you look at the git commits pulled when these tests first started failing on 7/21/2018, for example here, it seems likely the commits merged to 'develop' on 7/20/2018 by @cgcgcg in RR #3047 by @cgcgcg are the likely trigger of these new test failures.
Steps to reproduce
These failures should be reproducable on any SNL COE RHEL6 machine with the SEMS env or the Test Bed machines 'hansen' or 'shiller' as described at:
The specific instructions for a RHEL6 mahcine are given at:
After cloning Trilinos, the following commands should reproduce the test failures with:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh gnu-debug-serial
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnvSettings.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_PanzerMiniEM=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16