Many PanzerAdaptersSTK (random?) failures in ATDM sems-rhel7+cuda+complex builds
Created by: fryeguy52
CC: @trilinos/panzer, @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe, @fryeguy52
??: Add label "ATDM"> ??: Add label "bug"?> ??: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)> ??: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean"> ??: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files> ??: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>Next Action Status
Description
As shown in this query these tests:
- PanzerAdaptersSTK_CurlLaplacianExample
- PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1
- PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2
- PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3
- PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4
- PanzerAdaptersSTK_CurlLaplacianMultiblockExample-ConvTest-Quad-Order-1
- PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1
- PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2
- PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3
- PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1
- PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2
- PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1
- PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-1
- PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-2
- PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-3
- PanzerAdaptersSTK_MixedPoissonMultiblockExample-ConvTest-Hex-Order-1
- PanzerAdaptersSTK_PoissonInterfaceExample_2d_diffsideids_MPI_1
have failed at least once on one of these builds since 2019-03-21:
- Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
- Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug
Current Status on CDash
PanzerAdaptersSTK tests that failed on those builds for the current testing day can be found here
Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
More specifically, the commands given for with a sems rhel6 environment are provided at:
The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Panzer=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8