PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-2 failing(timeout) in ATDM cuda builds
Created by: fryeguy52
CC: @trilinos/panzer , @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe
Next Action Status
PR #3800 merged on 11/9/2018 resulted in this test passing in all CUDA 'opt' builds and being disabled in all CUDA 'debug' builds starting 11/10/2018 and for three consecutive days as of 11/12/2018.
Description
As shown in this query the tests:
- PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-2
are failing in the builds:
- Trilinos-atdm-white-ride-cuda-9.2-opt
- Trilinos-atdm-waterman-cuda-9.2-debug
- Trilinos-atdm-waterman-cuda-9.2-release-debug
- Trilinos-atdm-white-ride-cuda-9.2-opt
- Trilinos-atdm-white-ride-cuda-9.2-debug
- Trilinos-atdm-hansen-shiller-cuda-8.0-opt
- Trilinos-atdm-hansen-shiller-cuda-8.0-debug
- Trilinos-atdm-hansen-shiller-cuda-9.0-debug
- Trilinos-atdm-hansen-shiller-cuda-9.0-opt
Are these related to #2446 or #2751?
The timeouts started on 2018-10-31 for a few of the builds then others started in the following days. shown here on waterman cuda-9.2-debug
The new commits that were pulled on 2018-11-03 that look like possible causes are:
e8051f3: Panzer: update CurlLaplacian example
Author: Roger Pawlowski <rppawlo@sandia.gov>
Date: Tue Oct 23 09:51:52 2018 -0600
de16afa: Panzer: enable hierarchic parallelism in evaluators
Author: Roger Pawlowski <rppawlo@sandia.gov>
Date: Thu Oct 18 15:18:43 2018 -0600
@rppawlo can you look and see if anything in these commits may have caused this test to start timing out?
Current Status on CDash
The current status of these tests/builds for the current testing day can be found at:
Steps to Reproduce
One should be able to reproduce this failure waterman as described in:
- https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md More specifically, the commands given for the system waterman are provided at:
- https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Panzer=ON \
$TRILINOS_DIR
$ make NP=20
$ bsub -x -Is -n 20 ctest -j20