Many (50-80) Panzer tests failing on all ATDM cuda builds
Created by: fryeguy52
CC: @trilinos/panzer, @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe, @fryeguy52
Next Action Status
Description
As shown in this query about half of the Panzer tests started failing on 2019-01-11 on all the cuda 9.2 ATDM builds the failures are too numerous to list here but the largest groups are:
PanzerAdaptersSTK_*
PanzerDiscFE_*
PanzerDofMgr_*
PanzerMiniEM_MiniEM-BlockPrec_*
the affected builds are:
- Trilinos-atdm-waterman-cuda-9.2-debug
- Trilinos-atdm-waterman-cuda-9.2-opt
- Trilinos-atdm-waterman-cuda-9.2-release-debug
- Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug
- Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release
- Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug
the tests are failing with segmentation faults
0. tFilteredUGI_equivalence_test_UnitTest ... --------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node white22 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
@rppawlo there were a couple new commits that touched panzer files that you made yesterday. Can you see if these are what caused the new failures?
cf5e5a3: Panzer: hack for gcc 4.8 compiler
1333649: Panzer: fix all shadow warnings
Current Status of these builds on CDash
The status of these builds for the current testing day can be found at:
Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
More specifically, the commands given for ride or white are provided at:
The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Panzer=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16