Ifpack2: Ifpack2_MTSGS_belos_MPI_1 randomly failing in ATDM cee rhel6 intel build

Created by: fryeguy52

Bug Report

CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52

Next Action Status

Description

As shown in this query the test:

• Ifpack2_MTSGS_belos_MPI_1

is failing in the build:

• Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt

This has failed 9 times in the last 4 weeks with something similar to:

Achieved tolerance: 6.44895e-10
Actual iters(20) > expected number of iterations (19), or resid-norm(0) >= 1.e-7
proc 0 total program time: 0.0258739
End Result: TEST FAILED

Current Status on CDash

Current 2 week history on CDash

Steps to Reproduce

One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

More specifically, the commands given for a machine with a cee rhel6 environment are provided at:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment

The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
 $TRILINOS_DIR
$ make NP=16
$ ctest -j16