Stratimikos_test_single_belos_thyra_solver_driver_nos1_nrhs8_MPI_1 failing on KNL mutrino ATDM build

Created by: fryeguy52

CC: @trilinos/, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe

Next Action Status

Description

As shown in this query the test:

Stratimikos_test_single_belos_thyra_solver_driver_nos1_nrhs8_MPI_1

is failing in the builds:

Trilinos-atdm-mutrino-intel-opt-openmp-KNL

Detailed test output can be found here. Looks like a failure to say that it converged:

    Starting iterations with Belos:
      Using forward operator = N5Thyra14EpetraLinearOpE{rangeDim=237,domainDim=237}
      Using iterative solver = "Belos::BlockGmresSolMgr":
        Template parameters:
         ScalarType: double
         MV: N5Thyra15MultiVectorBaseIdEE
         OP: N5Thyra12LinearOpBaseIdEE
        Flexible: false
        Num Blocks: 300
        Maximum Iterations: 1000
        Maximum Restarts: 25
        Convergence Tolerance: 1e-13
      With #Eqns=237, #RHSs=8 ...
     
     The Belos solver of type ""Belos::BlockGmresSolMgr": {Flexible: false, Num Blocks: 300, Maximum Iterations: 1000, Maximum Restarts: 25, Convergence Tolerance: 1e-13}" returned a solve status of "SOLVE_STATUS_UNCONVERGED" in 1 iterations with total CPU time of 0.241756 sec
     
     => Solve time = 0.241863 sec
    
    solve status:
      solveStatus = SOLVE_STATUS_UNCONVERGED
      achievedTol = 3.94193e-16
      message:
       The Belos solver of type ""Belos::BlockGmresSolMgr": {Flexible: false, Num Blocks: 300, Maximum Iterations: 1000, Maximum Restarts: 25, Convergence Tolerance: 1e-13}" returned a solve status of "SOLVE_STATUS_UNCONVERGED" in 1 iterations with total CPU time of 0.241756 sec
      extraParameters:
                 Belos/Iteration Count : int = 1   [unused]
                 Iteration Count : int = 1   [unused]
                 Belos/Achieved Tolerance : double = 3.94193e-16   [unused]
    
    check: solveStatus = SOLVE_STATUS_UNCONVERGED == SOLVE_STATUS_CONVERGED : FAILED

Current Status on CDash

The current status of these tests/builds for the current testing day can be found at: Tests status for current day

This build takes a long time run and the results may not be available until later in the day, so the below is a link the status of the tests yesterday.

Test status of previous day

Steps to Reproduce

One should be able to reproduce this failure on the machine mutrino as described in:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md More specifically, the commands given for the system mutrino are provided at:
https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#mutrino The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-mutrino-intel-opt-openmp-KNL

$ cmake \
  -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
  -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Stratimikos=ON \
  $TRILINOS_DIR

$ make -j16

$ salloc -N 1 -p standard -J Trilinos-atdm-mutrino-intel-opt-openmp-KNL ctest -j16