*Created by: fryeguy52* CC: @trilinos/anasazi, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52 ## Next Action Status Triggered by the PR #3951 merged to 'develop' on 10/28/2018 that worked around Intel 18.0.2 MKL GEEV defect. Next: Try updated Intel MKL 18.0.5 on 'mutrino' (with local revert of #3951) and see all of these failures go away (@fryeguy52) ... ## Description As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=buildstarttime&compare3=83&value3=2018-11-04T00%3A00%3A00&field4=status&compare4=61&value4=Failed) the test: * Anasazi_Epetra_BKS_norestart_test_MPI_4 is failing in the builds: * Trilinos-atdm-mutrino-intel-opt-openmp-HSW (since ???) * Trilinos-atdm-mutrino-intel-opt-openmp-KNL (since ???) * Trilinos-atdm-cee-rhel6-intel-17.0.1-intelmpi-5.1.2-serial-static-opt (since 11/30/2018) * Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt (11/29/2018 & 12/1/2018) * Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt (on 12/2/2018) * Trilinos-atdm-cee-rhel6-gnu-4.9.3-openmpi-1.10.2-serial-static-opt (on 12/10/2018) <more-details> Looks like some of these failures are random like shown for the build [Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-clang-5.0.1-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=83&value4=2018-11-11T00%3A00%3A00) and the build [Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=Trilinos-atdm-cee-rhel6-gnu-7.2.0-openmpi-1.10.2-serial-static-opt&field2=testname&compare2=61&value2=Anasazi_Epetra_BKS_norestart_test_MPI_4&field3=site&compare3=61&value3=cee-rhel6&field4=buildstarttime&compare4=84&value4=2018-12-11T00%3A00%3A00&field5=buildstarttime&compare5=83&value5=2018-11-11T00%3A00%3A00). The errors look like [here](https://testing.sandia.gov/cdash/testDetails.php?test=61150478&build=4276066) for example: ``` Number of iterations performed in BlockKrylovSchur_test.exe: 30 Direct residual norms computed in BlockKrylovSchur_test.exe Eigenvalue Residual ---------------------------------------- 1.199112e+05 1.296543e-07 1.196455e+05 1.185550e-07 1.192047e+05 4.530562e-04 1.185918e+05 1.497329e-04 1.178109e+05 4.552932e-04 End Result: TEST FAILED ------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. ------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[25128,1],1] Exit code: 255 -------------------------------------------------------------------------- ... ``` ## Current Status on CDash The current status of these tests/builds for the current testing day can be found [here](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=buildname&compare2=62&value2=Trilinos-atdm-cee-rhel6-intel-18.0.2-mpich2-3.2-serial-static-opt&field3=testname&compare3=61&value3=Anasazi_Epetra_BKS_norestart_test_MPI_4&field4=buildstarttime&compare4=83&value4=1%20day%20ago&field5=status&compare5=61&value5=Failed&field6=site&compare6=62&value6=mutrino) ## Steps to Reproduce One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md More specifically, the commands given for a machine with a cee rhel6 environment are provided at: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment The exact commands to reproduce this issue should be: ``` $ cd <some_build_dir>/ $ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6-intel-17.0.1-intelmpi-5.1.2-serial-static-opt $ cmake \ -GNinja \ -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \ -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Anasazi=ON \ $TRILINOS_DIR $ make NP=16 $ ctest -j16 ```