*Created by: fryeguy52* CC: @trilinos/belos , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe ## Next Action Status PR #3951 merged to 'develop' on 11/28/2018 resulted in this test passing in the Intel 18.0.2 builds on 'mutrino' and the 'cee-rhel6' builds on 12/1/2018 and in all builds for several days as of 12/3/2018. ## Description As shown in [this query](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&date=2018-09-24&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=site&compare2=61&value2=mutrino&field3=status&compare3=62&value3=passed&field4=buildstarttime&compare4=83&value4=2018-09-01&field5=testname&compare5=63&value5=Belos) the test: * Belos_gcrodr_hb_MPI_4 is failing in the builds: * Trilinos-atdm-mutrino-intel-opt-openmp-HSW * Trilinos-atdm-mutrino-intel-opt-openmp-KNL some test output: ``` *** Error in `/lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/BUILD/packages/belos/epetra/test/GCRODR/Belos_gcrodr_hb.exe': free(): invalid pointer: 0x000001000011bba0 *** *** Error in `/lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/BUILD/packages/belos/epetra/test/GCRODR/Belos_gcrodr_hb.exe': free(): invalid pointer: 0x00000100004b4980 *** *** Error in `/lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/BUILD/packages/belos/epetra/test/GCRODR/Belos_gcrodr_hb.exe': free(): invalid pointer: 0x00000100004b4980 *** *** Error in `/lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/BUILD/packages/belos/epetra/test/GCRODR/Belos_gcrodr_hb.exe': free(): invalid pointer: 0x00000100004b4980 *** ``` ## Steps to Reproduce One should be able to reproduce this failure on the machine mutrino as described in: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md More specifically, the commands given for the system mutrino are provided at: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#mutrino The exact commands to reproduce this issue should be: ``` $ cd <some_build_dir>/ $ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh intel-opt-openmp-HSW $ cmake \ -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \ -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \ $TRILINOS_DIR $ make -j16 $ salloc -N 1 -p standard -J $JOB_NAME ctest -j16 ```