*Created by: fryeguy52* CC: @trilinos/anasazi , @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe ## Next Action Status ## Description As shown in [this query](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&date=2018-09-24&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=site&compare2=61&value2=mutrino&field3=status&compare3=62&value3=passed&field4=buildstarttime&compare4=83&value4=2018-09-01&field5=testname&compare5=63&value5=Anasazi) the tests: * `Anasazi_MultiVecTraitsTest2_MPI_4` * `Anasazi_Epetra_BKS_norestart_test_MPI_4` are failing in the builds: * Trilinos-atdm-mutrino-intel-opt-openmp-HSW * Trilinos-atdm-mutrino-intel-opt-openmp-KNL both of these tests started failing on 9-22-2018. The test `Anasazi_Epetra_BKS_norestart_test_MPI_4` is also failing in the build `Trilinos-atdm-cee-rhel6-intel-18.0.2-mpich2-3.2-serial-static-opt` for the 'cee-rhel6' inv since it was first set up. The first failures of the test `Anasazi_MultiVecTraitsTest2_MPI_4` on 9/22/2018 is shown [here](https://testing.sandia.gov/cdash-dev-view/testDetails.php?test=55028214&build=3963372) which shows: ``` Check B_view = CloneViewNonConst(B, ind): ind: [0, 2, 4, 6, 8] static_cast<size_t> (B_view->getNumVectors ()) = 5 == static_cast<size_t> (ind.size ()) = 5 : passed norms of CloneViewNonConst(B, ind): [2.42234, 2.43667, 2.43783, 2.39508, 2.97253] B_view_norms[j] = 2.42233923795730766e+00 == normsB1[ind[j]] = 2.42233923795730766e+00 : passed B_view_norms[j] = 2.43666989157577962e+00 == normsB1[ind[j]] = 2.43666989157578007e+00 : FAILED ==> /lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/Trilinos/packages/anasazi/tpetra/test/MVOPTester/MultiVecTraitsTest2.cpp:573 ... [FAILED] (0.158 sec) MultiVecTraits_TpetraSetBlock4_UnitTest Location: /lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/Trilinos/packages/anasazi/tpetra/test/MVOPTester/MultiVecTraitsTest2.cpp:434 The following tests FAILED: 3. MultiVecTraits_TpetraSetBlock4_UnitTest ... Total Time: 1.55 sec Summary: total = 4, run = 4, passed = 3, failed = 1 ``` The first failures of the test `Anasazi_Epetra_BKS_norestart_test_MPI_4` on 9/22/2018 is shown [here](https://testing.sandia.gov/cdash-dev-view/testDetails.php?test=55028175&build=3963372) which shows: ``` Anasazi_Epetra_BKS_norestart_test.exe: malloc.c:2392: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long double) : 2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long double) : 2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed. Anasazi_Epetra_BKS_norestart_test.exe: malloc.c:2392: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long double) : 2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long double) : 2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed. srun: error: nid00012: tasks 0,3: Segmentation fault srun: Terminating job step 11643442.1635 srun: error: nid00012: task 1: Aborted slurmstepd: error: *** STEP 11643442.1635 ON nid00012 CANCELLED AT 2018-09-22T07:50:20 *** srun: error: nid00012: task 2: Aborted (core dumped) ``` New commits for this build can be seen [here](https://testing.sandia.gov/cdash-dev-view/viewNotes.php?buildid=3963370#!#note6) ## Current Status on CDash See: * [Non passing Anasazi tests in 'mutrino' builds last 2 days](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=5&showfilters=1&filtercombine=and&field1=testname&compare1=65&value1=Anasazi_&field2=buildname&compare2=65&value2=Trilinos-atdm-mutrino-&field3=groupname&compare3=62&value3=Experimental&field4=status&compare4=62&value4=passed&field5=buildstarttime&compare5=83&value5=2%20days%20ago) ## Steps to Reproduce One should be able to reproduce this failure on the machine mutrino as described in: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md More specifically, the commands given for the system mutrino are provided at: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#mutrino The exact commands to reproduce this issue should be: ``` $ cd <some_build_dir>/ $ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh intel-opt-openmp-HSW $ cmake \ -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \ -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Anasazi=ON \ $TRILINOS_DIR $ make -j16 $ salloc -N 1 -p standard -J $JOB_NAME ctest -j16 ```