Segfault when updating SEACAS and STK
Created by: bmpersc
I'm seeing the following segfault in PanzerAdaptersSTK_tLineMeshFactory_MPI_2 when building with an updated STK. If this is insufficient to address this issue please let me know and I will find a way to get you the copy of Trilinos with the updates so you can debug the issue.
Output from test:
Start 75: PanzerAdaptersSTK_tLineMeshFactory_MPI_2
75: Test command: /sierra/sntools/SDK/mpi/openmpi/1.6.4-gcc-4.7.2-RHEL6/bin/mpiexec "-np" "2" "/scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/test/stk_interface_test/PanzerAdaptersSTK_tLineMeshFactory.exe" 75: Test timeout computed to be: 300 75: Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1103 and rank 0! 75: Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1103 and rank 1! 75: 75: *** 75: *** Unit test suite ... 75: *** 75: 75: 75: Sorting tests by group name then by the order they were added ... (time = 0.000201) 75: 75: Running unit tests ... 75: 75: [ceerws1103:61629] *** Process received signal *** 75: [ceerws1103:61630] *** Process received signal *** 75: [ceerws1103:61629] Signal: Segmentation fault (11) 75: [ceerws1103:61629] Signal code: Address not mapped (1) 75: [ceerws1103:61629] Failing at address: (nil) 75: [ceerws1103:61630] Signal: Segmentation fault (11) 75: [ceerws1103:61630] Signal code: Address not mapped (1) 75: [ceerws1103:61630] Failing at address: (nil) 75: [ceerws1103:61629] [ 0] /lib64/libpthread.so.0() [0x332540f710] 75: [ceerws1103:61629] [ 1] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh4impl37fill_element_side_nodes_from_topologyERKNS0_8BulkDataENS0_6EntityEjRSt6vectorIS5_SaIS5_EE+0x1a9) [0x7fd5d9116cb9] 75: [ceerws1103:61629] [ 2] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZNK3stk4mesh13ElemElemGraph30add_local_graph_edges_for_elemERKNS0_9MeshIndexElRSt6vectorINS0_9GraphEdgeESaIS6_EES9_b+0x16f) [0x7fd5d9106f2f] 75: [ceerws1103:61629] [ 3] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh13ElemElemGraph10fill_graphEv+0x198) [0x7fd5d9107688] 75: [ceerws1103:61629] [ 4] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh13ElemElemGraph14fill_from_meshEv+0x69) [0x7fd5d9108c89] 75: [ceerws1103:61629] [ 5] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh13ElemElemGraphC2ERNS0_8BulkDataE+0x14c) [0x7fd5d9108f3c] 75: [ceerws1103:61629] [ 6] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh8BulkData38initialize_face_adjacent_element_graphEv+0x48) [0x7fd5d8f8be48] 75: [ceerws1103:61629] [ 7] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh16create_all_sidesERNS0_8BulkDataERKNS0_8SelectorERKSt6vectorIPNS0_4PartESaIS8_EEb+0x25) [0x7fd5d903f3f5] 75: [ceerws1103:61629] [ 8] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh24create_adjacent_entitiesERNS0_8BulkDataERSt6vectorIPNS0_4PartESaIS5_EE+0xbc) [0x7fd5d906b24c] 75: [ceerws1103:61629] [ 9] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/src/libpanzer-stk.so.12(_ZN10panzer_stk13STK_Interface13buildSubcellsEv+0x2e) [0x7fd5e605cb9e] 75: [ceerws1103:61629] [10] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/src/libpanzer-stk.so.12(_ZNK10panzer_stk15LineMeshFactory24completeMeshConstructionERNS_13STK_InterfaceEP19ompi_communicator_t+0x6c) [0x7fd5e60845dc] 75: [ceerws1103:61629] [11] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/src/libpanzer-stk.so.12(_ZNK10panzer_stk15LineMeshFactory9buildMeshEP19ompi_communicator_t+0x69) [0x7fd5e6084429] 75: [ceerws1103:61629] [12] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/test/stk_interface_test/PanzerAdaptersSTK_tLineMeshFactory.exe(_ZNK10panzer_stk34tLineMeshFactory_defaults_UnitTest15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb+0x3a) [0x416a1a] 75: [ceerws1103:61629] [13] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/teuchos/core/src/libteuchoscore.so.12(_ZNK7Teuchos12UnitTestBase11runUnitTestERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x2b) [0x7fd5cb66985b] 75: [ceerws1103:61629] [14] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository15runUnitTestImplERKNS_12UnitTestBaseERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x16) [0x7fd5cb673256] 75: [ceerws1103:61629] [15] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/teuchos/core/src/libteuchoscore.so.12(_Z[ceerws1103:61630] [ 0] /lib64/libpthread.so.0() [0x332540f710] 75: [ceerws1103:61630] [ 1] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh4impl37fill_element_side_nodes_from_topologyERKNS0_8BulkDataENS0_6EntityEjRSt6vectorIS5_SaIS5_EE+0x1a9) [0x7f1bf1e3acb9] 75: [ceerws1103:61630] [ 2] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZNK3stk4mesh13ElemElemGraph30add_local_graph_edges_for_elemERKNS0_9MeshIndexElRSt6vectorINS0_9GraphEdgeESaIS6_EES9_b+0x16f) [0x7f1bf1e2af2f] 75: [ceerws1103:61630] [ 3] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh13ElemElemGraph10fill_graphEv+0x198) [0x7f1bf1e2b688] 75: [ceerws1103:61630] [ 4] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh13ElemElemGraph14fill_from_meshEv+0x69) [0x7f1bf1e2cc89] 75: [ceerws1103:61630] [ 5] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh13ElemElemGraphC2ERNS0_8BulkDataE+0x14c) [0x7f1bf1e2cf3c] 75: [ceerws1103:61630] [ 6] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh8BulkData38initialize_face_adjacent_element_graphEv+0x48) [0x7f1bf1cafe48] 75: [ceerws1103:61630] [ 7] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh16create_all_sidesERNS0_8BulkDataERKNS0_8SelectorERKSt6vectorIPNS0_4PartESaIS8_EEb+0x25) [0x7f1bf1d633f5] 75: [ceerws1103:61630] [ 8] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.so.12(_ZN3stk4mesh24create_adjacent_entitiesERNS0_8BulkDataERSt6vectorIPNS0_4PartESaIS5_EE+0xbc) [0x7f1bf1d8f24c] 75: [ceerws1103:61630] [ 9] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/src/libpanzer-stk.so.12(_ZN10panzer_stk13STK_Interface13buildSubcellsEv+0x2e) [0x7f1bfed80b9e] 75: [ceerws1103:61630] [10] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/src/libpanzer-stk.so.12(_ZNK10panzer_stk15LineMeshFactory24completeMeshConstructionERNS_13STK_InterfaceEP19ompi_communicator_t+0x6c) [0x7f1bfeda85dc] 75: [ceerws1103:61630] [11] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/src/libpanzer-stk.so.12(_ZNK10panzer_stk15LineMeshFactory9buildMeshEP19ompi_communicator_t+0x69) [0x7f1bfeda8429] 75: [ceerws1103:61630] [12] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/test/stk_interface_test/PanzerAdaptersSTK_tLineMeshFactory.exe(_ZNK10panzer_stk34tLineMeshFactory_defaults_UnitTest15runUnitTestImplERN7Teuchos18basic_FancyOStreamIcSt11char_traitsIcEEERb+0x3a) [0x416a1a] 75: [ceerws1103:61630] [13] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/teuchos/core/src/libteuchoscore.so.12(_ZNK7Teuchos12UnitTestBase11runUnitTestERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x2b) [0x7f1be438d85b] 75: [ceerws1103:61630] [14] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository15runUnitTestImplERKNS_12UnitTestBaseERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x16) [0x7f1be4397256] 75: [ceerws1103:61630] [15] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository12runUnitTestsERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x1132) [0x7fd5cb674872] 75: [ceerws1103:61629] [16] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository20runUnitTestsFromMainEiPPc+0xba) [0x7fd5cb675fda] 75: [ceerws1103:61629] [17] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/test/stk_interface_test/PanzerAdaptersSTK_tLineMeshFactory.exe(main+0x3c1) [0x425061] 75: [ceerws1103:61629] [18] /lib64/libc.so.6(__libc_start_main+0xfd) [0x332501ed5d] 75: [ceerws1103:61629] [19] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/test/stk_interface_test/PanzerAdaptersSTK_tLineMeshFactory.exe() [0x4166cd] 75: [ceerws1103:61629] *** End of error message *** 75: N7Teuchos18UnitTestRepository12runUnitTestsERNS_18basic_FancyOStreamIcSt11char_traitsIcEEE+0x1132) [0x7f1be4398872] 75: [ceerws1103:61630] [16] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/teuchos/core/src/libteuchoscore.so.12(_ZN7Teuchos18UnitTestRepository20runUnitTestsFromMainEiPPc+0xba) [0x7f1be4399fda] 75: [ceerws1103:61630] [17] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/test/stk_interface_test/PanzerAdaptersSTK_tLineMeshFactory.exe(main+0x3c1) [0x425061] 75: [ceerws1103:61630] [18] /lib64/libc.so.6(__libc_start_main+0xfd) [0x332501ed5d] 75: [ceerws1103:61630] [19] /scratch/bmpersc/sierra_integration/sierra_to_trilinos/checkintest_trilinos/MPI_DEBUG_SS/packages/panzer/adapters-stk/test/stk_interface_test/PanzerAdaptersSTK_tLineMeshFactory.exe() [0x4166cd] 75: [ceerws1103:61630] *** End of error message *** 75: -------------------------------------------------------------------------- 75: mpiexec noticed that process rank 0 with PID 61629 on node ceerws1103 exited on signal 11 (Segmentation fault). 75: -------------------------------------------------------------------------- 1/1 Test #75 (closed): PanzerAdaptersSTK_tLineMeshFactory_MPI_2 ...***Failed 2.60 sec