Panzer build failures in Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt starting 2/22/2019 taking out SPARC Trilinos Integration testing

Created by: bartlettroscoe

CC: @trilinos/panzer , @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe, @fryeguy52, @alanw0

Next Action Status

Panzer was disabled in this build in PR #4487 on 2/24/2019 and the build 100% passed on 2/25/2019. PR #4503 that re-enables Panzer in this build was merged on 2/26/2019. Panzer built and passed tests in clang-5.0.1 build on 2/27/2019 as shown here.

Description

As shown in this query starting 2/22/2018 Panzer has many build and install failures in the build

Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt

The build failures, for example, are shown here and show STK link failures like:

data<(stk::topology::topology_t)0, void>::spatial_dimension_vector
>>> referenced by /scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt/SRC_AND_BUILD/Trilinos/packages/stk/stk_mesh/stk_mesh/base/MetaData.cpp
>>>               MetaData.cpp.o:(stk::topology::apply_functor<stk::topology_detail::defined_on_spatial_dimension_impl>::operator()(stk::topology::topology_t)) in archive packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.a

/usr/netpub/lld/bin/ld: error: symbol 'stk::topology_detail::topology_data<(stk::topology::topology_t)0, void>::spatial_dimension_vector' defined in packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.a(MetaData.cpp.o) has no type
/usr/netpub/lld/bin/ld: error: undefined symbol: stk::topology_detail::topology_data<(stk::topology::topology_t)1, void>::spatial_dimension_vector
>>> referenced by /scratch/rabartl/Trilinos.base/NightlyBuilds/Trilinos-atdm-cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt/SRC_AND_BUILD/Trilinos/packages/stk/stk_mesh/stk_mesh/base/MetaData.cpp
>>>               MetaData.cpp.o:(stk::topology::apply_functor<stk::topology_detail::defined_on_spatial_dimension_impl>::operator()(stk::topology::topology_t)) in archive packages/stk/stk_mesh/stk_mesh/base/libstk_mesh_base.a

The new commits pulled on 2/22/2019 in this build are shown here and it seems likely the STK update in the commits shown below from @alanw0 are the likely cause.

STK Commits (click to expand)

400560bc40:  Make stk_ngp_test usable as header-only. (Try to avoid the need for --rdc in cuda build.)
Author: Alan Williams <william@sandia.gov>
Date:   Tue Feb 19 15:35:08 2019 -0700

M	packages/stk/stk_ngp_test/stk_ngp_test/CMakeLists.txt
M	packages/stk/stk_ngp_test/stk_ngp_test/GlobalReporter.cpp
M	packages/stk/stk_ngp_test/stk_ngp_test/GlobalReporter.hpp
M	packages/stk/stk_ngp_test/stk_ngp_test/Reporter.cpp
M	packages/stk/stk_ngp_test/stk_ngp_test/Reporter.hpp
M	packages/stk/stk_ngp_test/stk_ngp_test/ngp_test.cpp
M	packages/stk/stk_ngp_test/stk_ngp_test/ngp_test.hpp

b92e9d8bfe:  Try to solve the undefined extern function for cuda in stk_ngp_test.
Author: Alan Williams <william@sandia.gov>
Date:   Thu Feb 14 15:08:57 2019 -0700

M	packages/stk/stk_ngp_test/stk_ngp_test/Reporter.cpp
M	packages/stk/stk_ngp_test/stk_ngp_test/Reporter.hpp

6d16a65a94:  Fix compile error in stk_ngp_test file.
Author: Alan Williams <william@sandia.gov>
Date:   Wed Feb 13 11:50:20 2019 -0700

M	packages/stk/stk_ngp_test/stk_ngp_test/Reporter.hpp

dbf8d78fca:  Another tweak to deprecating ngp::Field::copy_* to become private.
Author: Alan Williams <william@sandia.gov>
Date:   Tue Feb 12 12:20:01 2019 -0700

M	packages/stk/stk_ngp/stk_ngp/NgpField.hpp

1ea65f4ffc:  Ok, one more try.
Author: Alan Williams <william@sandia.gov>
Date:   Tue Feb 12 12:05:08 2019 -0700

A	packages/stk/stk_unit_tests/stk_ngp_test/CMakeLists.txt

7557aafa7f:  Fix the macro managing the deprecation of ngp::Field's copy_* methods from public to private.
Author: Alan Williams <william@sandia.gov>
Date:   Mon Feb 11 13:26:26 2019 -0700

M	packages/stk/stk_ngp/stk_ngp/NgpField.hpp

62a1e8bb44:  Fix stk cmake files.
Author: Alan Williams <william@sandia.gov>
Date:   Mon Feb 11 08:28:40 2019 -0700

M	cmake/dependencies/TrilinosPackageDependencies.xml
M	packages/stk/cmake/Dependencies.cmake
M	packages/stk/stk_balance/stk_balance/balance.cpp
M	packages/stk/stk_balance/stk_balance/balanceUtils.cpp
M	packages/stk/stk_balance/stk_balance/balanceUtils.hpp
A	packages/stk/stk_balance/unit_tests/UnitTestColoring.cpp
M	packages/stk/stk_doc_tests/stk_io/howToWriteMesh.cpp
M	packages/stk/stk_mesh/stk_mesh/base/BulkData.cpp
M	packages/stk/stk_mesh/stk_mesh/base/Part.hpp
M	packages/stk/stk_mesh/stk_mesh/base/Relation.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/FieldBaseImpl.cpp
M	packages/stk/stk_ngp/cmake/Dependencies.cmake
M	packages/stk/stk_ngp_test/cmake/Dependencies.cmake
A	packages/stk/stk_ngp_test/stk_ngp_test/CMakeLists.txt
M	packages/stk/stk_search/stk_search/KDTree_impl.hpp
M	packages/stk/stk_unit_test_utils/CMakeLists.txt
M	packages/stk/stk_unit_test_utils/cmake/Dependencies.cmake
M	packages/stk/stk_unit_tests/cmake/Dependencies.cmake
M	packages/stk/stk_util/stk_util/util/SortAndUnique.hpp

a6c29b6b13:  Added new stk subpackage stk_ngp_test to TrilinosPackageDependencies.xml
Author: Alan Williams <william@sandia.gov>
Date:   Fri Feb 8 16:37:12 2019 -0700

M	cmake/dependencies/TrilinosPackageDependencies.xml

006dfdce93:  Bring recent stk development into Trilinos.
Author: Alan Williams <william@sandia.gov>
Date:   Thu Feb 7 10:30:23 2019 -0700

M	packages/stk/stk_balance/Jamfile
M	packages/stk/stk_balance/integration_tests/StkbalanceUserSupport.cpp
M	packages/stk/stk_balance/stk_balance/internal/StkBalanceUtils.cpp
M	packages/stk/stk_doc_tests/stk_io/howToWriteMesh.cpp
M	packages/stk/stk_doc_tests/stk_io/writeResults.cpp
M	packages/stk/stk_exp/unit_tests/kokkos/copy_kokkos_memory.cpp
M	packages/stk/stk_exp/unit_tests/kokkos/create_kokkos_view.cpp
M	packages/stk/stk_exp/unit_tests/kokkos/saxpy_kokkos.cpp
M	packages/stk/stk_exp/unit_tests/kokkos/saxpy_kokkos.hpp
M	packages/stk/stk_exp/unit_tests/kokkos/unit_test_kokkos.cpp
M	packages/stk/stk_exp/unit_tests/kokkos/unit_test_kokkos_saxpy.cpp
M	packages/stk/stk_expreval/Jamfile
M	packages/stk/stk_expreval/stk_expreval/Evaluator.cpp
M	packages/stk/stk_integration_tests/stk_mesh/IntegrationTestElementBlockMembership.cpp
M	packages/stk/stk_io/Jamfile
M	packages/stk/stk_io/stk_io/FieldAndName.hpp
M	packages/stk/stk_io/stk_io/IossBridge.cpp
M	packages/stk/stk_io/stk_io/IossBridge.hpp
M	packages/stk/stk_io/stk_io/OutputFile.cpp
M	packages/stk/stk_io/stk_io/ProcessSetsOrBlocks.hpp
M	packages/stk/stk_io/stk_io/StkIoUtils.cpp
M	packages/stk/stk_io/stk_io/StkIoUtils.hpp
M	packages/stk/stk_learning/Kokkos/KokkosFieldBLAS.C
M	packages/stk/stk_learning/Kokkos/KokkosTutorial04.C
M	packages/stk/stk_learning/Kokkos/KokkosViewAllocOnDevice.C
M	packages/stk/stk_learning/Kokkos/KokkosWithTopology.C
M	packages/stk/stk_learning/Kokkos/Kokkos_TeamReductionTest.C
M	packages/stk/stk_learning/Kokkos/mtk_kokkos.h
M	packages/stk/stk_learning/Kokkos/testKokkosEncapsulation.C
M	packages/stk/stk_mesh/Jamfile
M	packages/stk/stk_mesh/stk_mesh/base/BulkData.cpp
M	packages/stk/stk_mesh/stk_mesh/base/BulkData.hpp
M	packages/stk/stk_mesh/stk_mesh/base/CoordinateSystems.cpp
M	packages/stk/stk_mesh/stk_mesh/base/CoordinateSystems.hpp
M	packages/stk/stk_mesh/stk_mesh/base/CreateEdges.cpp
M	packages/stk/stk_mesh/stk_mesh/base/EntityKey.hpp
M	packages/stk/stk_mesh/stk_mesh/base/FEMHelpers.cpp
M	packages/stk/stk_mesh/stk_mesh/base/FieldBase.hpp
M	packages/stk/stk_mesh/stk_mesh/base/FieldParallel.cpp
M	packages/stk/stk_mesh/stk_mesh/base/MetaData.cpp
M	packages/stk/stk_mesh/stk_mesh/base/MetaData.hpp
M	packages/stk/stk_mesh/stk_mesh/base/Part.hpp
M	packages/stk/stk_mesh/stk_mesh/base/Relation.cpp
M	packages/stk/stk_mesh/stk_mesh/base/Relation.hpp
M	packages/stk/stk_mesh/stk_mesh/base/Selector.cpp
M	packages/stk/stk_mesh/stk_mesh/base/Selector.hpp
M	packages/stk/stk_mesh/stk_mesh/base/SkinBoundary.cpp
M	packages/stk/stk_mesh/stk_mesh/base/SkinBoundary.hpp
M	packages/stk/stk_mesh/stk_mesh/base/SkinMeshUtil.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/BucketRepository.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/BucketRepository.hpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/EquivalentEntityBlocks.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/EquivalentEntityBlocks.hpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/FieldBaseImpl.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/FieldRepository.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/MeshImplUtils.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/PartImpl.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/PartImpl.hpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/Partition.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/elementGraph/ElemElemGraph.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/elementGraph/ElemElemGraphImpl.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/elementGraph/ElemElemGraphImpl.hpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/elementGraph/ElemGraphShellConnections.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/elementGraph/GraphEdgeData.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/elementGraph/GraphEdgeData.hpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/elementGraph/ParallelInfoForGraph.cpp
M	packages/stk/stk_mesh/stk_mesh/baseImpl/elementGraph/SparseGraph.hpp
M	packages/stk/stk_ngp/Jamfile
M	packages/stk/stk_ngp/stk_ngp/Ngp.hpp
M	packages/stk/stk_ngp/stk_ngp/NgpAtomics.hpp
M	packages/stk/stk_ngp/stk_ngp/NgpDynamicMesh.hpp
M	packages/stk/stk_ngp/stk_ngp/NgpField.hpp
M	packages/stk/stk_ngp/stk_ngp/NgpFieldManager.hpp
M	packages/stk/stk_ngp/stk_ngp/NgpMesh.hpp
M	packages/stk/stk_ngp/stk_ngp/NgpMultistateField.hpp
M	packages/stk/stk_ngp/stk_ngp/NgpReductions.hpp
M	packages/stk/stk_ngp/stk_ngp/NgpSpaces.hpp
A	packages/stk/stk_ngp_test/CMakeLists.txt
C070	packages/stk/stk_ngp/Jamfile	packages/stk/stk_ngp_test/Jamfile
A	packages/stk/stk_ngp_test/cmake/Dependencies.cmake
A	packages/stk/stk_ngp_test/stk_ngp_test/GlobalReporter.cpp
A	packages/stk/stk_ngp_test/stk_ngp_test/GlobalReporter.hpp
A	packages/stk/stk_ngp_test/stk_ngp_test/NgpTestDeviceMacros.hpp
A	packages/stk/stk_ngp_test/stk_ngp_test/Reporter.cpp
A	packages/stk/stk_ngp_test/stk_ngp_test/Reporter.hpp
A	packages/stk/stk_ngp_test/stk_ngp_test/main/main.cpp
A	packages/stk/stk_ngp_test/stk_ngp_test/ngp_test.cpp
A	packages/stk/stk_ngp_test/stk_ngp_test/ngp_test.hpp
M	packages/stk/stk_performance_tests/stk_mesh/CommunicateFieldData.cpp
M	packages/stk/stk_search/Jamfile
M	packages/stk/stk_search_util/Jamfile
M	packages/stk/stk_simd/stk_simd/Simd.hpp
M	packages/stk/stk_simd/stk_simd/avx/Avx.hpp
M	packages/stk/stk_simd/stk_simd/avx512/Avx512.hpp
M	packages/stk/stk_simd/stk_simd/no_simd/NoSimd.hpp
M	packages/stk/stk_simd/stk_simd/sse/Sse.hpp
M	packages/stk/stk_simd/stk_simd_view/simd_index.hpp
M	packages/stk/stk_simd/stk_simd_view/simd_parallel.hpp
M	packages/stk/stk_simd/stk_simd_view/simd_view.hpp
M	packages/stk/stk_simd/unit_tests/UnitTestStkSimdMathDouble.cpp
M	packages/stk/stk_simd/unit_tests/UnitTestStkSimdView.cpp
M	packages/stk/stk_simd/unit_tests/UnitTestStkSimdView2d.cpp
M	packages/stk/stk_simd/unit_tests/UnitTestStkSimdView3d.cpp
M	packages/stk/stk_tools/Jamfile
A	packages/stk/stk_tools/stk_tools/mesh_tools/FixNodeSharingViaSearch.cpp
A	packages/stk/stk_tools/stk_tools/mesh_tools/FixNodeSharingViaSearch.hpp
M	packages/stk/stk_topology/Jamfile
M	packages/stk/stk_topology/stk_topology/topology_detail/meta_functions.hpp
A	packages/stk/stk_topology/stk_topology/topology_detail/topology_data.cpp
M	packages/stk/stk_topology/stk_topology/topology_detail/topology_data.hpp
M	packages/stk/stk_topology/stk_topology/topology_type.hpp
M	packages/stk/stk_transfer/Jamfile
M	packages/stk/stk_unit_test_utils/FaceTestingUtils.cpp
M	packages/stk/stk_unit_test_utils/Jamfile
M	packages/stk/stk_unit_test_utils/MeshFileFixture.hpp
M	packages/stk/stk_unit_test_utils/MeshFixture.hpp
M	packages/stk/stk_unit_test_utils/unit_main_lib/UnitTestMain.cpp
M	packages/stk/stk_unit_tests/CMakeLists.txt
M	packages/stk/stk_unit_tests/cmake/Dependencies.cmake
M	packages/stk/stk_unit_tests/stk_mesh/UnitTestBucketRepository.cpp
M	packages/stk/stk_unit_tests/stk_mesh/UnitTestField.cpp
M	packages/stk/stk_unit_tests/stk_mesh/UnitTestFieldImpl.cpp
M	packages/stk/stk_unit_tests/stk_mesh/UnitTestModificationEnd.hpp
M	packages/stk/stk_unit_tests/stk_mesh/UnitTestPartitions.cpp
M	packages/stk/stk_unit_tests/stk_mesh/face_creation/element_graph/BulkDataElementGraphTester.hpp
M	packages/stk/stk_unit_tests/stk_ngp/KokkosBulkDataBucketCentroidCalculation.cpp
M	packages/stk/stk_unit_tests/stk_ngp/KokkosCentroidCalculation.cpp
M	packages/stk/stk_unit_tests/stk_ngp/KokkosCentroidCalculation.hpp
M	packages/stk/stk_unit_tests/stk_ngp/KokkosNodalVolume.cpp
M	packages/stk/stk_unit_tests/stk_ngp/NgpFieldPerformanceTest.cpp
M	packages/stk/stk_unit_tests/stk_ngp/NpgMultistateFieldTest.cpp
M	packages/stk/stk_unit_tests/stk_ngp/howToNgp.cpp
A	packages/stk/stk_unit_tests/stk_ngp/ngpFieldTest.cpp
A	packages/stk/stk_unit_tests/stk_ngp_test/utest_KokkosConfigInfo.cpp
A	packages/stk/stk_unit_tests/stk_ngp_test/utest_NgpTestAPI.cpp
A	packages/stk/stk_unit_tests/stk_ngp_test/utest_NgpTestMacros.cpp
A	packages/stk/stk_unit_tests/stk_ngp_test/utest_Reporter.cpp
M	packages/stk/stk_unit_tests/stk_tools/createMesh.cpp
M	packages/stk/stk_unit_tests/stk_topology/unit_test_hex.cpp
M	packages/stk/stk_unit_tests/stk_topology/unit_test_validate_topology_data.cpp
M	packages/stk/stk_unit_tests/stk_util/environment/UnitTestThrowMacros.cpp
M	packages/stk/stk_usecases/mesh/MeshUseCase_3.cpp
M	packages/stk/stk_usecases/mesh/UseCase_ChangeOwner.cpp
M	packages/stk/stk_util/Jamfile
M	packages/stk/stk_util/stk_util/diag/Option.cpp
M	packages/stk/stk_util/stk_util/diag/Option.hpp
R099	packages/stk/stk_util/stk_util/util/Resource2.cpp	packages/stk/stk_util/stk_util/diag/Resource2.cpp
R100	packages/stk/stk_util/stk_util/util/Resource2.h	packages/stk/stk_util/stk_util/diag/Resource2.h
M	packages/stk/stk_util/stk_util/diag/WriterParser.cpp
M	packages/stk/stk_util/stk_util/parallel/CommBufferV.hpp
M	packages/stk/stk_util/stk_util/parallel/CommNeighbors.cpp
M	packages/stk/stk_util/stk_util/parallel/mpi_filebuf.cpp
M	packages/stk/stk_util/stk_util/util/StkNgpVector.hpp

Current Status on CDash

The status of the Panzer build in this clang-5.0.1 build over the last few days can be see in this query.

Steps to Reproduce

One should be able to reproduce this failure on any CEE RHEL6 machine as described in:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

More specifically, the commands given for the system are provided at:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment

The exact commands to reproduce the build failures should be:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh \
    cee-rhel6_clang-5.0.1_openmpi-1.10.2_serial_static_opt

$ cmake \
  -GNinja \
  -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
  -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Panzer=ON \
  $TRILINOS_DIR

$ ninja -j16