MueLu research and example code build failures for new CUDA ATDM build on hansen/shiller

Created by: bartlettroscoe

CC: @trilinos/muelu, @fryeguy52

Next Action Status:

Commit eee871d8 which sets MueLu_ENABLE_Epertra=OFF and fixes the build failures.

Description

The MueLu package shows build falures for the CUDA ATDM builds today on hansen shown at:

https://testing.sandia.gov/cdash/index.php?project=Trilinos&filtercount=1&showfilters=1&field1=buildname&compare1=63&value1=-atdm-

for the builds:

Trilinos-atdm-hansen-shiller-cuda-debug: https://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=3412693
Trilinos-atdm-hansen-shiller-cuda-opt: https://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=3412702

The build failures for example at:

https://testing.sandia.gov/cdash/viewBuildError.php?buildid=3412805

all show undefined reference link failues like:

CMakeFiles/MueLu_ImportTest.dir/Import.cpp.o: In function `int main_<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Teuchos::CommandLineProcessor&, Xpetra::UnderlyingLib, int, char**)':
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:153: undefined reference to `Tpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Map(unsigned long, Teuchos::ArrayView<int const> const&, int, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&)'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:154: undefined reference to `Tpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Map(unsigned long, Teuchos::ArrayView<int const> const&, int, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&)'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:156: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::MultiVector(Teuchos::RCP<Tpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const> const&, unsigned long, bool)'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:157: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::getDataNonConst(unsigned long)'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:167: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::MultiVector(Teuchos::RCP<Tpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const> const&, unsigned long, bool)'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:169: undefined reference to `Tpetra::Export<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Export(Teuchos::RCP<Tpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const> const&, Teuchos::RCP<Tpetra::Map<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const> const&)'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:170: undefined reference to `Tpetra::DistObject<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::doExport(Tpetra::SrcDistObject const&, Tpetra::Export<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, Tpetra::CombineMode)'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:171: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::getData(unsigned long) const'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:169: undefined reference to `Tpetra::Export<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::~Export()'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:167: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::~MultiVector()'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:156: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::~MultiVector()'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:169: undefined reference to `Tpetra::Export<int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::~Export()'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:167: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::~MultiVector()'
/home/jenkins/hansen/workspace/Trilinos-atdm-hansen-shiller-cuda-debug/SRC_AND_BUILD/Trilinos/packages/muelu/research/luc/region_algorithms/Import.cpp:156: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::~MultiVector()'
collect2: error: ld returned 1 exit status

but each executable has a slightly different set of link failures.

It looks like some explicit template instantiations are missing?

Steps to Reproduce:

The instructions to reproduce these build failures can be found starting at:

https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/ATDM+Builds+of+Trilinos

and clicking "Reproducing ATDM builds locally" which takes you to:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

Basically, on hansen or shiller, you just clone the Trilinos repo (with location depicted as $TRILINOS_DIR below), get on the develop branch. Then create a build directory and do the configure and build as:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-opt

$ cmake \
  -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
  -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_MueLu=ON \
  $TRILINOS_DIR

$ make -j16