Tpetra compile error build with serial build + complex types with gcc/7.2 and gcc/7.3 (#4095) · Issues · James Willenbring / Trilinos

Tpetra compile error build with serial build + complex types with gcc/7.2 and gcc/7.3

Created by: ndellingwood

This error reported by @hkthorn

Tpetra fails to compile with complex types with gcc/7.2 and gcc/7.3 and Tpetra_INST_SERIAL:BOOL=ON; my configuration is posted at end of this issue. Error occurs in Tpetra's AbsMaxOp functor's operator () at line 755 when calling Kokkos::Impl::atomic_fetch_oper.

Not sure if this is a Kokkos issue or something issue with execution space defaults.

I explicitly enabled only the Tpetra Serial backend and no Kokkos configure options were used on the cmake line, but the compilation error shows Kokkos::OpenMP as the execution space expected by AbsMaxOp which seems related to the error. Shouldn't the execution space in Tpetra and Kokkos agree if I only set Tpetra_INST_SERIAL:BOOL=ON and no Kokkos options?

Looking at code surrounding the error line 755 in this code block, the explicit specialization of AbsMaxOp within the KOKKOS_ENABLE_SERIAL guards is not called. Going back to previous question about default Kokkos execution space settings when Tpetra's execution space is explicitly set, I would expect that KOKKOS_ENABLE_SERIAL=ON be true if I only set Tpetra_INST_SERIAL:BOOL=ON, though this may be unrelated.

@hkthorn mentioned this configuration worked for her in the past. A Gblame on the error line shows it was modified by PR #3999, adding @mhoemmen and @rppawlo.

Edit: Strike out irrelevant info due to careless copy/paste of configuration options I made.

@trilinos/tpetra @trilinos/kokkos

Error message:

/home/ndellin/trilinos-ssh/Trilinos/packages/tpetra/core/src/kokkos_refactor/Tpetra_KokkosRefactor_Details_MultiVectorDistObjectKernels.hpp:755:39: error: no matching function for call to ‘atomic_fetch_oper(Tpetra::KokkosRefactor::Details::AbsMaxOper<Kokkos::complex<double>, Kokkos::complex<double> >, Kokkos::complex<double>*, const Kokkos::complex<double>&)’
       Kokkos::Impl::atomic_fetch_oper (AbsMaxOper<Scalar,Scalar>(), &dest, src); 

In file included from /home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/Kokkos_Atomic.hpp:288:0,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/impl/Kokkos_Atomic_View.hpp:47,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/impl/Kokkos_ViewMapping.hpp:56,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/Kokkos_View.hpp:451,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/Kokkos_Parallel.hpp:52,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/Kokkos_Serial.hpp:55,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/Kokkos_Core.hpp:53,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/teuchos/kokkoscompat/src/KokkosCompat_ClassicNodeAPI_Wrapper.hpp:5,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/tpetra/core/src/Tpetra_Details_DefaultTypes.hpp:47,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/tpetra/core/src/Tpetra_ConfigDefs.hpp:45,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/tpetra/core/src/Tpetra_Core.hpp:54,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/tpetra/core/test/Tpetra_TestingUtilities.hpp:55,
                 from /home/ndellin/trilinos-ssh/Trilinos/packages/tpetra/core/test/Block/BlockMultiVector.cpp:44:
/home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/impl/Kokkos_Atomic_Generic.hpp:155:3: note: candidate: template<class Oper, class T> T Kokkos::Impl::atomic_fetch_oper(const Oper&, volatile T*, typename Kokkos::Impl::enable_if<((sizeof (T) != sizeof (int)) && (sizeof (T) == sizeof (long long unsigned int))), const T>::type) 
 T atomic_fetch_oper( const Oper& op, volatile T * const dest ,
   ^~~~~~~~~~~~~~~~~
/home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/impl/Kokkos_Atomic_Generic.hpp:155:3: note:   template argument deduction/substitution failed:
/home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/impl/Kokkos_Atomic_Generic.hpp: In substitution of ‘template<class Oper, class T> T Kokkos::Impl::atomic_fetch_oper(const Oper&, volatile T*, typename Kokkos::Impl::enable_if<((sizeof (T) != sizeof (int)) && (sizeof (T) == sizeof (long long unsigned int))), const T>::type) [with Oper = Tpetra::KokkosRefactor::Details::AbsMaxOper<Kokkos::complex<double>, Kokkos::complex<double> >; T = Kokkos::complex<double>]’:
/home/ndellin/trilinos-ssh/Trilinos/packages/tpetra/core/src/kokkos_refactor/Tpetra_KokkosRefactor_Details_MultiVectorDistObjectKernels.hpp:755:39:   required from ‘void Tpetra::KokkosRefactor::Details::AbsMaxOp<ExecutionSpace>::operator()(Scalar&, const Scalar&) const [with Scalar = Kokkos::complex<double>; ExecutionSpace = Kokkos::OpenMP]’
/home/ndellin/trilinos-ssh/Trilinos/packages/tpetra/core/src/kokkos_refactor/Tpetra_KokkosRefactor_Details_MultiVectorDistObjectKernels.hpp:915:11:   required from ‘void Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumnWithBoundsCheck<ExecutionSpace, DstView, SrcView, IdxView, Op, SizeType>::operator()(const size_type&, Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumnWithBoundsCheck<ExecutionSpace, DstView, SrcView, IdxView, Op, SizeType>::value_type&) const [with ExecutionSpace = Kokkos::OpenMP; DstView = Kokkos::View<Kokkos::complex<double>**, Kokkos::LayoutLeft, Kokkos::OpenMP>; SrcView = Kokkos::View<Kokkos::complex<double>*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> >; IdxView = Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> >; Op = Tpetra::KokkosRefactor::Details::AbsMaxOp<Kokkos::OpenMP>; SizeType = long unsigned int; Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumnWithBoundsCheck<ExecutionSpace, DstView, SrcView, IdxView, Op, SizeType>::size_type = long unsigned int; Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumnWithBoundsCheck<ExecutionSpace, DstView, SrcView, IdxView, Op, SizeType>::value_type = int]’
/home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/OpenMP/Kokkos_OpenMP_Parallel.hpp:327:16:   required from ‘static typename std::enable_if<std::is_same<_Dummy, void>::value>::type Kokkos::Impl::ParallelReduce<FunctorType, Kokkos::RangePolicy<Traits ...>, ReducerType, Kokkos::OpenMP>::exec_range(const FunctorType&, Kokkos::Impl::ParallelReduce<FunctorType, Kokkos::RangePolicy<Traits ...>, ReducerType, Kokkos::OpenMP>::Member, Kokkos::Impl::ParallelReduce<FunctorType, Kokkos::RangePolicy<Traits ...>, ReducerType, Kokkos::OpenMP>::Member, Kokkos::Impl::ParallelReduce<FunctorType, Kokkos::RangePolicy<Traits ...>, ReducerType, Kokkos::OpenMP>::reference_type) [with TagType = void; FunctorType = Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumnWithBoundsCheck<Kokkos::OpenMP, Kokkos::View<Kokkos::complex<double>**, Kokkos::LayoutLeft, Kokkos::OpenMP>, Kokkos::View<Kokkos::complex<double>*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> >, Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> >, Tpetra::KokkosRefactor::Details::AbsMaxOp<Kokkos::OpenMP>, long unsigned int>; ReducerType = Kokkos::InvalidType; Traits = {Kokkos::OpenMP, long unsigned int}; typename std::enable_if<std::is_same<_Dummy, void>::value>::type = void; Kokkos::Impl::ParallelReduce<FunctorType, Kokkos::RangePolicy<Traits ...>, ReducerType, Kokkos::OpenMP>::Member = long unsigned int; Kokkos::Impl::ParallelReduce<FunctorType, Kokkos::RangePolicy<Traits ...>, ReducerType, Kokkos::OpenMP>::reference_type = int&]’
/home/ndellin/trilinos-ssh/Trilinos/packages/kokkos/core/src/OpenMP/Kokkos_OpenMP_Parallel.hpp:391:34:   required from ‘void Kokkos::Impl::ParallelReduce<FunctorType, Kokkos::RangePolicy<Traits ...>, ReducerType, Kokkos::OpenMP>::execute() const [with FunctorType = Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumnWithBoundsCheck<Kokkos::OpenMP, Kokkos::View<Kokkos::complex<double>**, Kokkos::LayoutLeft, Kokkos::OpenMP>, Kokkos::View<Kokkos::complex<double>*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> >, Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> >, Tpetra::KokkosRefactor::Details::AbsMaxOp<Kokkos::OpenMP>, long unsigned int>; ReducerType = Kokkos::InvalidType; Traits = {Kokkos::OpenMP, long unsigned int}]’

Configuration:

module load sems-gcc/7.2.0 sems-cmake/3.12.2

rm -rf CMake*

cmake \
-DCMAKE_C_COMPILER=`which gcc` \
-DCMAKE_CXX_COMPILER=`which g++` \
-DCMAKE_Fortran_COMPILER=`which gfortran` \
-DCMAKE_CXX_FLAGS="-Wall -O3 -DNDEBUG" \
-DCMAKE_C_FLAGS="-Wall -O3 -DNDEBUG" \
-D Trilinos_ENABLE_Belos:BOOL=ON \
-D Trilinos_ENABLE_Anasazi:BOOL=ON \
-D Trilinos_ENABLE_AztecOO:BOOL=ON \
-D Trilinos_ENABLE_Teuchos:BOOL=ON \
-D Trilinos_ENABLE_Ifpack:BOOL=ON \
-D Trilinos_ENABLE_Epetra:BOOL=ON \
-D Trilinos_ENABLE_EpetraExt:BOOL=ON \
-D Trilinos_ENABLE_Triutils:BOOL=ON \
-D Trilinos_ENABLE_Tpetra:BOOL=ON \
-D Tpetra_INST_SERIAL:BOOL=ON \
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
-D Teuchos_ENABLE_COMPLEX:BOOL=ON \
-D Trilinos_ENABLE_COMPLEX:BOOL=ON \
-D Trilinos_ENABLE_OpenMP:BOOL=ON \
-D CMAKE_BUILD_TYPE:STRING=DEBUG \
-D CMAKE_VERBOSE_MAKEFILE:BOOL=ON \
-D Trilinos_ENABLE_EXAMPLES:BOOL=ON \
-D Trilinos_ENABLE_TESTS:BOOL=ON \
-DTPL_ENABLE_BLAS=ON \
-DTPL_ENABLE_LAPACK=ON \
../../