GitLab

Kokkos-Kernels was moved from its previous location tpetra/kernels
to kokkos-kernels and now is its own package. Testing was performed
on multiple platforms with gcc, intel and cuda, also debug mode.
As far as I can tell the have the same errors as on current develop.
I.e. hopefully nothing related to this switch. Testing was done with
the samplescripts/Sandia-Sems/configure-testbeds-jenkins-all script.

   Lawyer stuff. Making copyright headers consistent etc.

This reverts commit a1507467, reversing
changes made to 518fa092.

This generally replaces device_type with execution_space
and DeviceType with ExecutionSpace except for some places where
the new compound device_type should be used.

I moved the KokkosLinAlg subpackage of Kokkos into Tpetra, and renamed
it "TpetraKernels".  This move will facilitate moving Kokkos from the
Trilinos repository into its own Github repository.  I renamed it from
"LinAlg" because all of Tpetra implements "linear algebra."  The name
"Kernels" better suggests the contents: single-MPI-process,
thread-parallel computational kernels.

I also fixed downstream packages that depended on KokkosLinAlg.  This
includes both the package name (KokkosLinAlg -> TpetraKernels) and
various library names.

This commit mainly affects Kokkos and Tpetra.  I also had to make
minor changes to Isorropia, ShyLu, Stokhos, TrilinosCouplings, and
Xpetra.  In particular, Xpetra's "FakeKokkos" (KokkosClassic
replacement headers for when building with Tpetra disabled) caused
some bizarre build errors until I finally figured out that Xpetra was
using "Xpetra_ENABLE_Kokkos" to key on whether to use FakeKokkos'
headers.  This manifested as Tpetra files not getting the contents of
TpetraClassic files (like Kokkos_ConfigDefs.hpp), but only when
building in Xpetra.  I fixed this by making Xpetra key on
Xpetra_ENABLE_Tpetra, since the former KokkosClassic now lives in
Tpetra.

NOTE: The Kokkos refactor version of MueLu does not currently build
with ETI enabled, because of some missing explicit instantiations.
Here is the error message I get:

Linking CXX executable MueLu_Challenge_XML.exe
../../../src/libmuelu.so.11.13: undefined reference to `Ifpack2::AdditiveSchwarz<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >, Ifpack2::Preconditioner<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >::AdditiveSchwarz(Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > const> const&, int)'
../../../src/libmuelu.so.11.13: undefined reference to `Ifpack2::Details::OneLevelFactory<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >::create(std::string const&, Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > const> const&) const'
../../../src/libmuelu.so.11.13: undefined reference to `Ifpack2::Chebyshev<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >::getLambdaMaxForApply() const'
../../../src/libmuelu.so.11.13: undefined reference to `Ifpack2::Hiptmair<Tpetra::CrsMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >::Hiptmair(Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > const> const&, Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > const> const&, Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > const> const&)'
../../../src/libmuelu.so.11.13: undefined reference to `Ifpack2::Krylov<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > >::Krylov(Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> > const> const&)'
collect2: error: ld returned 1 exit status

Note that KokkosDeviceWrapperNode<Kokkos::OpenMP> is NOT the default
Node type in this build; the default Node type is
KokkosDeviceWrapperNode<Kokkos::Serial>.  As far as I can tell,
Ifpack2 only does ETI for the default Node type.  It could be that
MueLu is explicitly requesting ETI for a NON-default Node type, or it
could be that I messed up somewhere with the macros.  I'll have to
work a little bit harder at this, but I think it's OK to finish the
subpackage migration process first.

The FENL example depends on KokkosCore, KokkosContainers, and
KokkosLinAlg.  No other examples in the KokkosExample subpackage
depend on KokkosLinAlg.  Since we want to move KokkosLinAlg out of
Kokkos and into Tpetra, as a first step, I have moved the FENL example
out of KokkosExample and into KokkosLinAlg.

I also changed KokkosExample's dependencies so that it no longer
depends on KokkosLinAlg.

1)  Kokkos_Macros.hpp    now processes only macros and has merged in the compiler detection macros.
2)  Kokkos_Core_fwd.hpp  now has all forward class declarations that were in Kokkos_Macros.hpp.
3)  Kokkos_Core.hpp      now includes all enabled memory space, execution space, view, and atomic headers.

Kokkos: Fix example/fixture incorrect array bounds check and increase fixture's upper bound from 30 to 62 processor neighbors.

Kokkos: Robustly handle an attempt to decrement tracked memory after memory tracking singleton has been destroyed by issuing a warning message.  Cannot throw an exception since Teuchos unit test framework relies upon construction of static objects with unmanaged destruction sequences.

Kokkos: Example implicit nonlinear finite element driver (FENL) with residual and jacobian assembly via atomic-add or gather-fill.  Using kokkos/linalg CrsMatrix and matrix-vector multiple.

Upgrade use case to use containers/, linalg/, example/fixture, and be fully thread parallel.
New thread parallel algorithm to construct sparse matrix graph from element->node connectivity.

Kokkos: Add kokkos/example dependency on kokkos/linalg in preparation of new finite element nonlinear proxy app.

A templated function interface can be disable through 'enable_if'.
A non-templated function interface can be disabled by substituting
its input or return type with a unusable error-tag class based upon
a compile time condition.  For example, a View has an device shared
memory constructor if it is unmanaged.  If the View is managed then
the device argument type is substituted for the class
'ViewError::device_shmem_constructor_requires_unmanaged'.

Mapping of nodal coordinates can be skewed.
Linear or quadratic elements.
Lumping of element integral to nodes via atomics or second pass gather-sum functor.
Integral of function has analytic answer for verification.

TODO: Clean and add extensive comments.
TODO: Build with cmake and add to cmake test.

Fix create_mirror_view to allow mirroring views with 'const' data type.

Kokkos: Add 'kokkos/example/' subpackage with thread-scalable generation of a box element mesh fixture.

Kokkos: Add the 'ALL' argument for a subview to specify an entire range of a dimension.
        This will lead to consistency of subviews when prior subview methods are removed.