Tpetra::MultiVector: Fix #4870 (prefer running norms on device)

James Willenbringrequested to merge
Fix-4870-4 into develop

Created by: mhoemmen

@trilinos/tpetra @trilinos/stokhos @vbrunini @etphipp

Superseded by https://github.com/trilinos/Trilinos/pull/4899. Supersedes PR #4881 and PR #4871.

Description

  • Prefer running Tpetra::MultiVector norms on device, even if it is currently sync'd to host, unless the vector has too few local rows.
  • Factor out the implementation of Tpetra::MultiVector::norm* into a separate function, Tpetra::Details::normImpl. I do ETI for that function, when ETI is enabled. This required adding new ETI macros for iterating over the (Scalar, Node) type combination.
  • Fix Stokhos to work around lack of ETI for Tpetra::Details::normImpl.

To elaborate the last point: Tpetra's ETI macro for Tpetra::Details::normImpl only takes 2 template parameters: Scalar and Node. This meant that I couldn't use Stokhos' ETI system directly (see TPETRA_ETI_CLASSES in stokhos/src/CMakeLists.txt). I had to take different approaches for PCE types vs. MP::Vector types. My solution for PCE types looks more permanent (read "is less of a hack") than my solution for MP::Vector types; it would be great if @etphipp could comment.

Motivation and Context

  • SPARC performance
  • Build time

Related Issues

Merge request reports