Skip to content

Tpetra,Stokhos: Fix #4870 (prefer running MV norms on device)

James Willenbring requested to merge Fix-4870-5 into develop

Created by: mhoemmen

@trilinos/tpetra @trilinos/stokhos @vbrunini @etphipp

Supersedes PR #4881, PR #4871, and PR #4885.

Description

  1. Make Tpetra::MultiVector prefer computing the local (to each MPI process) parts of norms (norm1, norm2, normInf) on device. Do so even if the MultiVector is currently sync'd to host, unless the MultiVector has too few local rows.
  2. Factor out the implementation of Tpetra::MultiVector::norm* into a separate function, Tpetra::Details::normImpl. Do ETI for that function, when ETI is enabled.
  3. Make Stokhos do ETI for Tpetra::Details::normImpl, for its UQ::PCE and MP::Vector Scalar types.

To elaborate the last point: Tpetra's ETI macro for Tpetra::Details::normImpl only takes 2 template parameters: Scalar and Node. This meant that I couldn't use Stokhos' ETI system directly (see TPETRA_ETI_CLASSES in stokhos/src/CMakeLists.txt). I had to take different approaches for PCE types vs. MP::Vector types. My solution for PCE types looks more permanent (read "is less of a hack") than my solution for MP::Vector types; it would be great if @etphipp could comment.

Motivation and Context

  • SPARC performance
  • Build time

Related Issues

Merge request reports