Belos: Fix #4626 (Tpetra specialization of MultiVecTraits)

Created by: mhoemmen

@trilinos/belos @trilinos/tpetra @vbrunini

Description

Make the Tpetra specialization of Belos::MultiVecTraits use static CUDA allocations for temporary local MultiVectors (e.g., for Tpetra::MultiVector::multiply results). The fix always creates contiguous allocations, so it does not depend on the fix for #4639 (closed) in PR #4747 (merged into develop yesterday).

This change supersedes PR #4648, which in turn superseded PR #4640.

Motivation and Context

We've seen this improve GPU performance in solves.

Related Issues

• Closes #4626 (closed)
• Related to #4648, #4734, #4640