Tpetra: Attempted fix for #3289

Created by: mhoemmen

@trilinos/tpetra

Description

Fix #1454 (closed) test as follows:

1. Don't give UVM buffers to Distributor for MPI communication
2. Use Tpetra::Details::Behavior to decide whether to give CudaSpace buffers to Distributor for MPI communication.

This hopefully will fix #3289 (closed).

Related Issues

• Related to #1454 (closed), #3289 (closed)

How Has This Been Tested?

CUDA 9.2. See commit messages for details.

Checklist

• My commit messages mention the appropriate GitHub issue numbers.
• My code follows the code style of the affected package(s).
• I have read the code contribution guidelines for this project.
• All new and existing tests passed.
• No new compiler warnings were introduced.