Tpetra::Distributor: Fix #3580

Created by: mhoemmen

@trilinos/tpetra @jjellio

Description

Fix #3580, by making all instances of the "slow path" in Distributor use a buffer long enough for all the messages, not just one at a time. Ignore the "Send type" and "Barrier between..." options, and always use MPI_Isend for sends.

• Closes #3580

How Has This Been Tested?

Mac, Clang, OpenMPI.