Belos,Tpetra: Fix #4626, #4639, #4633

Created by: mhoemmen

@trilinos/belos @trilinos/tpetra @cgcgcg @vbrunini

Description

This PR supersedes #4640.

• Fix #4639 (closed) (an actual correctness bug in Tpetra::MultiVector::reduce that hindered solving #4626 (closed))
• Fix #4633 (closed) (a strange thing in Tpetra::MultiVector that could lead to bugs, especially with FEMultiVector)
• Fix #4626 (closed) (a Belos performance issue, especially on GPUs)

Related Issues

• Closes #4626 (closed), #4639 (closed), #4633 (closed)