Tpetra::MultiVector: Fix "DualView" (better: "dual view") semantics for subviews
Created by: mhoemmen
Currently, all subviews (row or column) of a Tpetra::MultiVector share the same modify() / sync() flags. Consider the following case: X is the original MultiVector, and Y and Z are disjoint subviews of X. If Y syncs, currently that will cause all of X to sync, and thus force a (possibly incorrect) sync of Z as well.
The reason for this is that Tpetra::MultiVector lets Kokkos::DualView own the modify() flags. To fix this, Tpetra needs to control those flags and own sync'ing.
Fixing this would also give Tpetra the flexibility to do on-demand allocation, and even deallocate device / HBM memory when not in use (since it may be a scarce resource). However, we would need to get rid of the Tpetra::MultiVector::getDualView method (and thus no longer expose the Kokkos::DualView).