Tpetra::MultiVector: No CombineModes should do atomic updates with OpenMP if only 1 thread
Created by: mhoemmen
@trilinos/tpetra @crtrott @vbrunini @jjellio
Tpetra::MultiVector::unpackAndCombineNew currently uses atomic updates for all CombineModes (but see #4417 (closed)), except with Kokkos::Serial. This means that MultiVector will always use atomic updates with Kokkos::OpenMP, even if the number of threads is one. As a result, MultiVector penalizes users for enabling OpenMP. MultiVector should instead check if the number of threads is one, and not use atomic updates in that case.
Fixing this will require refactoring MultiVector's "Op"s that implement the CombineModes. Right now, the Ops are responsible for deciding whether to do atomic updates. To fix this issue, we could just change the Ops so that they have two methods, one for atomic update and one for nonatomic, and let the calling functors decide which to use (based on a run-time switch). (That would also let us fix #962 later if we wanted; the run-time switch would be "using only one thread, or no duplicate LIDs.") This would also let us remove the "ExecutionSpace" template parameter from the Ops (which currently only is a means to select whether to use atomic updates), so it should prove a net simplification.
Related Issues
- Related to #962, #4417 (closed)