Tpetra &/or Teuchos: Implement generic MPI_Iallreduce interface
Created by: mhoemmen
@trilinos/tpetra @trilinos/teuchos
Story: #748 (closed) Blocks: #753 (closed).
Implement a generic MPI_Iallreduce interface, analogous to the other MPI interfaces in Teuchos. Put it in Tpetra::Details for now. It should take a Kokkos::View as its input/output argument, and it should look like this:
template<class DataType, class DeviceType>
Teuchos::RCP<Teuchos::CommRequest<int> >
iallreduce (const Kokkos::View<const DataType*, DeviceType>& sendbuf,
const Kokkos::View<DataType*, DeviceType>& recvbuf,
const Teuchos::EReductionType reductType,
const Teuchos::Comm<int>& comm);
Notes:
Use Teuchos::Details::MpiTypeTraits (available due to #804 (closed) fix) to go from DataType to its corresponding MPI_Datatype
. We may need to support Teuchos' generic mechanism for packages like Stokhos. Consider adding Stokhos specializations rather than doing that, because the generic mechanism isn't even correct for reductions with large arrays.
Check MPI_VERSION >= 3
before using MPI_Iallreduce
. If MPI_VERSION < 3
, make the CommRequest call MPI_Allreduce
on wait. Also, if the RCP's reference count goes to zero before wait gets called, do MPI_Cancel
on the MPI_Request
. (Teuchos::MpiCommRequestBase
already calls MPI_Cancel
in its destructor, if the request is still active at that point. See teuchos/comm/src/Teuchos_DefaultMpiComm.hpp.)