Stop using MPI_Reduce_scatter (in Epetra and elsewhere)
Created by: mhoemmen
@trilinos/xpetra @trilinos/muelu @trilinos/zoltan @trilinos/zoltan2
At least two platforms report performance and/or memory issues with MPI_Reduce_Scatter
at scale. (See Bugzilla Bug 6336, and my Tpetra comments in 09 Jan 2012.) The easiest work-around is to replace calls to that function with MPI_Reduce
followed by MPI_Scatter
. Tpetra has done this for a few years already, for performance reasons. It looks like there used to be an MPICH (correctness) bug in MPI_Reduce_Scatter
, since Epetra and Zoltan both have work-arounds. Zoltan always just calls MPI_Reduce
followed by MPI_Scatter
:
Epetra has an option to do the right thing, depending on the REDUCE_SCATTER_BUG
macro (see line 570 of epetra/src/Epetra_MpiDistributor.cpp):
It would make sense to elevate that macro to a CMake option, if it is not one already.
This is relevant to the Xpetra and MueLu developers, in case they are using the Epetra back-end.
@pwxy @spdomin