Tpetra::Map::replaceCommWithSubset: Optimize for MPI_COMM_SELF case
Created by: mhoemmen
@trilinos/tpetra @ambrad
This relates to our discussion in #558 (closed). To summarize: Ifpack2::AdditiveSchwarz is supposed to take responsibility for the initial Import. That Import results in an input (Multi)Vector for the subdomain solver. The subdomain solver thus should not have to do another Import. This implies that the subdomain solver's column Map should be the same as its domain Map.
This introduces a minor problem, namely that domain and range Maps are supposed to be one to one (nonoverlapping), yet the subdomain solver's "domain Map," according to the criterion above, must be overlapping over the original matrix's communicator. The right way to fix this would be to restrict the subdomain solver's domain Map, so that its communicator only has one process. This correctly reflects that the subdomain solver only works on a single process. (A full generalization of AdditiveSchwarz that allows multiprocess subdomain solvers, would restrict the domain Map to the subdomain's subcommunicator.) As a result, after AdditiveSchwarz does its Import, what really should happen is that the resulting (Multi)Vector gets restricted to the subdomain's subcommunicator (which is MPI_COMM_SELF or its equivalent, in this case). This is a no-communication operation and should not require copying any data.
Tpetra's way to implement that restriction, is to call Tpetra::Map's replaceCommWithSubset method. It takes the subcommunicator as input, and returns a new Map with that subcommunicator as its communicator. replaceCommWithSubset has collective semantics over the subcommunicator (its input). That's good in this case, because we're giving it MPI_COMM_SELF
. However, the implementation currently builds a new Map (e.g., new FixedHashTable), whether or not it needs to. In particular, if the input subcomm is MPI_COMM_SELF
, the output Map should just be able to get all its data directly from the input Map, without any error checking. For example, if the input Map is noncontiguous, the output Map could just use its FixedHashTable directly.