Tpetra: Replace sync/modify with 'use'
Created by: mhoemmen
@trilinos/tpetra @crtrott
Tpetra::MultiVector currently implements "dual view" semantics through the sync / modify methods. Each takes a space as a template parameter. X.sync<Space>()
ensures that X's data are up-to-date in space Space
, and X.modify<Space>()
marks X's data as modified in that space.
We would like to change this syntax as follows:
X.use (Space1, Space2, ..., SpaceN, ReadAndOrWrite);
where Space1
, ..., SpaceN
are instances of memory spaces (execution space?), and ReadAndOrWrite
is an enum that takes one of the following values: ReadOnly
, WriteOnly
, or ReadWrite
. This is a declarative syntax, vs. the imperative syntax that sync and modify implement. It is only a syntax change; the underlying implementation and semantic possibilities would not change.
The typical use case is to "use" a Tpetra object in one space at a time. For example:
X.use (Cuda (), WriteOnly);
fillOnCudaDevice (X); // write-only device code
X.use (Host (), ReadOnly);
hostOnlyLegacyCode (X); // read-only access on host
X.use (Cuda (), ReadWrite);
modifyOnCudaDevice (X);
The above semantics using the sync / modify syntax look like this:
fillOnCudaDevice (X); // don't sync first
X.sync<Host> ();
hostOnlyLegacyCode (X); // read-only; don't call modify first
X.sync<Cuda> ();
X.modify<Cuda> ();
modifyOnCudaDevice (X);
Calling use
with multiple memory spaces effectively syncs to all of them. This allows (read-only) access to the data in concurrent kernels running from the different spaces.
Furthermore, we would like Tpetra objects' constructors to take optional (memory?) space arguments, to control where Tpetra initially allocates memory. Allocation in at least certain memory spaces would be lazy -- that is, Tpetra would only allocate if using in those memory spaces. For example (hypothetical), we might not like to allocate in HBM memory initially:
Tpetra::Vector<...> X (map, Capacity ()); // don't allocate in HBM initially