Inconsistent default execution spaces between kokkos and tpetra
Created by: rppawlo
@trilinos/tpetra
We are having some failures in panzer that have been tracked back to inconsistent default execution spaces between tpetra and kokkos. Panzer uses the tpetra default default execution space, but panzer depends on intrepid2 which uses the default execution space from kokkos.
When a user does not specify the kokkos and tpetra execution spaces explicitly, there is a high probability that the execution spaces are mismatched. The Threads TPL usually is automatically enabled due to ctest examining the environment (if not set explicitly in configure). Kokkos will choose a pthread execution space if TPL_ENABLE_Pthread is on. Tpetra, though, sees both a serial and the pthread execution spaces available, but chooses serial instead of pthread:
# Don't turn on both Serial and Threads by default. Prefer Serial,
# because Threads is a last resort. Of course, users may still
# enable Threads explicitly, by setting TPETRA_INST_PTHREAD=ON.
IF (HAVE_TPETRA_INST_SERIAL_DEFAULT AND HAVE_TPETRA_INST_PTHREAD_DEFAULT)
GLOBAL_SET(HAVE_TPETRA_INST_PTHREAD_DEFAULT OFF)
ENDIF ()
So what happens is that in trilinos ci testing, Panzer tests assume serial space from Tpetra default, but the intrepid2 kernels default to Threads and the tests fail because the threads space is not initialized by panzer tests.
All currently failing panzer tests are due to this. I tried switching panzer over to use the kokkos default node type, but then the tests fail to build because tpetra does not build the pthread node under ETI.
I know we can fix this by going into all our trilinos nightly testing configure scripts and explicitly specifying a consistent execution space, but I believe this is going to be an issue for Trilinos users in general. Many don't use kokkos and don't know to set the execution spaces and make sure that they are consistent across packages. Can we set the defaults consistently across packages (for Tpetra to directly key off of Kokkos), yet still allow users to override the default execution spaces in each package.
By the way - we have cases where application codes need both serial and pthread spaces. DTK only cares about MPI communication and is therefore hard coded to a Serial execution space in Tpetra (to avoid having to push templates through the entire stack). Albany uses DTK for multiphysics coupling but is also Kokkos aware and wants to use kokkos in a parallel execution space internally.