Add support for 'release-debug' build for ATDM Trilinos builds and use to avoid timeouts
*Created by: bartlettroscoe*
CC: @fryeguy52, @mhoemmen, @rppawlo, @bathmatt, @micahahoward, @trilinos/kokkos, @trilinos/kokkos-kernels
## Next Action Status
ATDM Trilinos scripts now support a `release-debug` build type and this has been used in new `release-debug` builds on 'waterman'. Converting `debug` builds to `release-debug` builds on other platforms will be done in follow-on issues ...
## Description
Currently, the ATDM Trilinos builds support `debug` and an `opt` build. The `debug` build uses `CMAKE_BUILD_TYPE=DEBUG` (with `-O0`) and enables runtime debug-mode checking while the `opt` build uses `CMAKE_BUILD_TYPE=RELEASE` (with `-O3`) and no runtime debug-mode checking. The problem with this approach is that some of the Trilinos tests (especially many of the Kokkos and KokkosKernels tests) run many times slower wtih `-O0` than with `-O3`. This has caused many tests to timeout at 10 minutes in `debug` builds that finish is well under 10 minutes in `opt` builds (e.g. #2964, #2921, #2461).
A solution that we discussed was to change most `debug` builds into `release-debug` builds that will set `CMAKE_BUILD_TYPE=RELEASE` (with `-O3`) but enable runtime debug-mode checking.
## Proposed solution
The idea would be to add a new `release-debug` keyword that matches before `opt` or `debug` which will set `ATDM_CONFIG_BUILD_TYPE=RELEASE_DEBUG` and then update the file `ATDMDevEnvSettings.cmake` accordingly. That will be easy. The harder part will be updating the tweaks `*.cmake` files and all of the Jenkins jobs to accommodate the name change. NOTE: calling this `release-debug` as apposed to `opt-debug` hopefully might be more clear.
We can still leave some full `debug` builds to help support full GDB debugging by the ATDM APPs teams but they should be sparing (because we are constantly dealing with timeouts with full debug builds).
## Tasks:
* ???
issue