Create ATTBDevEnv.cmake module to automatically use loaded env for Advanced Technology Test Bed platforms
Created by: bartlettroscoe
Just like we want to do for SEMS environments (see #158 (closed)), we want to create a file called ATTBDevEnv.cmake (stands for Advanced Technology Test Bed Development Environment) that will define the TriBITS CMake variables to find all of the right libraries and include directories when the ATTB environment is loaded.
The basic idea is that a developer will load a single module like:
$ module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18
and that module will load a bunch of other modules. These modules will define a bunch of environment variables of the form <TPLName>_ROOT
(where <TPLName>
= HDF5
, METIS
, PNETCDF
, etc.). Under these you would find:
${<TPLName>_ROOT}/include/
${<TPLName>_ROOT}/lib/
(or whereever the libraries exist).
In addition, the modules set up the variables pointing to the MPI compiler wrappers like MPICC
, MPICXX
, and MPIFC
.
In addition, the top-level module would define the environment variable ATTB_ENV
which would be set and will be non-empty when the env is loaded (later we can look at the value of ATTB_ENV
to be able to point to different versions of the env). The goal behind the ATTB_ENV is that we would put in automatic logic like:
IF (NOT "${ENV{ATTB_ENV}}" STREQUAL "")
INCLUDE(<some-base>/ATTBDevEnv.cmake)
ENDIF()
The idea would be to have Trilinos (or even TriBITS in some smart way) run this block of code and be able to automatically pick up the dev env and set the TriBITS CMake TPL vars correctly so someone could just load the top-level module and then configure Trilinos with:
cmake -DTPL_ENABLE_MPI=ON -DTPL_ENABLE_Netcdf=ON ... \
-DTrilinos_ENABLE_SECACAS=ON -DTrilinos_ENABLE_Panzer=ON ... \
... \
${TRILINOS_DIR}
and it would automatically know where to find the include dirs and libraries for all of the TPLs enabled (implicitly or explicitly).
What makes this tricky is that the ATTB env is set up to only use static libraries. But they build all of these with -fPIC
so you should still be able to build shared libraries in Trilinos and downstream.
CC list: @rppawlo, @bathmatt, @nmhamster
Next Action Status
Trilinos now automatically detects and sets up for ATTB dev env on hansen
and shiller
. Testing on other machine (see below) ...
Tasks:
- Get configure scripts for Trilinos+Drekar [Done]
- Create ATTBDevEnv.cmake that successfully configures and builds Trilinos+Drekar [Done]
- Set up Trilinos+Drekar to automatically include ATTBDevEnv.cmake on ATTB machine
hansen
[Done] - Get ATTBDevEnv.cmake working on
shiller
[Done] - See about setting policy https://cmake.org/cmake/help/v3.5/policy/CMP0060.html, test on Shiller and Hanssen and require CMake 3.3+ …
- Get working with
OMPI_CXX=nvcc_wrapper
onhansen
andshiller
... - Add
-march=core-avx2
toCMAKE_[C|CXX|Fortran]_FLAGS
to improve vectorization ... - Get ATTBDevEnv.cmake working on
shepard
:- Module
devpack/openmpi/1.10.0/intel/16.1.056/cuda/none
is missing It is missingBLAS_ROOT
andLAPACK_ROOT
...
- Module
- Get ATTBDevEnv.cmake working on
white
... - Get ATTBDevEnv.cmake working on
ride
... - Test full Trilinos+Drekar configure on all ATTB machines:
hansen
shiller
shepard
white
ride
- ???
- Document configuration of Trilinos using ATTBDevEnv.cmake on "Heterogeneous Advanced Architecture Platforms (HAAPs)" wiki page ...
- Set up Nightly builds posted to Trilinos CDash on ATTB platforms and hand over builds to SEMS team to run with Jenkins ...