TriBITS update to allow usage in Trilinos auto PR file change logic (#3133)
Created by: bartlettroscoe
@william76, @william76, @fryeguy52
Description
This PR branch contains a TriBITS snapshot update to allow the Trilinos auto PR tester to use TriBITS logic in determining what to enable based on git diff --name-only <branch-tip> ^github/devleop
. (See #3133 (closed)). This includes specialized logic for what file changes in Trilinos should trigger a global build.
I added strong strong automated testing for this new script and added a new TriBITS package called TrilinosFrameworkTests
to run these automated tests automatically.
Motivation and Context
See the motivation in #3133 (closed).
How Has This Been Tested?
In addition to the strong automated tests added in the new TrilinosFrameworkTests
package, I did some manual testing. For example, I looked at a large set of changes between the tip of this branch and a version of Trilinos 'develop' from 7/6/2018 de924c.
time ~/Trilinos.base/Trilinos/commonTools/framework/get-changed-trilinos-packages.sh de924c 3133-tribits-file-change-logic enablePackages.cmake
***
*** Generating set of Trilinos enables given modified packages from
**** git commit de924c to 3133-tribits-file-change-logic
***
TRILINOS_DIR=/home/rabartl/Trilinos.base/Trilinos/commonTools/framework/../..
TRIBITS_DIR=/home/rabartl/Trilinos.base/Trilinos/commonTools/framework/../../cmake/tribits
A) Generate the TrilinosPackageDependencies.xml file
B) Get the set of changed files
Current directory: /home/rabartl/Trilinos.base/Trilinos
Writing file /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED/changed-files.txt ...
git diff --name-only de924c..3133-tribits-file-change-logic > /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED/changed-files.txt
Current directory: /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED
C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')
CHANGED_PACKAGES_FULL_LIST='ALL_PACKAGES,TrilinosFrameworkTests,Amesos2,Anasazi,Belos,Galeri,Ifpack2,Kokkos,MueLu,PanzerAdaptersSTK,PanzerDofMgr,PanzerMiniEM,Phalanx,Piro,ROL,Sacado,SEACAS,SEACASAlgebra,SEACASAprepro,SEACASBlot,SEACASConjoin,SEACASEpu,SEACASEx1ex2v2,SEACASEx2ex1v2,SEACASExo2mat,SEACASExo_format,SEACASExodiff,SEACASExomatlab,SEACASExotxt,SEACASExplore,SEACASFastq,SEACASGen3D,SEACASGenshell,SEACASGjoin,SEACASGrepos,SEACASMapvar-kd,SEACASMapvar,SEACASMat2exo,SEACASNemslice,SEACASNemspread,SEACASNumbers,SEACASTxtexo,SEACASAprepro_lib,SEACASChaco,SEACASExoIIv2for32,SEACASExodus,SEACASExodus_for,SEACASIoss,SEACASMapvarlib,SEACASNemesis,SEACASPLT,SEACASSupes,SEACASSuplib,SEACASSuplibC,SEACASSuplibCpp,SEACASSVDI,ShyLU_DDFROSch,ShyLU_NodeBasker,Stokhos,Stratimikos,Tempus,TeuchosComm,TpetraClassic,TpetraCore,TrilinosCouplings,Xpetra,Zoltan2'
D) Filter list of changed pacakge to get only the PT set of packages
CHANGED_PACKAGES_PT_LIST='ALL_PACKAGES,TrilinosFrameworkTests,Amesos2,Anasazi,Belos,Galeri,Ifpack2,Kokkos,MueLu,PanzerAdaptersSTK,PanzerDofMgr,PanzerMiniEM,Phalanx,Piro,ROL,Sacado,SEACAS,SEACASAlgebra,SEACASAprepro,SEACASConjoin,SEACASEpu,SEACASEx1ex2v2,SEACASEx2ex1v2,SEACASExo_format,SEACASExodiff,SEACASExotxt,SEACASExplore,SEACASGen3D,SEACASGenshell,SEACASGjoin,SEACASGrepos,SEACASMapvar-kd,SEACASMapvar,SEACASNemslice,SEACASNemspread,SEACASNumbers,SEACASTxtexo,SEACASAprepro_lib,SEACASChaco,SEACASExoIIv2for32,SEACASExodus,SEACASExodus_for,SEACASIoss,SEACASMapvarlib,SEACASNemesis,SEACASSupes,SEACASSuplib,SEACASSuplibC,SEACASSuplibCpp,ShyLU_DDFROSch,Stokhos,Stratimikos,Tempus,TeuchosComm,TpetraClassic,TpetraCore,TrilinosCouplings,Xpetra,Zoltan2'
E) Generate the *.cmake enables file
real 0m14.791s
user 0m14.710s
sys 0m0.086s
That git range diff gives a list of 4342 files shown by the generated file:
$ wc -l changed-files.txt
4342 changed-files.txt
This script also generated the file:
$ wc -l TrilinosPackageDependencies.xml
2270 TrilinosPackageDependencies.xml
which gives the Trilinos project dependencies.
The generated file enablePackages.cmake
(the name is whatever you want it to be) looks like:
MACRO(PR_ENABLE_BOOL VAR_NAME VAR_VAL)
MESSAGE("-- Setting ${VAR_NAME} = ${VAR_VAL}")
SET(${VAR_NAME} ${VAR_VAL} CACHE BOOL "Set in enablePackages.cmake")
ENDMACRO()
PR_ENABLE_BOOL(Trilinos_ENABLE_ALL_PACKAGES ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_TrilinosFrameworkTests ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Amesos2 ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Anasazi ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Belos ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Galeri ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Ifpack2 ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Kokkos ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_MueLu ON)
...
When there are no Trilinos packages enabled, like when only changing files under cmake/std/atdm/
with a recent PR merged just today #3158, the script gives:
$ time ~/Trilinos.base/Trilinos/commonTools/framework/get-changed-trilinos-packages.sh be669e4^ be669e4 enablePackages.cmake
***
*** Generating set of Trilinos enables given modified packages from
**** git commit be669e4^ to be669e4
***
TRILINOS_DIR=/home/rabartl/Trilinos.base/Trilinos/commonTools/framework/../..
TRIBITS_DIR=/home/rabartl/Trilinos.base/Trilinos/commonTools/framework/../../cmake/tribits
A) Generate the TrilinosPackageDependencies.xml file
B) Get the set of changed files
Current directory: /home/rabartl/Trilinos.base/Trilinos
Writing file /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED/changed-files.txt ...
git diff --name-only be669e4^..be669e4 > /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED/changed-files.txt
Current directory: /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED
C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')
CHANGED_PACKAGES_FULL_LIST=''
D) Filter list of changed pacakge to get only the PT set of packages
CHANGED_PACKAGES_PT_LIST=''
E) Generate the *.cmake enables file
real 0m0.731s
user 0m0.664s
sys 0m0.065s
with the generated file enablePackage.cmake
:
MACRO(PR_ENABLE_BOOL VAR_NAME VAR_VAL)
MESSAGE("-- Setting ${VAR_NAME} = ${VAR_VAL}")
SET(${VAR_NAME} ${VAR_VAL} CACHE BOOL "Set in enablePackages.cmake")
ENDMACRO()
MESSAGE("-- NOTE: No packages are being enabled!")
Note that you can run these generated *.cmake
files like:
$ cmake -P enablePackages.cmake
-- NOTE: No packages are being enabled!
(This is used in the automated testing.)
Checklist
-
My commit messages mention the appropriate GitHub issue numbers.