Skip to content

TriBITS update to allow usage in Trilinos auto PR file change logic (#3133)

Created by: bartlettroscoe

@william76, @william76, @fryeguy52

Description

This PR branch contains a TriBITS snapshot update to allow the Trilinos auto PR tester to use TriBITS logic in determining what to enable based on git diff --name-only <branch-tip> ^github/devleop. (See #3133 (closed)). This includes specialized logic for what file changes in Trilinos should trigger a global build.

I added strong strong automated testing for this new script and added a new TriBITS package called TrilinosFrameworkTests to run these automated tests automatically.

Motivation and Context

See the motivation in #3133 (closed).

How Has This Been Tested?

In addition to the strong automated tests added in the new TrilinosFrameworkTests package, I did some manual testing. For example, I looked at a large set of changes between the tip of this branch and a version of Trilinos 'develop' from 7/6/2018 de924c.

 time ~/Trilinos.base/Trilinos/commonTools/framework/get-changed-trilinos-packages.sh de924c 3133-tribits-file-change-logic enablePackages.cmake

***
*** Generating set of Trilinos enables given modified packages from
**** git commit de924c to 3133-tribits-file-change-logic
***

TRILINOS_DIR=/home/rabartl/Trilinos.base/Trilinos/commonTools/framework/../..
TRIBITS_DIR=/home/rabartl/Trilinos.base/Trilinos/commonTools/framework/../../cmake/tribits

A) Generate the TrilinosPackageDependencies.xml file


B) Get the set of changed files


Current directory: /home/rabartl/Trilinos.base/Trilinos

Writing file /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED/changed-files.txt ...

git diff --name-only de924c..3133-tribits-file-change-logic > /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED/changed-files.txt

Current directory: /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED

C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')

CHANGED_PACKAGES_FULL_LIST='ALL_PACKAGES,TrilinosFrameworkTests,Amesos2,Anasazi,Belos,Galeri,Ifpack2,Kokkos,MueLu,PanzerAdaptersSTK,PanzerDofMgr,PanzerMiniEM,Phalanx,Piro,ROL,Sacado,SEACAS,SEACASAlgebra,SEACASAprepro,SEACASBlot,SEACASConjoin,SEACASEpu,SEACASEx1ex2v2,SEACASEx2ex1v2,SEACASExo2mat,SEACASExo_format,SEACASExodiff,SEACASExomatlab,SEACASExotxt,SEACASExplore,SEACASFastq,SEACASGen3D,SEACASGenshell,SEACASGjoin,SEACASGrepos,SEACASMapvar-kd,SEACASMapvar,SEACASMat2exo,SEACASNemslice,SEACASNemspread,SEACASNumbers,SEACASTxtexo,SEACASAprepro_lib,SEACASChaco,SEACASExoIIv2for32,SEACASExodus,SEACASExodus_for,SEACASIoss,SEACASMapvarlib,SEACASNemesis,SEACASPLT,SEACASSupes,SEACASSuplib,SEACASSuplibC,SEACASSuplibCpp,SEACASSVDI,ShyLU_DDFROSch,ShyLU_NodeBasker,Stokhos,Stratimikos,Tempus,TeuchosComm,TpetraClassic,TpetraCore,TrilinosCouplings,Xpetra,Zoltan2'

D) Filter list of changed pacakge to get only the PT set of packages

CHANGED_PACKAGES_PT_LIST='ALL_PACKAGES,TrilinosFrameworkTests,Amesos2,Anasazi,Belos,Galeri,Ifpack2,Kokkos,MueLu,PanzerAdaptersSTK,PanzerDofMgr,PanzerMiniEM,Phalanx,Piro,ROL,Sacado,SEACAS,SEACASAlgebra,SEACASAprepro,SEACASConjoin,SEACASEpu,SEACASEx1ex2v2,SEACASEx2ex1v2,SEACASExo_format,SEACASExodiff,SEACASExotxt,SEACASExplore,SEACASGen3D,SEACASGenshell,SEACASGjoin,SEACASGrepos,SEACASMapvar-kd,SEACASMapvar,SEACASNemslice,SEACASNemspread,SEACASNumbers,SEACASTxtexo,SEACASAprepro_lib,SEACASChaco,SEACASExoIIv2for32,SEACASExodus,SEACASExodus_for,SEACASIoss,SEACASMapvarlib,SEACASNemesis,SEACASSupes,SEACASSuplib,SEACASSuplibC,SEACASSuplibCpp,ShyLU_DDFROSch,Stokhos,Stratimikos,Tempus,TeuchosComm,TpetraClassic,TpetraCore,TrilinosCouplings,Xpetra,Zoltan2'

E) Generate the *.cmake enables file


real    0m14.791s
user    0m14.710s
sys     0m0.086s

That git range diff gives a list of 4342 files shown by the generated file:

$ wc -l changed-files.txt 
4342 changed-files.txt

This script also generated the file:

$ wc -l TrilinosPackageDependencies.xml 
2270 TrilinosPackageDependencies.xml

which gives the Trilinos project dependencies.

The generated file enablePackages.cmake (the name is whatever you want it to be) looks like:

MACRO(PR_ENABLE_BOOL  VAR_NAME  VAR_VAL)
  MESSAGE("-- Setting ${VAR_NAME} = ${VAR_VAL}")
  SET(${VAR_NAME} ${VAR_VAL} CACHE BOOL "Set in enablePackages.cmake")
ENDMACRO()

PR_ENABLE_BOOL(Trilinos_ENABLE_ALL_PACKAGES ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_TrilinosFrameworkTests ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Amesos2 ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Anasazi ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Belos ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Galeri ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Ifpack2 ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_Kokkos ON)
PR_ENABLE_BOOL(Trilinos_ENABLE_MueLu ON)
...

When there are no Trilinos packages enabled, like when only changing files under cmake/std/atdm/ with a recent PR merged just today #3158, the script gives:

$ time ~/Trilinos.base/Trilinos/commonTools/framework/get-changed-trilinos-packages.sh  be669e4^  be669e4  enablePackages.cmake

***
*** Generating set of Trilinos enables given modified packages from
**** git commit be669e4^ to be669e4
***

TRILINOS_DIR=/home/rabartl/Trilinos.base/Trilinos/commonTools/framework/../..
TRIBITS_DIR=/home/rabartl/Trilinos.base/Trilinos/commonTools/framework/../../cmake/tribits

A) Generate the TrilinosPackageDependencies.xml file


B) Get the set of changed files


Current directory: /home/rabartl/Trilinos.base/Trilinos

Writing file /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED/changed-files.txt ...

git diff --name-only be669e4^..be669e4 > /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED/changed-files.txt

Current directory: /ascldap/users/rabartl/Trilinos.base/BUILDS/GCC-4.8.3/MPI_RELEASE_DEBUG_SHARED

C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')

CHANGED_PACKAGES_FULL_LIST=''

D) Filter list of changed pacakge to get only the PT set of packages

CHANGED_PACKAGES_PT_LIST=''

E) Generate the *.cmake enables file


real    0m0.731s
user    0m0.664s
sys     0m0.065s

with the generated file enablePackage.cmake:

MACRO(PR_ENABLE_BOOL  VAR_NAME  VAR_VAL)
  MESSAGE("-- Setting ${VAR_NAME} = ${VAR_VAL}")
  SET(${VAR_NAME} ${VAR_VAL} CACHE BOOL "Set in enablePackages.cmake")
ENDMACRO()

MESSAGE("-- NOTE: No packages are being enabled!")

Note that you can run these generated *.cmake files like:

$ cmake -P enablePackages.cmake 
-- NOTE: No packages are being enabled!

(This is used in the automated testing.)

Checklist

  • My commit messages mention the appropriate GitHub issue numbers.

Merge request reports