In order to maintain the stability of Primary Tested packages and code on the 'develop' branch of Trilinos, the checkin-test.py script can be used to test code before any push that changes source code. A standard environment for running the pre-push CI tests has been set up based on the SEMS Development Environment encapsulated in the script checkin-test-sems.sh. This script makes it easy to run on any machine that has the SEMS Dev Env mounted. To set up to use this script to test and push changes to Trilinos, do:
cd Trilinos/ mkdir CHECKIN/ # Or put this anywhere you want on Linux systems cd CHECKIN ln -s ../cmake/std/sems/checkin-test-sems.sh .
(WARNING: The Trilinos git repo must use SSH with the remote URL
email@example.com:trilinos/Trilinos.git and not HTTPS for access to github. See below note).
Then when one wants to test changes (and the local git repo is "clean" with no modified or untracked files), one can do:
cd Trilinos/CHECKIN/ ./checkin-test-sems.sh --do-all
DEPRECATED: In the past there was a recommendation to use --push with this script for the develop branch. With the advent of the feature-branch workflow this is deprecated behaviour. The --push flag is still useable for feature branches.
(see default options in local-checkin-test-defaults.py). That will automatically figure out what packages are changed and will enable those packages and all of their downstream packages and BASIC tests. Then if everything passes, it will rebase the commits on top of
origin/develop, amend the top commit message with the test results summary, and push the commits (i.e. it implements the simple centralized workflow by default).
If one is merging and pull-request branch
<some-branch> from the GitHub fork account
<some-github-id>, then one can merge the branch locally and run with
git checkout develop git pull git remote add <some-github-id> git@github:<some-github-id>/Trilinos.git git fetch <some-github-id> git merge --no-ff <some-gitub-id>/<some-branch> cd CHECKIN/ ./checkin-test-sems.sh --do-all --no-rebase --push
That will avoid a rebase and leave the merge commit (which will be amended with the test results). Once the commits are pushed to the 'develop' branch, GitHub will automatically close the associated pull-request issue.
If one is nervous testing and pushing in one shot, then one can break this up into two commands with:
./checkin-test-sems.sh --do-all <manually inspect what packages got enabled and tested and make sure looks good> ./checkin-test-sems.sh --push
(Using the checkin-test-sems.sh script to push is critical in order to mark known-tested commits to allow for robust usage of git bisect. Without this, git bisection is very hard to do with Trilinos. Also, local commits will be rebased by default, keeping a nice linear history on 'develop'.)
If one changes a given Trilinos package and one knows that it only impacts that one package (such as when only modifying tests but no changes to library code), then one can run the build and tests for only the directly modified packages with:
./checkin-test-sems.sh --enable-all-packages=off --no-enable-fwd-packages --do-all --push
Also, one can test any set of packages
<pkg1>, etc. (and not downstream packages) with the local git repo in any arbitrary state (i.e. has modified or untracked files, is in a detached-head state, etc.) using:
./checkin-test-sems.sh --enable-all-packages=off --enable-packages=<pkg0>,<pkg1>,... \ --no-enable-fwd-packages --local-do-all
One can also run any individual step by replacing
--local-do-all with the action commands
--test (the script keep memory of what steps have been performed already).
If the configure, build or any tests fail, then one can easily reproduce them just like for any regular build of Trilinos by sourcing the script load_sems_dev_env.sh and doing the following (in an
sh compatible shell, not (t)csh):
Reproducing failures manually
cd Trilinos/CHECKIN/ source ../cmake/load_sems_dev_env.sh # NOTE: Does 'module purge' then loads new env! cd MPI_RELEASE_DEBUG_SHARED_PT_OPENMP/ less configure.out # View cmake configure output/errors ./do-configure # Reproduce configure failure less make.out # View make build output/errors make -j<N> # Reproduce build failure less ctest.out # View ctest output/failures ctest -j<N> -R <test-name> # Reproduce failing test(s)
(Warning: loading the SEMS env with
source ../cmake/load_sems_dev_env.sh will wipe out your current modules so you might want to open a new bash shell before loading it.)
Many other use cases are also supported. Some detailed documentation on the checkin-test.py script can be obtained using checkin-test.py --help.
[ssh-vs-https] In order for the automated pulls and pushes that the script does to work, the Trilinos git repos must be set up to use SSH to access git instead of HTTPS (i.e. the remote must be
https://github.com/trilinos/Trilinos.git). See Connecting to GitHub with SSH.
[local-checkin-test-defaults.py] After the first time
checkin-test-sems.shis run (e.g. with just
--help), optionally edit the file
local-checkin-test-defaults.pyto adjust defualt options like
-j<N>to better match your machine (see comments inside of the script
checkin-test-sems.shitself for how to adjust these default options).
checkin-test-sems.shscript, by default, runs a single build called
MPI_RELEASE_DEBUG_SHARED_PT_OPENMP(see CMake options in
MPI_RELEASE_DEBUG_SHARED_PT_OPENMP/do-configure.base) with the env defined by the source script load_sems_dev_env.sh. This is a build designed to best protect developers and users of Trilinos. This build allows the enable of any Primary Tested (PT) packages and enables several TPLs provided by the SEMS env (see
Final set of enabled TPLsin
The standard pre-push CI build env is currently chosen to be the Sandia Linux RHEL 6 or 7 COE with the SEMS env. It is recommended that Trilinos developers test and push from one of these machines, or ask someone else with access to one of these machines to push for you. (Please contact trilinos-framework at software.sandia.gov if you do not have access and time on one of these systems. Most Sandia staff can be given lab-support systems for this purpose.).
One can do development on any machine with any env one wishes and then use a remote SNL Linux RHEL 6 or 7 machine with the SEMS env to do a remote pull, test, and push of the commits. (Or, one can just manually move the branch from the local repo to an SNL RHEL 6 or 7 machine and run the checkin-test-sems.sh script from there.)
[raw_checkin_test_py] The raw script
Trilinos/checkin-test.pycan still be used for doing local development work and testing or driving any workflow. It can even be used for pushing (but please prefer to use the
checkin-test-sems.shscript on a RHEL 6 or 7 machine with the SEMS env when pushing to the main Trilinos 'develop' branch). However, unless you can define the
MPI_RELEASE_DEBUG_SHARED_PT_OPENMP.configfile to support the full set of PT packages and TPLs, then just disable that build by running with your own pre-defined builds (i.e. creating
<build1>.config) by passing in
--default-builds= --st-extra-builds=<build0>,<build1>,.... (See the checkin-test-sems.sh script for an example of how to do this.)
[disable_already_failing] You can see if the CI build is currently clean by looking at the CI build on CDash. If you see failing tests then you can disable those tests when invoking the checkin-test-sems.sh script by passing in
--extra-cmake-options="-D<failing_test_0>_DISABLE=ON -D<failing_test_1>_DISABLE=ON ...". That will allow you to push by ignoring those failing tests. If entire packages are failing to build or have catastrophic test failures, then you can disable those using
The checkin-test-sems.sh script is not supported on systems like Mac OSX or Windows. It only works and has been tested on Linux machines (in particular on RHEL 6 and 7 with the SEMS env at present).
[csh_shell_issues] Some people have reported module load problems when running the
checkin-test-sems.shscript directly from a
cshshell. Therefore, if there are issues, first open a
bashshell and then run the script from there. Also note that one can not manually source the
cshbut instead must use
bash(see above). If one wants to manually load the SEMS modules on a non-
(t)csh) and then reproduce failures , then one will need to manually load the SEMS modules (direct support for
(t)cshis not provided).