Pull Request Testing
Changes to the Trilinos develop branch must be made through Pull Requests. Prior to applying changes to the develop branch, the Trilinos Project supports a pull request testing feature. Pull request testing checks a small number of builds to support basic stability and provide developers with confidence that their changes are not breaking key configurations. Pull request testing is not comprehensive, and is not meant to guarantee broad portability. Also, pull request testing is only supported for pull requests submitted against the develop branch of Trilinos. Pull requests to all other branches are ignored.
The Autotester tool that drives pull request testing for Trilinos is scheduled to run every 20 minutes, and does so unless another pull request being tested blocks it from doing so, in which case, the tester will run again after the current test run has completed. The tool has limited ability to run in parallel. Prior to being tested, a pull request must pass pre-test inspection. This inspection can be satisfied by either
- the pull request submitter being a member of the Trilinos GitHub organization
- a member of the Trilinos GitHub organization approving the pull request
There is also a pre-merge inspection. This inspection can only be satisfied through a member of the Trilinos GitHub organization approving the pull request (with no other members requesting changes), or through special exception.
The Autotester schedules Jenkins jobs to confirm the submitted code will configure, build, and test using broad but fairly vanilla configurations and implements the Pull Request Autotester inspection. The current set of builds performed on each pull request includes:
- GCC 4.8.4 with openmpi-1.10.1 and pthreads
- OpenMP enabled
- GCC 4.9.3 SERIAL (no MPI) and pthreads
- Fortran disabled
- Intel 17.0.1 with MPICH 3.2 and pthreads
- GCC 7.2 with openmpi-1.10.1 and pthreads
- Complex double support
- Warnings as errors flags (applies to only a subset of packages)
- CUDA 9.2 build
- Some package and test disables
Note that these are meant to assure that the basic work of other developers can continue, not to replace the fuller SQA cycle represented by the nightly dashboard and release process.
When the three status tests listed above (pre-test inspection, autotester inspection, and pre-merge inspection) all pass, and GitHub indicates that the merge can be done without conflict, the merge button will turn green as an indicator the PR is ready to be merged. Note that having the
AT:WIP label set will stop the autotester from proceeding to this check.
Understanding and controlling what the autotester does
Turn off testing - Add the
AT:WIP(Work In Progress) label to the PR. No testing will occur until it is removed.
Retest - Add the
AT:RETESTlabel to the PR. The autotester will remove this label when it schedules the next build and test.
Automerge when testing completes - Add the
AT:AUTOMERGElabel to the PR. Once all required checks have passed, the autotester will merge the PR without further user intervention.
Prevent stale PRs from being merged - If testing passed, but it has been at least 6 days since the PR was tested, the autotester will apply the
AT:STALElabel to the PR. This will prevent the PR from being merged until the PR is tested again. Apply the
AT:RETESTto request that a stale PR be retested.
Finding the errors from an autotest build
The Jenkins builds from the autotester report to the CDash page under the Pull Request Group. These builds have names of the form
<build-num> are shown in the PR comments created by the autotester when testing is started and when testing is completed.
Note that the CDash entry you are looking in the Pull Request Group for may not be on the current date. A full query can be constructed using the PR number and all dates before today similar to this query - just replace the 99999 (the
<pr-id>) with your PR number.