Issues to be address before making automated PR testing and merging mandatory
Created by: bartlettroscoe
CC: @trilinos/framework, @rppawlo, @maherou, @srajama1
Description:
There has been recent discussion about when the time to make automated PR testing and merging system being developed by the @trilinos/framework team mandatory. From my own experience and conversations with other people, the following issues would need to be addressed before that can occur:
- Update and streamline the builds used by the automated PR testing system (to make sure that we are getting the needed testing, having the builds not take too many computer resoruces, and getting the biggest bang for our buck in protecting important functionality.) (see #2317 (closed))
- STATUS: ???
- Post all build results to CDash to see exactly what is getting tested and what the failures are (and allow it to be reviewed before the final merge).
- Complete the upgrade of the new CDash version that supports all-at-once configure, build, test, and submit (see https://gitlab.kitware.com/snl/project-1/issues/33).
- STATUS: Currently, the PR build results are being submitted to the trial testing-vm.sandia.gov/cdash site and the Framework team is okay with the stability of that site for managing PR builds so this is not currently considered an impediment.
- Upgrade the minimum version of CMake used by Trilinos from 2.8.11 to 3.10 in order to take advantage of all-at-once configure, build, test, and submit to new CDash version. (Also, once the auto PR testing is using CMake 3.10 it will allow CMake code in packages that does not work with older versions of CMake and therefore will destabilize other builds of Trilinos unless we force the upgrade).
- STATUS: This can be done later. For now, this might allow CMakeLists.txt files to get pushed that will break the configure of Trilinos with older CMake versions but the current post-push CI build and other nightly builds would catch that.
- Complete the upgrade of the new CDash version that supports all-at-once configure, build, test, and submit (see https://gitlab.kitware.com/snl/project-1/issues/33).
- Make it easy and obvious how to reproduce the exact auto PR builds and results shown on CDash and to allow people to run these locally even before they post their PR if they would like (e.g. #2295)
(Which includes making the env and/or machine accessible to all Trilinos developers, see https://github.com/trilinos/Trilinos/issues/2317#issuecomment-369987520.)
- STATUS: ???
- Auto-merge with final staged testing before merge to ‘develop’ (i.e. avoid violations of the additive test assumption of branches that resulted #2264 (closed)).
- STATUS: Currently plan is to mark older PRs as "Stale" but not strictly guarantee stability of 'develop' due to violations of the "additive test assumption of branches"?
- Auto wiping the build directory when requested so that it will deal with cases where you need to build from scratch (e.g. when Panzer files get update such as described in https://github.com/trilinos/Trilinos/issues/1304#issuecomment-355786180 and https://github.com/trilinos/Trilinos/issues/1304#issuecomment-356053919).
- STATUS: Currently a complete build from scratch with each PR build is being done to address this. With this implementation, this is not an impediment.
- Address cases where no packages get enabled and therefore no tests are run. Will the auto PR system mark them as "passed" and allow the PR to be merged?
- STATUS: ???
- Document the PR and auto testing process so people can understand how to use it correctly and understand why it is firing off tests in some cases and not in other cases.
- STATUS: ???
- Ensure the current testing infrastructure will scale when everyone is required to use topic branches for pushing anything (even documentation changes). For example, will this scale when there are 20 active PRs being edited at the same time? When there are more and more topic branches, how long with the lags be before things get tested and eventually merged?
- STATUS: This will be addressed by "Update and streamline the builds used by the automated PR testing system" above?
Other issues that should be resolve (quickly) after initial adoption:
- Remove manual updates of the list of packages from mapping of directories to packages. (e.g. use self-maintaining existing code in TriBITS tools for that?). If you don’t things will not get tested correctly without manual intervention to constantly keep that manual list up to date.
The motivation for this issue was conversations with @rppawlo, @jwillenbring and the 2018-02-28 Trilinos Planning meeting
Related Issues
- Related to: #1154, #1304 (closed)
- Composed of: #2317 (closed)
Tasks:
- Define the set of builds to be used in the auto PR system with a working group (#2317 (closed)). See #2317 (closed) ...
- Complete and robustify the submitting of results to CDash (there is a start to this but it is having problems, see https://github.com/trilinos/Trilinos/pull/2310#issuecomment-369390502).
- ???