Be able to annotate commit with test results from multiple builds
Created by: mhoemmen
Here are the relevant parts of the #1304 (closed) discussion that led to this:
@bartlettroscoe, this may be addressed elsewhere, but is there a best practice for the case that I want to test on multiple machines before pushing (or opening a PR)? For example, with Tpetra, I want to make sure that the standard checkin tests pass on my RHEL blade (obviously), but also different builds on different architectures. In this case, I may push to a branch on my fork and then update and run tests on the several different machines from that branch. The only way (that I know of) to then report test results is to amend the last commit manually with test results and force push. Perhaps there is a better way?
@tjfulle I like your thinking :-) . Right now, one must add text manually to the commit message, explaining what tests passed where. We don't necessarily want to annotate every commit with all the information needed to replicate a test, but some brief mention that (e.g.,) it was tested with CUDA in a debug build could be nice.
That would not be too hard to do with crafty usage of the checkin-test.py script. You could run the
checkin-test.pyscript on each machine separately (e.g. with
--local-do-all) for special
--extra-st-builds=<specialBuildNamei>and then you could copy a subset of the
<specialBuildNamei>/*.outfiles and all of the
<specialBuildNamei>/*.successfiles to your CEE LAN machine and then the
checkin-test-sems.shscript could be run with the full list of
--extra-st-builds=<specialBuildName1>,<specialBuildName2>,...and it would correctly list those extra builds and would amend the top commit message with all of those builds and would also archive the details of those builds to the
trilinos-checkin-testsemail list on push. If you combine the moving of a branch and remote run approach demonstrated in
remote-pull-test-push.shand combine that with the aggregation of multiple runs of the
checkin-test-crf450-cmake-2.8.11.shand add some
scpcommands to copy files back, then you basically have it. One could even write some reusable utility scripts to help drive a process like this so many developers could use it.