Be able to annotate commit with test results from multiple builds
Created by: mhoemmen
@trilinos/framework
Here are the relevant parts of the #1304 (closed) discussion that led to this:
@tjfulle wrote:
@bartlettroscoe, this may be addressed elsewhere, but is there a best practice for the case that I want to test on multiple machines before pushing (or opening a PR)? For example, with Tpetra, I want to make sure that the standard checkin tests pass on my RHEL blade (obviously), but also different builds on different architectures. In this case, I may push to a branch on my fork and then update and run tests on the several different machines from that branch. The only way (that I know of) to then report test results is to amend the last commit manually with test results and force push. Perhaps there is a better way?
I wrote:
@tjfulle I like your thinking :-) . Right now, one must add text manually to the commit message, explaining what tests passed where. We don't necessarily want to annotate every commit with all the information needed to replicate a test, but some brief mention that (e.g.,) it was tested with CUDA in a debug build could be nice.
@bartlettroscoe wrote:
That would not be too hard to do with crafty usage of the checkin-test.py script. You could run the
checkin-test.py
script on each machine separately (e.g. with--local-do-all
) for special--extra-st-builds=<specialBuildNamei>
and then you could copy a subset of the<specialBuildNamei>/*.out
files and all of the<specialBuildNamei>/*.success
files to your CEE LAN machine and then thecheckin-test-sems.sh
script could be run with the full list of--extra-st-builds=<specialBuildName1>,<specialBuildName2>,...
and it would correctly list those extra builds and would amend the top commit message with all of those builds and would also archive the details of those builds to thetrilinos-checkin-tests
email list on push. If you combine the moving of a branch and remote run approach demonstrated inremote-pull-test-push.sh
and combine that with the aggregation of multiple runs of thecheckin-test.py
script incheckin-test-crf450-cmake-2.8.11.sh
and add somescp
commands to copy files back, then you basically have it. One could even write some reusable utility scripts to help drive a process like this so many developers could use it.