Pull Request Autotester will continue testing even if commit to PR occurs while testing.
Created by: allevin
The Pull Request Autotester (PRTester) checks that the latest SHA of the PR (source repo/branch) matches the SHA used to kick off the testing before merging to ensure that the PR has not changed.
This check is performed before reporting the SUCCESS/FAILURE status of the testing. This way even if testing was a success, the PR will receive an ERROR comment in the PR stating that a commit occurred during testing. The status flags will not show an error indicator because the status is tied to and updated against the previous commit sha. Whereas the status(s) that Github shows are always related to the latest commit.
This is an area that some users might get frustrated. The PRTester doesn't check that the sha has been changed during the testing phase. If a user realizes that they made a mistake and try to slip in another commit to their PR, the testing does not stop and restart (they would have to wait for the current testing to finish, and a new testing cycle for the new commit would have to be run).
We could change the code (in the PRTester) to check that the SHA's are the same every time we peek at the Jenkins run status (every minute to see if the jobs are finished). Then we could kill the job(s). Then when the PRTester runs again, the status would signal the tester to re-attempt testing.