autotester not testing new pull requests
Created by: ibaned
@trilinos/framework @jwillenbring @allevin
Expectations
New pull requests should get tested once they are created, and "old" pull requests should be tested less to avoid a spam-like series of "passed" comments.
Current Behavior
Currently, the pull request autotester seems to work like a FIFO queue, meaning that new pull requests do not get attention until they are in the top 3 or 4 at the front of the queue (they start at the back). Also, until pull requests are merged, they remain at the front of the queue and continue to receive repeated (often unnecessary) testing.
Motivation and Context
I've put up a pull request but the autotester has not examined it in several days, rather its resources are focused on re-testing older PRs that have already passed autotesting several times and are waiting for action from the relevant developers. There is a significant trend in Trilinos of pull requests sitting idle for many days because their changes are non-trivial and package owners have limited availability to review them. Furthermore, even marking those older PRs as [WIP]
does not seem to free up the autotester to look at newer ones.
Definition of Done
-
Brand new pull requests get at least one test right soon after they are created -
Older pull requests waiting on developer action don't get constantly re-tested
In short, the expensive operation of testing a commit should be used wisely to maximize throughput of pull requests.
Possible Solution
There are multiple possible approaches. A simple one would be to make the autotester work as a LIFO stack. It could also be event-based, with testing only occuring after certain events. Being created should be a key event, being approved by a developer could be another. If a developer has requested changes, there should be no testing until all changes are fully approved again. When resources are available to test something, PRs could be prioritized by most recent event.