Create automated tool to update ATDM Trilinos GitHub issues with current status of tests attached to those issues
Created by: bartlettroscoe
CC: @fryeguy52, @trilinos/framework
Description
As part of the work to implement a Python tool to pull down, analyze and the summarize CDash build and test data in #2933, we realized that we had all of the information that would be needed to update Trilinos GitHub issues about the status of tests associated with Trilinos GitHub issues. The idea would be to add a GitHub issue comment containing tables showing the current status of the tests related to a GitHub Issue. An example of what this could look like is shown in https://github.com/trilinos/Trilinos/issues/3579#issuecomment-438324283 and https://github.com/trilinos/Trilinos/issues/3833#issuecomment-438317812 where I just manually copied and pasted the HTML-formatted tables and rows for those issues right into the the GitHub comments.
For #3579 (closed) on 11/14/2018, the results for 11/13/2018 might look like:
#3579 (closed) as of testing day 2018-11-13
Test results forTests with issue trackers Passed: twip=2 Tests with issue trackers Missing: twim=3
NOTE: With all of the tests associated with this issue passing or missing (i.e. disabled), might this Issue be addressed and perhaps be closed?
Tests with issue trackers Passed: twip=2
Site | Build Name | Test Name | Status | Details | Consecutive Days Pass | Nopass last 30 Days | Pass last 30 Days | Tracker |
---|---|---|---|---|---|---|---|---|
hansen | Trilinos-atdm-hansen-shiller-cuda-8.0-opt | PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 | Passed | Completed | 4 | 34 | 4 | #3579 |
white | Trilinos-atdm-white-ride-cuda-9.2-opt | PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 | Passed | Completed | 2 | 26 | 3 | #3579 |
Tests with issue trackers Missing: twim=3
Site | Build Name | Test Name | Status | Details | Consecutive Days Missing | Nopass last 30 Days | Pass last 30 Days | Tracker |
---|---|---|---|---|---|---|---|---|
hansen | Trilinos-atdm-hansen-shiller-cuda-8.0-debug | PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 | Missing | Missing | 4 | 26 | 0 | #3579 |
hansen | Trilinos-atdm-hansen-shiller-cuda-9.0-debug | PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 | Missing | Missing | 4 | 25 | 0 | #3579 |
white | Trilinos-atdm-white-ride-cuda-9.2-debug | PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 | Missing | Missing | 4 | 25 | 0 | #3579 |
So the idea is is that once that comment was added, the Trilinos developer responsible for the github issue could add a comment stating that this was not a randomly failing test so having this test pass or be disabled indicated that the issue could be resolved and then close the issue. No need to look at CDash directly, add in new CDash links, etc. Just comment and close. That would save a lot of human time.
So we would add these comments in the following cases:
-
Once a week just as a reminder of the current status of the tests related to this issue (so that Trilinos developers would not forget about the issue).
-
When one of the associated tests changed status (e..g went from passing to failing or failings to passing). (But not for frequent randomly failing tests or that would create a lot of spam updates).
-
When all of associated tests were were all passing or missing for X (e.g. 2) consecutive days, like shown above. (But not for any randomly failing tests or that would create a lot of spam.)
-
When all of the associated tests are all passing for the full acquired test history (e.g. 30 days) or are missing for frequent randomly failing tests. (But not for rare randomly failing tests.)
This should make it so that people don't need to manually check on the status of the associated tests for a GitHub issue. They could just let an automated system (that will be created in this Story) update the GitHub issue when something worth noting has occurred and when the issue might be closed.
Also, this tool could check that if all of the tests are passing and the GitHub issue was closed, then it could automatically remove the entries for that issue from a list of *.csv
files. And, if it detects that tests are not all passing and the GitHub issue is closed, then it could automatically re-open the issue and provide a summary of the test results.
Task
- Experiment with how to robustly add comments to a GitHub issue. (e.g. look into PyGitHub?)
- Update the
cdash_analyze_and_report.py
tool to write detailed test information and history for all tests with issue trackers being monitored by the tool. - Write Python code with unit tests and a Python tool called something like
update_github_issues_for_tests_status.py
that will update GitHub issues as described above given the output from thecdash_analyze_and_report.py
tool described above. - ???