Skip to content

Improve testing for updating from 'develop' branch to 'master' branch by querying CDash results

Created by: bartlettroscoe

Next Action Status:

Waiting for #370 (closed) to be completed before getting started on this ...

Blocked By: #370 (closed)

CC: @jwillenbring, @bmpersc, @mhoemmen, @trilinos/framework

Description:

The Story #370 (closed) implemented the initial transition to the 'develop'/'master' branch workflow using just a simple automated job that just does one build with the checkin-test.py script and then pushes the updated 'master' branch. This is a follow-on Story to put the infrastructure and the process in place to beef up the level of testing that needs to pass before updating the 'master' branch. The approach will be to take advantage of the new CDash API that allows you to download CDash query results as a JSON data-structure and then use a Python script to inspect it and make sure all of the targeted builds and packages all passed.

This has already been implemented for CASL VERA in the Python script vera_cdash_pass_fail.py. This script is run as:

$ cd VERA/
$ ./cmake/ctest/drivers/vera_cdash_pass_fail.py --date=2016-04-28

…

Getting data from:

  https://casl-dev.ornl.gov/testing/api/v1/index.php?project=VERA&date=2016-04-28&filtercount=4&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=Nightly&field2=subprojects&compare2=92&value2=VUQCore&field3=subprojects&compare3=92&value3=VUQDemos

VERA builds failed!

Error, the build {u'buildname': u'Linux-GCC-4.8.3-MPI_RELEASE_SHARED_HEAVY', u'test': {u'fail': 2, u'timefull': 44707, u'time': u'12h 25m 7s ', u'notrun': 0, u'pass': 1750}, … } failed!
…
FINAL: One of the VERA or VERADriver builds on 2016-04-28 FAILED

This script uses the reusable Python module:

  tribits/ci_support/CDashQueryPassFail.py

The script vera_cdash_pass_fail.py does one query of the VERA CDash project and one of the VERADriver CDash project (created using the TriBITS Dashboard Driver (TDD) system). The VERA CDash query is shown above and it is examined to make sure that all of the expected builds are present and that there are no failing configures, builds or tests. The VERADriver query is done to make sure that all of the outer CTest driver builds ran and there was no failures (e.g. timeouts). Without querying the VERADriver project to make sure that none of the CTest drivers timed out, then having all passing VERA builds is not sufficient (because packages may have never been tested).

Because Trilinos uses Jenkins to run CTest build drivers, a query of Jenkins may be needed to ensure that none of the selected builds timed out. However, it appears that Jenkins supports a remote API so developing queries of Jenkins should be possible as well.

Tasks:

  1. Upgrade the Trilinos CDash server to a current version that supports the new CDash API interface. (NOTE: This might also require upgrading the server hardware and env so that it can handle the required volume of Trilinos CDash submits.)
  2. Develop a Python script trilinos_cdash_pass_fail.py that will run and inspect one or more queries of Trilinos CDash project and return false if any of these queries show failures. Also, do queries of the Jenkins CDash drivers to make sure that the outer CTest driver invocations don't timeout or otherwise fail.
  3. Select a (small) initial set of Trilinos builds and set of packages in one or more CDash queries that must pass before updating the 'master' branch from the 'develop' branch. (NOTE: Adding these extra builds are likely to be separate stories. The story will be just to set up the infrastructure to query CDash correctly and get just a small number of extra builds in the initial set. Every new build you add creates "value" and can therefore be its own Story.)