Send out summary email of failing Trilinos Nightly builds and tests at end of each testing day
Created by: bartlettroscoe
CC: @trilinos/framework
Related to: #380
Description:
At a meeting last week between Jim W. (@jwillenbring), Brent P. (@bmpersc), Erik S. and myself (@bartlettroscoe), Erik mentioned that one of the internal projects had/has a process where they send out a summary email at the end of each testing day that summarizes the failing tests. This email is seen by every developer on the project, the project lead, and the manager. If these tests are allowed to fail day after day, then the project lead (and then the managers) will step in and provide motivation and encouragement for the development team to help address the issues.
With the current CDash setup that is used by Trilinos, a targeted email goes out for every configure, build, or test failure for a package as soon as the results are uploaded and processed by the CDash. This email is targeted to the email list that is attached to a given Trilinos package. This gives that package team a heads up about the failure as soon as it is known. While this is good because it alerts the targeted people as soon as possible, one can't see the fully set of failures at the end of a testing day without going directly to CDash and looking.
This Story is to implement a process where a single email would be send out at the end of each testing day (e.g. at 1 am after the last testing day ended at 12 pm) to all of the Trilinos developers, project leads, and responsible managers with the full list of failing tests for the Nightly Trilinos builds. As described for the other internal SNL project described above, having this feature would help to elevate the importance of keeping the Nightly builds clean and therefore would support beefing up the testing to promote from the 'develop' branch to the 'master' branch.
There are several ways to implement this:
-
Add this as an official feature to CDash and pay Kitware to add it.
-
Implement this as a python script that reads from CDash using the new CDash API.
In my opinion, it would be pretty easy to implement option-2 above given the new CDash API that returns a JSON file for any CDash PHP page. This would be very similar to the script send_kanban_wip_reminder_emails.py
which I (Ross) wrote for CASL PHI that queries the Trac DB and then creates a customized HTML-formatted email with a nice formatted table. My guess is that writing a script like this could be done in about 4 hours, including some unit tests to define and protect this functionality. You would then set up a cron job to run this script at the same time at the end of each testing day.