Resolve build failures for checkin-test-sems.sh MPI_RELEASE_DEBUG_SHARED_PT build on OSX
Created by: bartlettroscoe
CC: @trilinos/framework, @trilinos/stokos, @trilinos/muelu, @trilinos/stk, @trilinos/zoltan
Description:
The checkin-test-sems.sh default build MPI_RELEASE_DEBUG_SHARED_PT was set up to use the GCC 5.3.0 compiler stack as discussed in #482 (closed). The goal was to provide a uniform build between Linux and OSX machines. That failed very badly because while things built and ran just fine on Linux, the code did not even build with the same compilers and TPLs on OSX. However, it would still be very useful to have a standard PT pre-push CI build working on OSX for OSX developers. A recent Trilinos survey shown here:
showed that nearly 32% of Trilinos developers consider OSX to be the primary development platform for Trilinos. In addition, 16% said they did not have access to a SNL RHEL 6 COE with the SEMS env to test and push from.
Therefore, one strategy to accommodate these Trilinos developers is to allow them to push directly from OSX. The full details of how such a system would work is beyond the scope of this story but the first step would be get an initial passing build and test suite for Trilinos for this build.
Currently the Trilinos PT packages don't even build for this MPI_RELEASE_DEBUG_SHARED_PT build and there were many test failures for some packages that did not have build failures.
To demonstrate, I ran:
$ ./checkin-test-sems.sh --enable-all-packages=on --local-do-all
on the Trilinos version:
*** Base Git Repo: Trilinos
c6eb35a [Tue Nov 29 16:18:17 2016 -0500] <rppawlo@sandia.gov>
Phalanx: update for kokkos TaskPolicy --> TaskScheduler
from an OSX machine that had the SEMS env mounted (gaia).
I then ran make dashbaord
in the MPI_RELEASE_DEBUG_SHARED_PT build directory and it uploaded the results to:
This shows 2 build failures in many Trilinos packages (I think CTest is reporting this incorrectly) but there was really only build failures in the STK package and the ROL package. The build failures in the STK package where in the library and therefore none of the STK tests were run. The build failure in the ROL package was for a single example that caused 58 tests not to run. The remaining 61 ROL tests all passed.
Other than these ROL test failures, there were test falures in the packages TrilinosCouplings, Stokhos, MueLu, and Zoltan shown here:
There is no sense listing out the tests in detail in this Issue but one can reproduce these failures for any package <PKG>
on OSX by checking out the Trilinos version c6eb35a1 (or the current version) and then running:
./checkin-test-sems.sh --enable-packages=<PKG> --local-do-all
If we want to allow Trilinos developers to push from OSX, then we need to clean up these failures and then we can talk about a sane and workable process to allow Trilinos developers to push from either Linux and OSX. It will be more complicated and will require things like a --compare-to-control-build
option for the checkin-test.py script but I think that is a very realistic goal.
Is there energy and interest to fix these problems on OSX?