Auto PR build failures due to Intel License server problems

Created by: bartlettroscoe

CC: @trilinos/framework

Expectations

Auto PR builds should be robust and not fail unless there is a failure in code itself.

Current Behavior

The new Intel auto PR build fails randomly due to Intel license server problems such as shown at:

• https://github.com/trilinos/Trilinos/pull/2860#issuecomment-393742327

which showed the build failure:

• https://testing-vm.sandia.gov/cdash/viewBuildError.php?buildid=3564874

which showed:

Error: A license for Comp-CL is not available now (-15,570,115).

A connection to the license server could not be made.  You should
make sure that your license daemon process is running: both an
lmgrd process and an INTEL process should be running
if your license limits you to a specified number of licenses in use
at a time.  Also, check to see if the wrong port@host or the wrong
license file is being used, or if the port or hostname in the license
file has changed.

License file(s) used were (in this order):
    1.  Trusted Storage
**  2.  /projects/sems/install/rhel6-x86_64/sems/compiler/intel/17.0.1/base/Licenses/intel-Linux-SRN.lic
**  3.  /projects/sems/install/rhel6-x86_64/sems/compiler/intel/17.0.1/base/compilers_and_libraries_2017.1.132/linux/bin/intel64/../../Licenses
**  4.  /ascldap/users/trilinos/Licenses

The reason that I noticed this was because it happened in my PR #2860.

Looking at the current PR build history at:

• https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=-test-Trilinos_pullrequest_intel_17.0.1&field2=buildstarttime&compare2=84&value2=2018-06-02&field3=buildstarttime&compare3=83&value3=2018-05-14

out of 10 builds, it failed twice with these Intel license server problems. That is an only 80% success rate so far. That is not robust enough for an auto PR build.

Motivation and Context

Auto PR builds block what goes into the 'develop' branch and long delays make things harder.

Definition of Done

Auto PR builds should only fail due to non-code issues very infrequently.

Possible Solution

Don't know.

Steps to Reproduce

Don't know.

Your Environment

N.A. This is the auto PR builds that define their own env.