Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2018-11-07T17:49:20Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3440Framework: Add link to PR Test Replication to Autotester message2018-11-07T17:49:20ZJames WillenbringFramework: Add link to PR Test Replication to Autotester message*Created by: william76*
@trilinos/framework @allevin
It's come up that it would be tremendously helpful to the developer community if a link were provided to the wiki page: https://github.com/trilinos/Trilinos/wiki/Reproducing-PR-Te...*Created by: william76*
@trilinos/framework @allevin
It's come up that it would be tremendously helpful to the developer community if a link were provided to the wiki page: https://github.com/trilinos/Trilinos/wiki/Reproducing-PR-Testing-Errors in the test-result message the autotester kicks out.
I think a good place for this would be right after the `CDash Test Results for PR# NNNN.` link. Something like:
```
CDash Test Results for PR# NNNN.
Wiki: How to Reproduce PR Testing Builds and Errors
```
This would be especially useful to folks who are infrequent committers who might not be as familiar with navigation around the CDash system, etc.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3435Framework: Autotester console output unhelpful2018-09-13T17:27:05ZJames WillenbringFramework: Autotester console output unhelpful*Created by: csiefer2*
Can we make this more useful to the developer?
See this PR:
https://github.com/trilinos/Trilinos/pull/3429
We get this unhelpful output:
17) atdm-ninja_fortran/1.7.2
MPI type = sems-mpich/3.2
CDash Track...*Created by: csiefer2*
Can we make this more useful to the developer?
See this PR:
https://github.com/trilinos/Trilinos/pull/3429
We get this unhelpful output:
17) atdm-ninja_fortran/1.7.2
MPI type = sems-mpich/3.2
CDash Track = Pull Request
*** Generating set of Trilinos enables given modified packages from
*** git commit origin/develop to HEAD
TRILINOS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos
TRILINOS_SCRIPTS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos/cmake/std/../..
TRIBITS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos/cmake/std/../../cmake/tribits
A) Generate the Trilinos Packages definition and depencencies XML file
Wrote the file 'TrilinosPackageDependencies.xml'
B) Get the set of changed files
Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos
git diff --name-only origin/develop..HEAD > /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/changed-files.txt
Wrote file 'changed-files.txt'
Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1
C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')
CHANGED_PACKAGES_FULL_LIST='MueLu'
D) Filter list of changed packages to get only the PT packages
CHANGED_PACKAGES_PT_LIST='MueLu'
E) Generate the *.cmake enables file
Wrote file 'packageEnables.cmake'
Enabled packages:
-- Setting Trilinos_ENABLE_MueLu = ON
Build name = PR-3429-test-Trilinos_pullrequest_intel_17.0.1-836
Cur dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/TFW_testing_single_configure_prototype
Source dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos
Binary dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/pull_request_test
Parallel level = 18
skip_by_parts_submit = OFF
skip_single_submit = ON
skip_update_step = ON
skip_upload_config_files = OFF
skip_clean_build_dir = OFF
Subproject count = 53
Dashboard model = Experimental
Dashboard track = Pull Request
Running configuration:
/projects/sems/install/rhel6-x86_64/atdm/binary-install/cmake-3.11.1-Linux-x86_64/bin/cmake
-C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos/cmake/std/PullRequestLinuxIntelTestingSettings.cmake"
-C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/packageEnables.cmake"
-DTrilinos_ENABLE_TESTS:BOOL=ON
-G "Ninja"
/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos
CTEST_DROP_LOCATION = /cdash/submit.php?project=Trilinos
CDash URL1 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=3&showfilters=1&filtercombine=and&field1=site&compare1=61&value1=ascic158&field2=buildname&compare2=61&value2=PR-3429-test-Trilinos_pullrequest_intel_17.0.1-836&field3=buildstamp&compare3=61&value3=20180912-0916-Pull Request
CDash URL2 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3429-test-Trilinos_pullrequest_intel_17.0.1-836&field2=buildstamp&compare2=61&value2=20180912-0916-Pull Request
CDash URL3 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3429-test-Trilinos_pullrequest_intel_17.0.1-836&field2=buildstamp&compare2=61&value2=20180912-0916-Pull Request
Starting configure step.
Each . represents 1024 bytes of output
.................................................. Size: 50K
.................................................. Size: 100K
.................................... Size of output: 135K
configure submit error = 0
Configure suceeded.
Starting build step.
Each symbol represents 1024 bytes of output.
.................................................. Size: 49K
.................................................. Size: 99K
.................................................. Size: 149K
.................................................. Size: 199K
.................................................. Size: 249K
.................................................. Size: 299K
.................................................. Size: 349K
.................................................. Size: 400K
.................................................. Size: 449K
.................................................. Size: 499K
.................................................. Size: 549K
.................................................. Size: 599K
.................................................. Size: 649K
.................................................. Size: 699K
.......... Size of output: 709K
Build succeeded.
build submit error = 0
Starting testing step.
CMake Error at /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/TFW_testing_single_configure_prototype/simple_testing.cmake:193 (message):
Test failed with error -1
test submit error = 0
File upload submit error = 0
Single configure/build/test failed. The error code was: 255
Build step 'Execute shell' marked build as failure
Finished: FAILUREhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3407Framework: Recent STK update breaks ROL's downward dependency testing2018-09-13T18:52:12ZJames WillenbringFramework: Recent STK update breaks ROL's downward dependency testing*Created by: dridzal*
@trilinos/framework @trilinos/stk @trilinos/panzer @trilinos/rol
## Expectations
Restore ROL's downward dependency testing.
## Current Behavior
https://testing-vm.sandia.gov/cdash/viewBuildError.php?buil...*Created by: dridzal*
@trilinos/framework @trilinos/stk @trilinos/panzer @trilinos/rol
## Expectations
Restore ROL's downward dependency testing.
## Current Behavior
https://testing-vm.sandia.gov/cdash/viewBuildError.php?buildid=3913799
Build error:
`/ascldap/users/dridzal/development/TEST/rol-trilinos/Trilinos/packages/stk/stk_util/stk_util/parallel/CommNeighbors.cpp: In member function ‘virtual ompi_communicator_t* stk::CommNeighbors::setup_neighbor_comm(stk::ParallelMachine, const std::vector<int>&, const std::vector<int>&)’: /ascldap/users/dridzal/development/TEST/rol-trilinos/Trilinos/packages/stk/stk_util/stk_util/parallel/CommNeighbors.cpp:81:30: error: ‘MPI_UNWEIGHTED’ was not declared in this scope const int* weights = (int*)MPI_UNWEIGHTED;
^
/ascldap/users/dridzal/development/TEST/rol-trilinos/Trilinos/packages/stk/stk_util/stk_util/parallel/CommNeighbors.cpp:86:43: error: ‘MPI_Dist_graph_create_adjacent’ was not declared in this scope info, reorder, &neighborComm); ^ /ascldap/users/dridzal/development/TEST/rol-trilinos/Trilinos/packages/stk/stk_util/stk_util/parallel/CommNeighbors.cpp: In member function ‘virtual void stk::CommNeighbors::perform_neighbor_communication(MPI_Comm, const std::vector<unsigned char>&, const std::vector<int>&, const std::vector<int>&, std::vector<unsigned char>&, std::vector<int>&, std::vector<int>&)’: /ascldap/users/dridzal/development/TEST/rol-trilinos/Trilinos/packages/stk/stk_util/stk_util/parallel/CommNeighbors.cpp:221:71: error: ‘MPI_Neighbor_alltoall’ was not declared in this scope (void*)recvCountsPtr, 1, MPI_INT, neighborComm); ^ /ascldap/users/dridzal/development/TEST/rol-trilinos/Trilinos/packages/stk/stk_util/stk_util/parallel/CommNeighbors.cpp:236:78: error: ‘MPI_Neighbor_alltoallv’ was not declared in this scope (void*)recvBufPtr, recvCountsPtr, recvDisplsPtr, MPI_BYTE, neighborComm); ^`
<!---
Tell us how the current behavior fails to meet your expectations in some way.
-->
## Possible Solution
I believe that MPI that is loaded for ROL's nightly testing (1.6.5??) no longer works for STK. This needs to be updated ASAP. I simply call
`$TRILINOS_DIR/cmake/load_sems_dev_env.sh "default"`
so the issue must be in load_sems_dev_env or the definition of "default". I wonder why others are not seeing this issue. Is the "default" abandoned or renamed? On the other hand, pre-push test scripts seem to work just fine ... how are they different?
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3373PR testing instability2018-11-15T20:22:39ZJames WillenbringPR testing instability*Created by: jwillenbring*
@trilinos/framework
## Expectations
<!---
Tell us what you think should happen, how you think things should work, what
you would like to see in the documentation, etc.
-->
We need PR testing fo...*Created by: jwillenbring*
@trilinos/framework
## Expectations
<!---
Tell us what you think should happen, how you think things should work, what
you would like to see in the documentation, etc.
-->
We need PR testing for Trilinos that runs reliably not only currently, but also when we continue to scale up the testing to include additional builds.
## Current Behavior
There are two primary causes of instability in the PR testing currently:
1) Communication issues
2) Machine overloading issues
We have seen a number of distinct types of communication issues. Here is a partial list:
a) clone/Fetch timeouts on GitHub
b) clone/fetch timeouts on internal gitlab-ex
c) failed reporting of results to CDash
d) Failure to communicate PR info from GitHub issues to autotester
e) Failure to communicate from autotester back to GitHub issues
Machine overloading issues have shown up in the form of internal compiler errors and test timeouts.
## Motivation and Context
<!---
How has this expectation failure affected you? What are you trying to
accomplish? Why do we need to address this? What does it have to do with
anything? Providing context helps us come up with a solution that is most
useful in the real world.
-->
We need to be able to run our PR testing in parallel (multiple PR testing at the same time and also multiple builds for each PR at the same time). If the environment won't scale, we will not be able to test PRs fast enough to make the system practical.
## Definition of Done
<!---
Tell us what needs to happen. If necessary, give us a task list along the
lines of:
- [ ] First do this.
- [ ] Then do that.
- [ ] Also this other thing.
-->
Communication errors occur on average less than one time per day. Machine overload issues occur less than twice a week.
## Possible Solution
<!---
Not obligatory, but suggest a fix for the bug or documentation, or suggest
ideas on how to implement the addition or change.
-->
We are looking into several possible solutions.
1) Reducing the -j for builds so we don't run into compiler errors
2) Reducing the number of available executors on testing nodes so that fewer tests can run at the same time
3) Speaking with networking staff about why operations are timing out
4) Reducing the number and/or frequency of PR testing instances to reduce the average communication traffic.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3360Trilinos PR testing needs a build with KOKKOS_ENABLE_DEPRECATED_CODE=OFF2018-08-27T17:35:56ZJames WillenbringTrilinos PR testing needs a build with KOKKOS_ENABLE_DEPRECATED_CODE=OFF*Created by: mhoemmen*
@trilinos/framework
Trilinos developers keep adding use of deprecated Kokkos functions back into Trilinos. See e.g., #3358. They do so because no PR test builds disable deprecated Kokkos functions, and becau...*Created by: mhoemmen*
@trilinos/framework
Trilinos developers keep adding use of deprecated Kokkos functions back into Trilinos. See e.g., #3358. They do so because no PR test builds disable deprecated Kokkos functions, and because Trilinos enables deprecated Kokkos functions by default. If we disable deprecated Kokkos functions in at least one PR test build, Trilinos developers will learn not to rely on those functions.
## Motivation and Context
See Sierra Ticket 19694.
## Possible Solution
In at least one PR test build, set KOKKOS_ENABLE_DEPRECATED_CODE=OFF.
## Related Issues
* Related to #3358 https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3347stk broken in clean Linux-gcc-4.8.4-MPI_Release_gcc_4.8.4_openmpi_1.8.7_DEV b...2018-09-04T13:32:57ZJames Willenbringstk broken in clean Linux-gcc-4.8.4-MPI_Release_gcc_4.8.4_openmpi_1.8.7_DEV build*Created by: prwolfe*
Looks like the recent import of stk (commit de635abc146449dc1c4dbbea65cefda91946755e) caused failures in the Linux-gcc-4.8.4-MPI_Release_gcc_4.8.4_openmpi_1.8.7_DEV build.
<!---
Provide a general summary of the...*Created by: prwolfe*
Looks like the recent import of stk (commit de635abc146449dc1c4dbbea65cefda91946755e) caused failures in the Linux-gcc-4.8.4-MPI_Release_gcc_4.8.4_openmpi_1.8.7_DEV build.
<!---
Provide a general summary of the issue in the Title above. If this issue
pertains to a particular package in Trilinos, it's worthwhile to start the
title with "PackageName: ".
-->
Gcc 4.8.4 does not support brace-list initialization which is used in one of the unit tests. This was turned off in the PR testing, but looks like it was missed in this build. See commit 465f04753fcb8e6bf7047cc03d1cce54e254d25f for details.
@trilinos/framework
@trilinos/stk
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3346SEACAS not included files...in neither src directory nor install directory2018-11-30T03:12:47ZJames WillenbringSEACAS not included files...in neither src directory nor install directory*Created by: kyungjoo-kim*
<!---
Provide a general summary of the issue in the Title above. If this issue
pertains to a particular package in Trilinos, it's worthwhile to start the
title with "PackageName: ".
-->
@trilinos/seacas...*Created by: kyungjoo-kim*
<!---
Provide a general summary of the issue in the Title above. If this issue
pertains to a particular package in Trilinos, it's worthwhile to start the
title with "PackageName: ".
-->
@trilinos/seacas
After I installed Trilinos with SEACAS and compile a code against them, I have cmake error reporting following packages are missing.
```
CMake Error at /home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/SEACASConfig.cmake:146 (INCLUDE):
INCLUDE could not find load file:
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASTxtexo/SEACASTxtexoConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASNumbers/SEACASNumbersConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASMapvar-kd/SEACASMapvar-kdConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASMapvar/SEACASMapvarConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASMapvarlib/SEACASMapvarlibConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASExplore/SEACASExploreConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASGrepos/SEACASGreposConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASGenshell/SEACASGenshellConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASGen3D/SEACASGen3DConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASGjoin/SEACASGjoinConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASEx1ex2v2/SEACASEx1ex2v2Config.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASExotxt/SEACASExotxtConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASAlgebra/SEACASAlgebraConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASSuplib/SEACASSuplibConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASSupes/SEACASSupesConfig.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASExoIIv2for32/SEACASExoIIv2for32Config.cmake
/home/kyukim/Work/lib/trilinos/install/white/empire/release-k80-cuda-9.2/lib/cmake/SEACAS/../SEACASExodus_for/SEACASExodus_forConfig.cmake
```
CMakeCache output indicates SEACAS has dependency on those packages
```
SEACAS_FULL_ENABLED_DEP_PACKAGES:INTERNAL=SEACASTxtexo;SEACASNumbers;SEACASNemspread;SEACASNemslice;SEACASMapvar-kd;SEACASMapvar;SEACASMapvarlib;SEACASExplore;SEACASGrepos;SEACASGenshell;SEACASGen3D;SEACASGjoin;SEACASEx1ex2v2;SEACASExo_\
format;SEACASExotxt;SEACASExomatlab;SEACASExodiff;SEACASEpu;SEACASEjoin;SEACASConjoin;SEACASAprepro;SEACASAlgebra;SEACASSuplibCpp;SEACASSuplibC;SEACASSuplib;SEACASSupes;SEACASAprepro_lib;SEACASChaco;SEACASIoss;SEACASNemesis;SEACASExoIIv\
2for32;SEACASExodus_for;SEACASExodus;Pamgen;Zoltan;Kokkos;KokkosAlgorithms;KokkosContainers;KokkosCore
```
## Expectations
<!---
Tell us what you think should happen, how you think things should work, what
you would like to see in the documentation, etc.
-->
Trilinos cmake system should be compiled with other cmake system used by applications. The dependence on missing packages should be configured correctly.
## Current Behavior
<!---
Tell us how the current behavior fails to meet your expectations in some way.
-->
cmake looks up missing packages.
## Motivation and Context
<!---
How has this expectation failure affected you? What are you trying to
accomplish? Why do we need to address this? What does it have to do with
anything? Providing context helps us come up with a solution that is most
useful in the real world.
-->
This does not compile with other applications that rely on cmake e.g., empire.
## Definition of Done
Corrrect cmake dependence.
- check install directory ``${INSTALL_DIR}/lib/cmake/SEACASConfig.cmake``
- check if it does not include ``INCLUDE("${CMAKE_CURRENT_LIST_DIR}/../SEACASTxtexo/SEACASTxtexoConfig.cmake") `` as well as all other packages listed above.
- or check CMakeCache does not include dependency on the packages above.
## Steps to Reproduce
Configure Trilinos with SEACAS
```
cmake \
-D BUILD_SHARED_LIBS:BOOL=${SHARED_LIBS} \
-D CMAKE_BUILD_TYPE:STRING=${BUILD_TYPE} \
-D CMAKE_C_COMPILER:FILEPATH="mpicc" \
-D CMAKE_C_FLAGS:STRING="${EXTRA_C_FLAGS}" \
-D CMAKE_CXX_COMPILER:FILEPATH="mpicxx" \
-D CMAKE_CXX_FLAGS:STRING="${EXTRA_CXX_FLAGS}" \
-D CMAKE_Fortran_COMPILER:FILEPATH="mpif77" \
-D CMAKE_Fortran_FLAGS:STRING="${EXTRA_F77_FLAGS}" \
-D CMAKE_SKIP_RULE_DEPENDENCY:BOOL=ON \
-D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \
-D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
-D Trilinos_CXX11_FLAGS:STRING="-std=c++11" \
-D Trilinos_ENABLE_INSTALL_CMAKE_CONFIG_FILES:BOOL=ON \
-D Trilinos_VERBOSE_CONFIGURE:BOOL=OFF \
-D Trilinos_ENABLE_DEBUG:BOOL=OFF \
-D Trilinos_ENABLE_EXAMPLES:BOOL=${EXAMPLE} \
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-D Trilinos_ENABLE_Fortran:BOOL=OFF \
-D Trilinos_ENABLE_STRONG_CXX_COMPILE_WARNINGS:BOOL=OFF \
-D Trilinos_ENABLE_STRONG_C_COMPILE_WARNINGS:BOOL=OFF \
-D Trilinos_ENABLE_SHADOW_WARNINGS:BOOL=OFF \
-D Trilinos_ENABLE_TESTS:BOOL=${TEST} \
-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_OpenMP:BOOL=${USE_OPENMP} \
-D Trilinos_ENABLE_Amesos:BOOL=ON \
-D Trilinos_ENABLE_Amesos2:BOOL=ON \
-D Trilinos_ENABLE_Belos:BOOL=ON \
-D Trilinos_ENABLE_KokkosCore:BOOL=ON \
-D Trilinos_ENABLE_KokkosAlgorithms:BOOL=ON \
-D Trilinos_ENABLE_ML:BOOL=ON \
-D Trilinos_ENABLE_MueLu:BOOL=ON \
-D Trilinos_ENABLE_Panzer:BOOL=ON \
-D Trilinos_ENABLE_Pamgen:BOOL=ON \
-D Trilinos_ENABLE_SEACAS:BOOL=ON \
-D Trilinos_ENABLE_SEACASIoss:BOOL=ON \
-D Trilinos_ENABLE_SEACASEx2ex1v2:BOOL=OFF \
-D Trilinos_ENABLE_Shards:BOOL=ON \
-D Trilinos_ENABLE_STK:BOOL=ON \
-D Trilinos_ENABLE_STKMesh:BOOL=ON \
-D Trilinos_ENABLE_STKUtil:BOOL=ON \
-D Trilinos_ENABLE_STKTopology:BOOL=ON \
-D Trilinos_ENABLE_STKSearch:BOOL=OFF \
-D Trilinos_ENABLE_STKTransfer:BOOL=OFF \
-D Trilinos_ENABLE_STKDoc_tests:BOOL=OFF \
-D Trilinos_ENABLE_STKUnit_tests:BOOL=OFF \
-D Trilinos_ENABLE_STKUnit_test_utils:BOOL=OFF \
-D Trilinos_ENABLE_Stokhos:BOOL=OFF \
-D Trilinos_ENABLE_Stratimikos:BOOL=ON \
-D Trilinos_ENABLE_Teko:BOOL=ON \
-D Trilinos_ENABLE_Tempus:BOOL=ON \
-D Trilinos_ENABLE_Zoltan:BOOL=ON \
-D Trilinos_ENABLE_Zoltan2:BOOL=ON \
-D EpetraExt_ENABLE_HDF5:BOOL=OFF \
-D Kokkos_ENABLE_OpenMP:BOOL=${USE_OPENMP} \
-D Kokkos_ENABLE_Pthread:BOOL=${USE_PTHREADS} \
-D Kokkos_ENABLE_Cuda_UVM:BOOL=${USE_CUDA} \
-D Kokkos_ENABLE_Cuda_Lambda:BOOL=${USE_CUDA} \
-D Kokkos_ENABLE_Debug_Bounds_Check:BOOL=${BOUNDS_CHECK} \
-D MueLu_ENABLE_Experimental:BOOL=ON \
-D Panzer_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-D Panzer_ENABLE_TESTS:BOOL=OFF \
-D Phalanx_KOKKOS_DEVICE_TYPE:STRING="${NODE_TYPE}" \
-D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
-D SEACASExodus_ENABLE_MPI:BOOL=ON \
-D TeuchosCore_ENABLE_yaml-cpp:BOOL=ON \
-D Xpetra_ENABLE_Experimental:BOOL=ON \
-D TPL_ENABLE_GLM:BOOL=OFF \
-D TPL_ENABLE_Matio:BOOL=OFF \
-D TPL_ENABLE_SuperLU:BOOL=OFF \
-D TPL_ENABLE_X11:BOOL=OFF \
-D TPL_ENABLE_MPI:BOOL=ON \
-D TPL_ENABLE_BLAS:BOOL=ON \
-D TPL_BLAS_LIBRARIES:FILEPATH="-lblas" \
-D TPL_ENABLE_Boost:BOOL=ON \
-D Boost_INCLUDE_DIRS:FILEPATH="${BOOST_ROOT}/include" \ \
-D Boost_LIBRARY_DIRS:FILEPATH="${BOOST_ROOT}/lib" \
-D TPL_ENABLE_BoostLib:BOOL=ON \
-D BoostLib_INCLUDE_DIRS:FILEPATH="${BOOST_ROOT}/include" \
-D BoostLib_LIBRARY_DIRS:FILEPATH="${BOOST_ROOT}/lib" \
-D TPL_ENABLE_CUDA:BOOL=${USE_CUDA} \
-D TPL_ENABLE_CUSPARSE:BOOL=${USE_CUDA} \
-D TPL_ENABLE_LAPACK:BOOL=ON \
-D TPL_LAPACK_LIBRARIES:FILEPATH="-llapack" \
-D TPL_ENABLE_HDF5:BOOL=ON \
-D HDF5_INCLUDE_DIRS:FILEPATH="${HDF5_ROOT}/include" \
-D HDF5_LIBRARY_DIRS:FILEPATH="${HDF5_ROOT}/lib" \
-D TPL_HDF5_LIBRARIES:FILEPATH="${HDF5_LIBS}" \
-D TPL_ENABLE_Netcdf:BOOL=ON \
-D Netcdf_INCLUDE_DIRS:FILEPATH="${NETCDF_ROOT}/include" \
-D TPL_Netcdf_LIBRARIES:FILEPATH="${NETCDF_LIBS};${NETCDF_LIBS}" \
-D TPL_ENABLE_yaml-cpp:BOOL=ON \
-D yaml-cpp_LIBRARY_DIRS="${YAMLCPP_ROOT}/lib" \
-D KOKKOS_ARCH="Power8;Kepler37" \
$TRILINOS_DIR
```
## Your Environment
on WHITE
```
module load devpack/20180521/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88
module load yamlcpp/0.5.3/gcc/7.2.0
```
<!---
Include relevant details about your environment such that we can replicate this
issue.
-->
- **Relevant repo SHA1s:**
- **Relevant configure flags or configure script:**
- **Operating system and version:**
- **Compiler and TPL versions:**
Keep promoted "ATDM" builds of Trilinos cleanhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3306Failure of ATDM build script on waterman2019-01-09T23:48:45ZJames WillenbringFailure of ATDM build script on waterman*Created by: keitat*
I am seeing the latest version of ATDM build script is not working on waterman.
## Expectations
I am following the instruction of budding on internal machines, but it's not working.
## Current Behavior
[kn...*Created by: keitat*
I am seeing the latest version of ATDM build script is not working on waterman.
## Expectations
I am following the instruction of budding on internal machines, but it's not working.
## Current Behavior
[knteran@waterman11 buildTrilinos]$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh kt_cuda
Error, hostname = 'waterman11' not recognized as a known ATDM system name!
## Steps to Reproduce
```
module load devpack/20180517/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88
export TRILINOS_DIR=your directory
source $TRILINOS_DIR/cmake/std/atdm/load-env.sh job name
```
## Your Environment
```
[knteran@waterman11 buildTrilinos]$ module list
Currently Loaded Modulefiles:
1) binutils/2.30.0 8) valgrind/3.12.0 15) pnetcdf-exo/1.9.0/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88
2) gcc/7.2.0 9) openblas/0.2.20/gcc/7.2.0 16) netcdf-exo/4.6.1/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88
3) cuda/9.2.88 10) metis/5.0.1/gcc/7.2.0 17) parmetis/4.0.3/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88
4) java/ibm/sdk/8.0.0 11) zlib/1.2.8 18) boost/1.65.1/gcc/7.2.0
5) numa/2.0.11 12) hdf5/1.8.20/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88 19) superlu-dist/4.3.0/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88
6) openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88 13) cgns/20180517/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88 20) devpack/20180517/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88
7) cmake/3.6.2 14) matio/1.5.11/gcc/7.2.0
```https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3285Transition to new site testing-vm.sandia.gov/cdash/2018-08-30T18:09:35ZJames WillenbringTransition to new site testing-vm.sandia.gov/cdash/*Created by: bartlettroscoe*
CC: @trilinos/framework
## Next Action Status
PR #3305 merged on 8/16/2018 that submits only to testing-vm.sandia.gov/cdash/. Redirect from https://testing.sandia.gov/cdash/ to https://testing-vm.san...*Created by: bartlettroscoe*
CC: @trilinos/framework
## Next Action Status
PR #3305 merged on 8/16/2018 that submits only to testing-vm.sandia.gov/cdash/. Redirect from https://testing.sandia.gov/cdash/ to https://testing-vm.sandia.gov/cdash/ completed on 8/30/2018. Next: Change all wiki references of https://testing-vm.sandia.gov/cdash/ to https://testing.sandia.gov/cdash/?
## Description
This Story is to plan and track efforts to switch from the existing testing.sandia.gov/cdash/ site to the new just approved testing-vm.sandia.gov/cdash/ site (see https://gitlab.kitware.com/snl/project-1/issues/33). The machine testing.sandia.gov is old and needs to be retired and the new site testing-vm.sandia.gov/cdash/ supports all-at-once configure/build/test/submit and lots of other improvements and bug fixes.
## Tasks
* [x] Test and approve testing-vm.sandia.gov/cdash/ (see https://gitlab.kitware.com/snl/project-1/issues/33)
* [x] Turn on CDash emails from testing-vm.sandia.gov/cdash/
* [x] Send out email to trilinos-developers warning about the transition and stopping of sending CDash data to testing.sandia.gov shortly.
* [x] Submit and merge to 'develop' a PR that stops dual submits to testing.sandia.gov/cdash/ by updating `Trilinos/CTestConfig.cmake`.
* [x] Update Trilinos wiki pages to change from testing.sandia.gov to testing-vm.sandia.gov (can do this with a git clone of the wiki repo).
* [x] Update 'master' from 'develop' with the updated `Trilinos/CTestConfig.cmake` file.
* [x] Wait a day or more and verify that no data is submitted to testing.sandia.gov/cdash/ over a full 24-hour testing day period (to any CDash project on that site). (Or verify that no dual submits will occur after putting in redirect testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/.)
* [x] Transition to testing-vm.sandia.gov/cdash/ and turn off testing.sandia.gov/cdash/ (all together)
* [x] Set up a URL redirect from testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/ (effectively taking testing.sandia.gov offline)
* [x] Send out email to Trilinos developers announcing the transition is complete
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3276Trilinos auto PR tester stability issues2019-05-02T13:20:11ZJames WillenbringTrilinos auto PR tester stability issues*Created by: bartlettroscoe*
@trilinos/framework
## Description
Over the last few weeks and months, the Trilinos auto PR tester has seen several cases where one or more PR builds for a given PR testing iteration failed to produce ...*Created by: bartlettroscoe*
@trilinos/framework
## Description
Over the last few weeks and months, the Trilinos auto PR tester has seen several cases where one or more PR builds for a given PR testing iteration failed to produce results on CDash or showed build or test failures that were not related to the changes on that particular PR.
This Story is to log these fails and keep track of them in order to provide some statistics about these cases in order to inform how to address them. This should replace making comments in individual PRs that exhibit these types of problems like #3260 and #3213.
## PR Builds Showing Random Failures
Below are a few examples of the stability problems (but are not all of the problems).
| PR ID | Num PR Builds to reach passing | First test trigger | Start first test| Passing test | Merge PR |
| --: | --: | --: | --: | --: | --: |
| #3258 | 2 | [8/8/2018 2:35 PM ET](https://github.com/trilinos/Trilinos/pull/3258#issue-207098955) | [8/8/2018 2:44 PM](https://github.com/trilinos/Trilinos/pull/3258#issuecomment-411510956) | [8/8/2018 9:15 PM ET]() | Not merged |
| #3260 | 4 | [8/8/2018 5:22 PM ET](https://github.com/trilinos/Trilinos/pull/3260#issue-207141537) | [8/8/2018 6:31 PM ET](https://github.com/trilinos/Trilinos/pull/3260#issuecomment-411574370) | [8/10/2018 4:13 AM ET](https://github.com/trilinos/Trilinos/pull/3260#issuecomment-412010497) | [8/10/2018 8:25 AM](https://github.com/trilinos/Trilinos/pull/3260#event-1782381644) |
| #3213 | 3 | [7/31/2018 4:30 PM ET](https://github.com/trilinos/Trilinos/pull/3213#issue-205233060) | [7/31/2018 4:57 PM ET](https://github.com/trilinos/Trilinos/pull/3213#issuecomment-409365522) | [8/1/2018 9:48 AM ET](https://github.com/trilinos/Trilinos/pull/3213#issuecomment-409580677) | [8/1/2018 9:53 AM ET](https://github.com/trilinos/Trilinos/pull/3213#event-1765281809) |
| #3098 | 4 | [7/12/2018 12:52 PM ET](https://github.com/trilinos/Trilinos/pull/3098#issue-201063953) | [7/12/2018 1:07 PM ET](https://github.com/trilinos/Trilinos/pull/3098#issuecomment-404582631) | [7/13/2018 11:12 PM ET](https://github.com/trilinos/Trilinos/pull/3098#issuecomment-404994581) | [7/14/2018 10:59 PM ET](https://github.com/trilinos/Trilinos/pull/3098#event-1733896640) |
| #3369 | 6 | [8/29/2018 9:08 AM ET](https://github.com/trilinos/Trilinos/pull/3369#issue-211746901) | [8/29/2018 9:16 AM ET](https://github.com/trilinos/Trilinos/pull/3369#issuecomment-416948915) | [8/31/2018 6:09 AM ET](https://github.com/trilinos/Trilinos/pull/3369#issuecomment-417618824) | [8/31/2018 8:33 AM ET](https://github.com/trilinos/Trilinos/pull/3369#event-1820478271) |
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3272package_subproject_list.cmake static in Trilinos PR tester?2018-08-09T19:58:47ZJames Willenbringpackage_subproject_list.cmake static in Trilinos PR tester?*Created by: bartlettroscoe*
@trilinos/framework,
Is the file package_subproject_list.cmake used at:
https://github.com/trilinos/Trilinos/blob/0df591fd6d9b9ccd91bd2695cc22665aa468886d/cmake/std/PullRequestLinuxDriver.sh#L206
st...*Created by: bartlettroscoe*
@trilinos/framework,
Is the file package_subproject_list.cmake used at:
https://github.com/trilinos/Trilinos/blob/0df591fd6d9b9ccd91bd2695cc22665aa468886d/cmake/std/PullRequestLinuxDriver.sh#L206
statically generated and require manual updating? If so, that would explain why the PR builds are not testing the new PT package TrilinosFrameworkTests.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3265Problems with autotester SHA1s passed into get-changed-trilinos-packages.sh i...2018-09-13T18:50:18ZJames WillenbringProblems with autotester SHA1s passed into get-changed-trilinos-packages.sh in Trilinos auto-tester*Created by: bartlettroscoe*
@trilinos/framework, @william76
## Next Action Status
PR #3258 merged on 8/14/2018 which should fix this and this seems to be fixed as noted in closed #3133.
## Description
There seems to be a ...*Created by: bartlettroscoe*
@trilinos/framework, @william76
## Next Action Status
PR #3258 merged on 8/14/2018 which should fix this and this seems to be fixed as noted in closed #3133.
## Description
There seems to be a problem with the Trilinos autotester in passing in the right range of commits to the `get-changed-trilinos-packages.sh` script (see #3133 and #3218). The evidence for this is the PR testing iteration for PR #3260 which only changes files under the `cmake/std/atdm/` directory yet it triggered the enable of several Trilinos packages as shown in [this PR iteration this morning](https://testing-vm.sandia.gov/cdash/viewConfigure.php?buildid=3815005) which shows:
```
loading initial cache file /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/packageEnables.cmake
-- Setting Trilinos_ENABLE_ALL_PACKAGES = ON
-- Setting Trilinos_ENABLE_Belos = ON
-- Setting Trilinos_ENABLE_Ifpack2 = ON
-- Setting Trilinos_ENABLE_Piro = ON
-- Setting Trilinos_ENABLE_ROL = ON
-- Setting Trilinos_ENABLE_TpetraCore = ON
```
My guess is that the Python script is passing in the git SHA1 for the tip of the 'develop' branch instead of the base commit that the topic branch is created off of.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3263Test ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1 randomly failing in CI and...2018-08-09T19:59:42ZJames WillenbringTest ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1 randomly failing in CI and PR GCC 4.8.4 + OpenMP builds*Created by: bartlettroscoe*
@trilinos/shylu, @trilinos/framework, @srajama1 (Trilinos Linear Solvers Product Lead)
## Expectations
A test should not fail unless a changes is made to break it. A test should not randomly fail.
...*Created by: bartlettroscoe*
@trilinos/shylu, @trilinos/framework, @srajama1 (Trilinos Linear Solvers Product Lead)
## Expectations
A test should not fail unless a changes is made to break it. A test should not randomly fail.
## Current Behavior
Looking at the four most recent failures of the test `ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1` in [this query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=testname&compare1=61&value1=ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1&field2=status&compare2=62&value2=passed&field3=status&compare3=62&value3=notrun&field4=buildstarttime&compare4=84&value4=now) the test appears to be randomly failing in the GCC 4.8.4 OpenMPI builds. In the most recent case, this test broke the auto PR GCC 4.8.4 + OpenMP build in PR #3260. IN each of the last for failures of this test dating back to 6/28/2018, they all fail showing:
```
....
[ RUN ] CrsMatrixBase.matrixmarket
unknown file: Failure
C++ exception with description "View bounds error of view ap ( 13 < 13 )
Traceback functionality not available
" thrown in the test body.
[ FAILED ] CrsMatrixBase.matrixmarket (23 ms)
...
[ FAILED ] 1 test, listed below:
[ FAILED ] CrsMatrixBase.matrixmarket
1 FAILED TEST
```
<!---
Tell us how the current behavior fails to meet your expectations in some way.
-->
## Motivation and Context
<!---
How has this expectation failure affected you? What are you trying to
accomplish? Why do we need to address this? What does it have to do with
anything? Providing context helps us come up with a solution that is most
useful in the real world.
-->
## Definition of Done
The test `ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1` is fixed to make it so that it does not randomly fail or is removed for CI and auto PR testing.
## Possible Solution
Fix it so that it does not randomly fail or remove it from CI and auto PR testing.
## Steps to Reproduce
See https://github.com/trilinos/Trilinos/wiki/Reproducing-PR-Testing-Errors.
## Your Environment
Standard SEMS GCC 4.8.4 auto PR build env (see above).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3178Pull request testing should set -Werror2019-05-02T22:07:04ZJames WillenbringPull request testing should set -Werror*Created by: mhoemmen*
@vbrunini asks whether pull request testing could set `-Werror`, so as to avoid issues like #3177.
@trilinos/framework @khpierson
## Expectations
Trilinos -- at least the library, not necessarily tests a...*Created by: mhoemmen*
@vbrunini asks whether pull request testing could set `-Werror`, so as to avoid issues like #3177.
@trilinos/framework @khpierson
## Expectations
Trilinos -- at least the library, not necessarily tests and examples -- should build warning-free.
## Current Behavior
See #3177. There is an issue that it's impossible to fix warnings in some packages.
## Motivation and Context
Sierra builds with warnings as errors, so they want Trilinos to build warning-free.
## Possible Solution
Exclude legacy packages like ML. Fix warnings. Add `-Werror` to at least one PR build.
## Related Issues
* Related to #3177 Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3175Framework: Windows Testing is Down Again2018-07-24T14:39:28ZJames WillenbringFramework: Windows Testing is Down Again*Created by: csiefer2*
That.*Created by: csiefer2*
That.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3133Get Trilinos auto PR testing code to use TriBITS-based package enable logic2018-09-13T18:49:19ZJames WillenbringGet Trilinos auto PR testing code to use TriBITS-based package enable logic*Created by: bartlettroscoe*
CC: @william76, @jwillenbring
## Next Action Status
With the merge of PR #3258 8/14/2018 this should be complete. As of 9/12/2018 no problems have been reported in last 4+ weeks so all is good!
...*Created by: bartlettroscoe*
CC: @william76, @jwillenbring
## Next Action Status
With the merge of PR #3258 8/14/2018 this should be complete. As of 9/12/2018 no problems have been reported in last 4+ weeks so all is good!
## Description
This story is to get the Trilinos auto PR testing code to use TriBITS-based logic for determining what to enable and test based on files that are changed in the PR branch.
The benefits of making this change are:
* When the set of Primary Tested packages changes, the code will automatically adjust
* Specialized logic can be applied to determine if all packages need to be tested or not. For example, if files are changed under `Trilinos/cmake/ctest/drivers` then that should not trigger the auto PR tester to test all of Trilinos.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3120Weird MPI run-time error in Dashboard build using ancient OpenMPI version2019-04-01T21:46:38ZJames WillenbringWeird MPI run-time error in Dashboard build using ancient OpenMPI version*Created by: mhoemmen*
@trilinos/tpetra @trilinos/framework
I noticed that Tpetra's "ReadTriples" test showed a failure on one Dashboard build:
https://testing-vm.sandia.gov/cdash/testDetails.php?test=50492359&build=3725219
Th...*Created by: mhoemmen*
@trilinos/tpetra @trilinos/framework
I noticed that Tpetra's "ReadTriples" test showed a failure on one Dashboard build:
https://testing-vm.sandia.gov/cdash/testDetails.php?test=50492359&build=3725219
The failure looks spurious, possibly due to a full `/tmp` filesystem:
```
[ascic114:11890] opal_os_dirpath_create: Error: Unable to create the sub-directory (/tmp/openmpi-sessions-trilinos@ascic114_0/49948) of (/tmp/openmpi-sessions-trilinos@ascic114_0/49948/0/0), mkdir failed [1]
[ascic114:11890] [[49948,0],0] ORTE_ERROR_LOG: Error in file util/session_dir.c at line 107
[ascic114:11890] [[49948,0],0] ORTE_ERROR_LOG: Error in file util/session_dir.c at line 402
[ascic114:11890] [[49948,0],0] ORTE_ERROR_LOG: Error in file ess_hnp_module.c at line 638
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
orte_session_dir failed
--> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
```
I also noticed that the build uses OpenMPI 1.8.7. @prwolfe and/or @rrdrake reported to me that Sierra skipped over that version of OpenMPI in favor of 1.10.x, because 1.8.y caused tests to fail. 1.8 is also a retired version of OpenMPI. (So is 1.10, but at least it's a bit newer.) We should get rid of that build or update the OpenMPI version.
## Expectations
Nobody tests with OpenMPI versions that old. OpenMPI doesn't support them, and neither Sierra nor ATDM apps use them.
## Possible Solution
1. Eliminate that build, or
2. update its OpenMPI version.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3118Kokkos CMake error (fixed in kokkos:develop)2018-07-16T15:37:16ZJames WillenbringKokkos CMake error (fixed in kokkos:develop)*Created by: mhoemmen*
@trilinos/kokkos @trilinos/framework
## Current Behavior
At least one Trilinos build on the Dashboard shows the following CMake error:
```
CMake Error at packages/kokkos/cmake/kokkos_functions.cmake:50 (s...*Created by: mhoemmen*
@trilinos/kokkos @trilinos/framework
## Current Behavior
At least one Trilinos build on the Dashboard shows the following CMake error:
```
CMake Error at packages/kokkos/cmake/kokkos_functions.cmake:50 (string):
Syntax error in cmake code at
/home/jenkins/slave/workspace/Trilinos_apollo_gcc_4.9.3_cuda_8.0.44/MPI_RELEASE_DEV_DownStream_ETI_SERIAL-ON_OPENMP-OFF_PTHREAD-OFF_CUDA-ON_COMPLEX-OFF/Trilinos/packages/kokkos/cmake/kokkos_functions.cmake:50
when parsing string
[0-9]+\.[0-9]+\.[0-9]+$
Invalid escape sequence \.
Call Stack (most recent call first):
cmake/ProjectCompilerPostConfig.cmake:34 (set_kokkos_cxx_compiler)
/home/jenkins/slave/workspace/Trilinos_apollo_gcc_4.9.3_cuda_8.0.44/Trilinos/cmake/tribits/core/package_arch/TribitsGlobalMacros.cmake:1847 (INCLUDE)
/home/jenkins/slave/workspace/Trilinos_apollo_gcc_4.9.3_cuda_8.0.44/Trilinos/cmake/tribits/core/package_arch/TribitsProjectImpl.cmake:188 (TRIBITS_SETUP_ENV)
/home/jenkins/slave/workspace/Trilinos_apollo_gcc_4.9.3_cuda_8.0.44/Trilinos/cmake/tribits/core/package_arch/TribitsProject.cmake:93 (TRIBITS_PROJECT_IMPL)
CMakeLists.txt:90 (TRIBITS_PROJECT)
```
This is https://github.com/kokkos/kokkos/issues/1661 , which has been fixed in kokkos:develop by Kokkos PR https://github.com/kokkos/kokkos/pull/1662 .
## Possible Solution
Wait until the next Kokkos promotion into Trilinos.
## Related Issues
* Related to https://github.com/kokkos/kokkos/issues/1661https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3106Failing ShyLU_DD tests in GCC 4.8.4 OpenMP CI and PR builds somehow pushed to...2018-07-24T20:04:19ZJames WillenbringFailing ShyLU_DD tests in GCC 4.8.4 OpenMP CI and PR builds somehow pushed to 'develop' branch*Created by: bartlettroscoe*
@trilinos/framework, @trilinos/shylu, @searhein, @srajama1 (Trilinos Linear Solvers Product Lead)
The post-push CI build `Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI` shown [here](https://testin...*Created by: bartlettroscoe*
@trilinos/framework, @trilinos/shylu, @searhein, @srajama1 (Trilinos Linear Solvers Product Lead)
The post-push CI build `Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI` shown [here](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=-MPI_RELEASE_DEBUG_SHARED_PT&field2=groupname&compare2=61&value2=Continuous&field3=buildstarttime&compare3=84&value3=now&field4=buildstarttime&compare4=83&value4=4%20weeks%20ago) failed this morning for the first time in [this CI iteration](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3719977&filtercount=4&showfilters=1&field1=groupname&compare1=61&value1=Continuous&field2=buildstarttime&compare2=84&value2=now&field3=buildstarttime&compare3=83&value3=4%20weeks%20ago&filtercombine=and) due to the failing tests shown [here](https://testing-vm.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=3719977) which were:
|Test Name | Status | Test Time | Details |
| :-- | :-- | :-- | :-- |
| ShyLU_DDFROSch_test_laplacian_epetra_2d_gdsw_MPI_4 | Failed | 1s 170ms | 4s 680ms | Completed (Failed) |
| ShyLU_DDFROSch_test_laplacian_epetra_2d_rgdsw_MPI_4 | Failed | 1s 170ms | 4s 680ms | Completed (Failed) |
| ShyLU_DDFROSch_test_interfacesets_2D_MPI_4 | Failed | 710ms | 2s 840ms | Completed (Failed) |
|ShyLU_DDFROSch_test_interfacepartitionofunity_MPI_4 | Failed | 700ms | 2s 800ms | Completed (Failed) |
Looking at the commits pulled this first CI iteration [here](https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3720229#!#note2) it seems likely these failures are due to one of the commits by `Alexander Heinlein <alexander.heinlein@uni-köln.de>` (@searhein).
These failing ShyLU_DD tests did not only impact the post-push CI OpenMP build above but it also impacted the auto PR OpenMP build `PR-<pr-id>-test-Trilinos_pullrequest_gcc_4.8.4-<build-num>`. For example, you can see this in the PR builds:
* [PR-3098-test-Trilinos_pullrequest_gcc_4.8.4-781](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3719867)
* [PR-3053-test-Trilinos_pullrequest_gcc_4.8.4-782](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3720319)
If you look, it is the same 4 failing tests. If you look at those PR #3098 the changes in that branch don't impact ShyLU_DD or the PR builds in any way.
https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3720229#!#note2https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3105Random timeouts in auto PR builds2018-07-13T14:33:40ZJames WillenbringRandom timeouts in auto PR builds*Created by: bartlettroscoe*
@trilinos/framework
## Expectations
Tests will not fail or timeout unless changes in a PR branch cause the failures or timeouts.
## Current Behavior
Several auto PR build iterations have been fa...*Created by: bartlettroscoe*
@trilinos/framework
## Expectations
Tests will not fail or timeout unless changes in a PR branch cause the failures or timeouts.
## Current Behavior
Several auto PR build iterations have been failing for a while due to random timeouts in various auto PR builds. For example, the first PR testing iteration in my PR #3104 failed last night due to random timeouts as shown [here](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=PR-3104-test&field2=buildstarttime&compare2=84&value2=NOW). It is impossible for that one change to have impacted these test timeouts in any way.
You can see timeouts impacting other PR testing iterations as well in just the last 12 days in [this query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-07-12&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildstarttime&compare1=83&value1=2018-07-01&field2=groupname&compare2=61&value2=Pull%20Request&field3=details&compare3=63&value3=timeout). This shows 12 tests timing out in 3 different PRs over the last 12 days. Looking at the numbering of the builds, it seems likely that up to 4 or 5 PR testing iterations failed due to this issue. (It is hard to know how many PR testing iterations failed due to these random timeouts due to the naming of the PR builds since it is hard to differentiate different PR testing iterations in the same PR or to match up which builds of different compilers match up to the same PR testing iteration). Also, it is possible that some of these timeouts were due to changes in the PR branch.
## Motivation and Context
We want PR testing iterations to only fail if they are triggered by changes in the specific topic branch being tested.
While 4 or 5 failed PR testing iterations over 12 days may not seem like a lot, when combined with randomly failing tests (see #3103) and randomly failures in the Jenkins jobs (git pull fails, etc.), these add up to make the auto PR testing pretty unstable.
## Definition of Done
* Eliminate randomly failing tests.
## Possible Solution
The likely cause is that the Jenkins build farm machines are being overloaded. The very setup of the Jenkins site allows for this to occur because jobs can use more cores (i.e. "executors") than they declare and therefore overload the machine. Many of these timeouts occur late at night or in the early morning when the Jenkins build farm machines are likely to be processing nightly jobs.
## Steps to Reproduce
Hard to reproduce because these are random failures. I wish we had load statistics on these Jenkins build farm machine so that we could know when these we see timeouts like this.
Improve productivity, stability, and quality of Trilinos