Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2018-07-24T20:04:19Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3106Failing ShyLU_DD tests in GCC 4.8.4 OpenMP CI and PR builds somehow pushed to...2018-07-24T20:04:19ZJames WillenbringFailing ShyLU_DD tests in GCC 4.8.4 OpenMP CI and PR builds somehow pushed to 'develop' branch*Created by: bartlettroscoe*
@trilinos/framework, @trilinos/shylu, @searhein, @srajama1 (Trilinos Linear Solvers Product Lead)
The post-push CI build `Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI` shown [here](https://testin...*Created by: bartlettroscoe*
@trilinos/framework, @trilinos/shylu, @searhein, @srajama1 (Trilinos Linear Solvers Product Lead)
The post-push CI build `Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI` shown [here](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=-MPI_RELEASE_DEBUG_SHARED_PT&field2=groupname&compare2=61&value2=Continuous&field3=buildstarttime&compare3=84&value3=now&field4=buildstarttime&compare4=83&value4=4%20weeks%20ago) failed this morning for the first time in [this CI iteration](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3719977&filtercount=4&showfilters=1&field1=groupname&compare1=61&value1=Continuous&field2=buildstarttime&compare2=84&value2=now&field3=buildstarttime&compare3=83&value3=4%20weeks%20ago&filtercombine=and) due to the failing tests shown [here](https://testing-vm.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=3719977) which were:
|Test Name | Status | Test Time | Details |
| :-- | :-- | :-- | :-- |
| ShyLU_DDFROSch_test_laplacian_epetra_2d_gdsw_MPI_4 | Failed | 1s 170ms | 4s 680ms | Completed (Failed) |
| ShyLU_DDFROSch_test_laplacian_epetra_2d_rgdsw_MPI_4 | Failed | 1s 170ms | 4s 680ms | Completed (Failed) |
| ShyLU_DDFROSch_test_interfacesets_2D_MPI_4 | Failed | 710ms | 2s 840ms | Completed (Failed) |
|ShyLU_DDFROSch_test_interfacepartitionofunity_MPI_4 | Failed | 700ms | 2s 800ms | Completed (Failed) |
Looking at the commits pulled this first CI iteration [here](https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3720229#!#note2) it seems likely these failures are due to one of the commits by `Alexander Heinlein <alexander.heinlein@uni-köln.de>` (@searhein).
These failing ShyLU_DD tests did not only impact the post-push CI OpenMP build above but it also impacted the auto PR OpenMP build `PR-<pr-id>-test-Trilinos_pullrequest_gcc_4.8.4-<build-num>`. For example, you can see this in the PR builds:
* [PR-3098-test-Trilinos_pullrequest_gcc_4.8.4-781](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3719867)
* [PR-3053-test-Trilinos_pullrequest_gcc_4.8.4-782](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3720319)
If you look, it is the same 4 failing tests. If you look at those PR #3098 the changes in that branch don't impact ShyLU_DD or the PR builds in any way.
https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3720229#!#note2https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3029Framework: Windows build didn't post again last night2018-06-28T21:05:18ZJames WillenbringFramework: Windows build didn't post again last night*Created by: csiefer2*
... but it did the night before*Created by: csiefer2*
... but it did the night beforehttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2995Framework: Autotester may be busted?2018-06-21T13:54:55ZJames WillenbringFramework: Autotester may be busted?*Created by: csiefer2*
Been waiting 5 hours now for the testing to finish... See #2988
@trilinos/framework *Created by: csiefer2*
Been waiting 5 hours now for the testing to finish... See #2988
@trilinos/framework https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2974Trilinos GCC 4.8.4 PR build and running of tests is broken2018-06-19T20:25:41ZJames WillenbringTrilinos GCC 4.8.4 PR build and running of tests is broken*Created by: bartlettroscoe*
@trilinos/framework, @eric-c-cyr, @rppawlo, @trilinos/nox
## Expectations
The GCC 4.8.4 auto PR build should work unless the PR code itself is broken.
## Current Behavior
The GCC 4.8.4 auto PR bu...*Created by: bartlettroscoe*
@trilinos/framework, @eric-c-cyr, @rppawlo, @trilinos/nox
## Expectations
The GCC 4.8.4 auto PR build should work unless the PR code itself is broken.
## Current Behavior
The GCC 4.8.4 auto PR build has broken NOX test builds and all of the tests in every package fail, starting in the auto PR testing iteration build last night:
* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3627737&filtercount=3&showfilters=1&field1=groupname&compare1=61&value1=Pull%20Request&field2=buildstarttime&compare2=84&value2=NOW&field3=buildstarttime&compare3=83&value3=1%20week%20ago&filtercombine=and
YOu can see the NOX failures at:
* https://testing-vm.sandia.gov/cdash/viewBuildError.php?buildid=3627737
And all of the tests fail because `--bind-to none` is being used with `mpiexec` with OpenMP 1.6.5 as shown, for example, at:
* https://testing-vm.sandia.gov/cdash/testDetails.php?test=48792061&build=3627756
which shows the commandline:
```
"/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.8.4/openmpi/1.6.5/bin/mpiexec \"--bind-to\" \"none\" \"-np\" \"1\" \"/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/pull_request_test/packages/kokkos-kernels/unit_test/KokkosKernels_blas_serial.exe\" \"--gtest_filter=-serial.gemm_double\"
```
As described in #2462 and #2788, you can't use those options with OpenMPI 1.6.5. OpenMPI 1.10.1 is the only version in the SEMS env where those options work (see #2462 and #2788).
It is important to fix this ASAP since every PR build will fail and therefore no one will be able to merge any PRs until this gets fixed.
## Motivation and Context
I can't merge my PR #2964.
## Definition of Done
PR builds are working unless code changes break them.
## Possible Solution
Use the source and *.cmake scripts in #2788. That will fix it.
## Steps to Reproduce
## Your Environment
N.A. This is the auto PR testing having this issue.
## Related Issues
* Blocks: #2964
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2864Auto PR build failures due to Intel License server problems2018-06-01T18:11:28ZJames WillenbringAuto PR build failures due to Intel License server problems*Created by: bartlettroscoe*
CC: @trilinos/framework
## Expectations
Auto PR builds should be robust and not fail unless there is a failure in code itself.
## Current Behavior
The new Intel auto PR build fails randomly due ...*Created by: bartlettroscoe*
CC: @trilinos/framework
## Expectations
Auto PR builds should be robust and not fail unless there is a failure in code itself.
## Current Behavior
The new Intel auto PR build fails randomly due to Intel license server problems such as shown at:
* https://github.com/trilinos/Trilinos/pull/2860#issuecomment-393742327
which showed the build failure:
* https://testing-vm.sandia.gov/cdash/viewBuildError.php?buildid=3564874
which showed:
```
Error: A license for Comp-CL is not available now (-15,570,115).
A connection to the license server could not be made. You should
make sure that your license daemon process is running: both an
lmgrd process and an INTEL process should be running
if your license limits you to a specified number of licenses in use
at a time. Also, check to see if the wrong port@host or the wrong
license file is being used, or if the port or hostname in the license
file has changed.
License file(s) used were (in this order):
1. Trusted Storage
** 2. /projects/sems/install/rhel6-x86_64/sems/compiler/intel/17.0.1/base/Licenses/intel-Linux-SRN.lic
** 3. /projects/sems/install/rhel6-x86_64/sems/compiler/intel/17.0.1/base/compilers_and_libraries_2017.1.132/linux/bin/intel64/../../Licenses
** 4. /ascldap/users/trilinos/Licenses
```
The reason that I noticed this was because it happened in my PR #2860.
Looking at the current PR build history at:
* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=-test-Trilinos_pullrequest_intel_17.0.1&field2=buildstarttime&compare2=84&value2=2018-06-02&field3=buildstarttime&compare3=83&value3=2018-05-14
out of 10 builds, it failed twice with these Intel license server problems. That is an only 80% success rate so far. That is not robust enough for an auto PR build.
## Motivation and Context
Auto PR builds block what goes into the 'develop' branch and long delays make things harder.
## Definition of Done
Auto PR builds should only fail due to non-code issues very infrequently.
## Possible Solution
Don't know.
## Steps to Reproduce
Don't know.
## Your Environment
N.A. This is the auto PR builds that define their own env.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2812Trilinos Framework: Enable testing against installation builds2018-05-23T20:17:40ZJames WillenbringTrilinos Framework: Enable testing against installation builds*Created by: dridzal*
@trilinos/framework
## Expectations
There should be a capability in Trilinos to run tests and examples against an **installation build** (generated using `make install`). Additionally, nightly builds should i...*Created by: dridzal*
@trilinos/framework
## Expectations
There should be a capability in Trilinos to run tests and examples against an **installation build** (generated using `make install`). Additionally, nightly builds should include at least one installation build.
## Current Behavior
There is no active installation build testing. It appears that some infrastructure exists in Tribits, but it is neither used nor tested.
## Motivation and Context
We find out from _customers_ that our builds are broken. This is unacceptable.
## Definition of Done
(1) Installation build testing enabled and (2) performed nightly.
## Proposed Solution
(1) Develop infrastructure to enable marking of tests and examples with “INSTALLATION_TEST” or a similar label.
(2) Such tests should be able to use additional test-specific headers that are not part of the main installation.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2795Epetra requires C++11 now; update documentation2018-05-22T20:13:48ZJames WillenbringEpetra requires C++11 now; update documentation*Created by: mhoemmen*
I merged the following pull request that fixes GLIBCXX debug checking for deal.II's tests:
https://github.com/trilinos/Trilinos/pull/2783
This pull request introduces a modest C++11 requirement into Epetra. ...*Created by: mhoemmen*
I merged the following pull request that fixes GLIBCXX debug checking for deal.II's tests:
https://github.com/trilinos/Trilinos/pull/2783
This pull request introduces a modest C++11 requirement into Epetra. Please see discussion there for rationale. In particular, no one could come up with a current use case demanding that Epetra be built with a C++98 compiler. Even the Epetra-only MueLu build for Windows requires Teuchos, which in turn requires C++11. More importantly, we have no idea if Epetra builds with a C++98 compiler, since we have not tested this use case for a long time.
@trilinos/epetra @trilinos/framework @trilinos/muelu @maherou @trilinos/tpetra
If y'all have a problem with that, now might be a good time to say something. Otherwise, I hereby declare that Epetra now requires C++11. Next step is to update Epetra's documentation to make this clear.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2788Replace existing GCC 4.8.4 auto PR serial Kokkos build with updated GCC 4.8.4...2018-08-01T20:50:44ZJames WillenbringReplace existing GCC 4.8.4 auto PR serial Kokkos build with updated GCC 4.8.4 + OpenMPI 1.10.1 + OpenMP configuration*Created by: bartlettroscoe*
CC: @trilinos/framework
## Description
This story is to solicit a review of the GCC 4.8.4 + OpenMPI 1.10.1 + OpenMP build which is currently submitting to CDash shown at:
* https://testing-vm.sandi...*Created by: bartlettroscoe*
CC: @trilinos/framework
## Description
This story is to solicit a review of the GCC 4.8.4 + OpenMPI 1.10.1 + OpenMP build which is currently submitting to CDash shown at:
* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&date=2018-05-20&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP&field2=buildstarttime&compare2=84&value2=now
merged in PRs #2694 and #2851 and then to consider swapping out the existing GCC 4.8.4 auto PR build with this build.
This build 100% matches the agreed-upon GCC 4.8.4 PR build described in #2317 and #2462 . And this build also uses ninja as well, and therefore adds that testing to PR testing (and greatly speeds up the build). This would be a PR build that tests Trilinos with OpenMP enabled and an updated version of OpenMPI compared to the existing auto PR builds.
The key files to review and be used in the auto PR tester are:
* [GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP_env.sh](https://github.com/trilinos/Trilinos/blob/develop/cmake/std/GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP_env.sh): Just source this to load env
* [GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP.cmake](https://github.com/trilinos/Trilinos/blob/develop/cmake/std/GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP.cmake): Just include this in the CMake configure `-C <base-path>` and enable any packages you want
Hopefully the auto PR testing infrastructure will allow these two files to be plugged in pretty easily.
The other files that are also of some interest (but don't need to be used in auto PR testing) are:
* [ctest_std_driver.cmake](https://github.com/trilinos/Trilinos/blob/develop/cmake/ctest/drivers/sems_ci/ctest_std_driver.cmake): Generic ctest -S driver for any build using this model
* [ctest_std_driver.sh](https://github.com/trilinos/Trilinos/blob/develop/cmake/ctest/drivers/sems_ci/ctest_std_driver.sh): Generic bash/Jenkins driver for any build using this model
* [ctest_GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP.sh](https://github.com/trilinos/Trilinos/blob/develop/cmake/ctest/drivers/sems_ci/ctest_GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP.sh): Specific Jenkins driver
These files shows how easy it is to set up a system based on `<build_name>` with a env script `<build_name>_env.sh` and set of CMake settings `<build_name>.cmake` and then drive everything from local builds to ctest -S/Jenkins drivers. Hopefully the auto PR system can be set up this way to make it easy to add new auto PR builds and also make it easy for developer to reproduce these builds.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2767Question: generating file "TrilinosConfig.cmake"2018-05-16T22:47:40ZJames WillenbringQuestion: generating file "TrilinosConfig.cmake"*Created by: jhux2*
Is there a Trilinos cmake option that suppresses the creation/installation of the file `TrilinosConfig.cmake`? I am trying to figure out why my build of Trilinos does not have this file.
@trilinos/framework *Created by: jhux2*
Is there a Trilinos cmake option that suppresses the creation/installation of the file `TrilinosConfig.cmake`? I am trying to figure out why my build of Trilinos does not have this file.
@trilinos/framework https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2726Please disable the "Allow rebase merging" merge button2018-10-25T15:17:22ZJames WillenbringPlease disable the "Allow rebase merging" merge button*Created by: bartlettroscoe*
@trilinos/framework
Can someone with the GitHub permissions please uncheck the "Allow rebase merging" option under the "Merge button" section at the page:
* https://github.com/trilnos/Trilinos/setting...*Created by: bartlettroscoe*
@trilinos/framework
Can someone with the GitHub permissions please uncheck the "Allow rebase merging" option under the "Merge button" section at the page:
* https://github.com/trilnos/Trilinos/settings
?
That will stop people from rebasing and then fast-forward merging a bunch of commits on the 'develop' branch. For example, currently, someone with 30 commits on their topic branch could rebase all of these commits and fast-forward merge them to 'develop'. If those commits badly broke an ATDM build for example, then we would have to back out 30 commits, one `git revert <sha1>` at a time.
By disabling this option, the options of "Merge" and "Squash" are still available, which is all we want/need.
If the topic branch has more than one commit, and they are well-formed "Separate Changes" (see [workflows(7)](https://mirrors.edge.kernel.org/pub/software/scm/git/docs/gitworkflows.html)), then you really should select the "Merge" option. But if you have multiple commits, and they are kind of a mess, then you might as well select the "Squash" option. (But people really should be cleaning up their topic branches before their final topic branch push as described [here](https://docs.google.com/document/d/1uVQYI2cmNx09fDkHDA136yqDTqayhxqfvjFiuUue7wo/edit#bookmark=id.5lteak78cig8)).
If you only have one commit on your topic branch, then it is silly to create a merge commit for this. In that case, you should select "Squash". When you select "Squash" with a single commit, GitHub populates the squash commit summary and log text the same as for the single commit. Therefore, it is equivalent to rebasing a single commit in this case. And a single squashed commit is as easy to revert as a single merge commit. In fact, a single squash commit is easier to revert because you don't need to provide the parent index with `git revert -m <sha>`. You can just call `git revert <sha1>`.
## Definition of Done
* Uncheck the button "Allow rebase merging" on the page https://github.com/trilnos/Trilinos/settings
## Possible Solution
* Uncheck the button "Allow rebase merging" on the page https://github.com/trilnos/Trilinos/settings
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2698MPI_Finalize slow with CUDA + OpenMPI 2.x (known OpenMPI issue; fixed in 3.1)2018-05-14T18:34:47ZJames WillenbringMPI_Finalize slow with CUDA + OpenMPI 2.x (known OpenMPI issue; fixed in 3.1)*Created by: mhoemmen*
Tpetra::CrsMatrix UnitTests2 takes > 560s in a CUDA 8 release build on K80. Seriously, what's going on? Do I need different `KOKKOS_ARCH` settings? It sure would have been nice to have had some performance trac...*Created by: mhoemmen*
Tpetra::CrsMatrix UnitTests2 takes > 560s in a CUDA 8 release build on K80. Seriously, what's going on? Do I need different `KOKKOS_ARCH` settings? It sure would have been nice to have had some performance tracking so we could have caught this. I don't think this is anything we did; we've only been fixing CUDA issues over time.
```
36/145 Test #36: TpetraCore_CrsMatrix_UnitTests2_MPI_4 ....................................................... Passed 561.02 sec
```
@trilinos/tpetra https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2674Add MueLu "Refactor" enables to auto PR and ATDM Trilinos builds2018-11-07T20:23:32ZJames WillenbringAdd MueLu "Refactor" enables to auto PR and ATDM Trilinos builds*Created by: bartlettroscoe*
**CC:** @trilinos/framework, @rppawlo, @bathmatt, @micahahoward, @fryeguy52, @trilinos/muelu, @trilinos/kokkos
## Next Action Status
PR #3716 merged on 10/23/2018 automatically enables these "Kokkos...*Created by: bartlettroscoe*
**CC:** @trilinos/framework, @rppawlo, @bathmatt, @micahahoward, @fryeguy52, @trilinos/muelu, @trilinos/kokkos
## Next Action Status
PR #3716 merged on 10/23/2018 automatically enables these "Kokkos Refactor" options by default for all OpenMP and CUDA builds. This was confirmed in the CMakeCache file for the build `Trilinos-atdm-waterman-cuda-9.2-debug` on 10/25/2018 shown [here](https://testing.sandia.gov/cdash-dev-view/viewNotes.php?buildid=4093463#!#note1). Next: Get confirmation from @jmgate that these options have been removed from the EMPIRE's EM-Plasma `BuildScripts/ALL/configure-trilinos.sh` script ...
## Description
As of 5/2/2018, The EMPIRE build of Trilinos is now enabling the options:
```
-D MueLu_ENABLE_Kokkos_Refactor:BOOL=ON \
-D Xpetra_ENABLE_Kokkos_Refactor:BOOL=ON \
-D MueLu_ENABLE_Kokkos_Refactor_Use_By_Default:BOOL=ON \
```
in the script `BuildScripts/ALL/configure-trilinos.sh` in the `EM-Plasma/BuildScripts` git repo as of commit:
```
commit fc167d95c1ac4bc810050ababbf94a80c860dbe2
Author: Matt Bettencourt <mbetten@sandia.gov>
Date: Tue May 1 10:31:30 2018 -0600
Added options suggested by JHu
diff --git a/ALL/configure-trilinos.sh b/ALL/configure-trilinos.sh
index c23b5fa..d189d5c 100755
--- a/ALL/configure-trilinos.sh
+++ b/ALL/configure-trilinos.sh
@@ -201,6 +201,9 @@ cmake \
-D Kokkos_ENABLE_Cuda_Lambda:BOOL=${USE_CUDA} \
-D Kokkos_ENABLE_Debug_Bounds_Check:BOOL=${BOUNDS_CHECK} \
-D KOKKOS_ARCH=${Kokkos_Arch} \
+-D MueLu_ENABLE_Kokkos_Refactor:BOOL=ON \
+-D Xpetra_ENABLE_Kokkos_Refactor:BOOL=ON \
+-D MueLu_ENABLE_Kokkos_Refactor_Use_By_Default:BOOL=ON \
-D Panzer_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-D Panzer_ENABLE_FADTYPE:STRING="Sacado::Fad::DFad<RealType>" \
-D Panzer_ENABLE_TESTS:BOOL=ON \
```
Because of this, we need to add these options to the Trilinos auto PR builds (#2317) and the ATDM Trilinos builds (TRIL-171).
Another solution would be to change MueLu to turn these options on by default for non-Windows systems and then to support Windows, disable these by default.
That latter is likely preferable from a SNL Trilinos customer perspective. Trilinos, by default, should configure in a way that supports our most important customers and then require extra options to configure in other ways for other customers. Right now, enabling Kokkos by default should be the norm, not the exception.
## Related Issues
* Related to: #2317
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2620Clean up builds and tests for all Primary Tested packages for CUDA build on w...2018-04-25T23:45:07ZJames WillenbringClean up builds and tests for all Primary Tested packages for CUDA build on white/ride*Created by: bartlettroscoe*
**CC:** @trilinos/framework
## Description
This issue is a placeholder for cleaning up the builds and tests for all of the Primary Tested Trilinos packages for the `cuda-debug` build on `white/ride` as...*Created by: bartlettroscoe*
**CC:** @trilinos/framework
## Description
This issue is a placeholder for cleaning up the builds and tests for all of the Primary Tested Trilinos packages for the `cuda-debug` build on `white/ride` as described in https://github.com/trilinos/Trilinos/issues/2464#issuecomment-383646788.
As of 4/23/2018, the build `Trilinos-atdm-white-ride-cuda-debug-pt-all-at-once` on `white` for all 53 of the current Primary Tested Trilinos packages passed the configure but has several build and runtime test failures as shown at:
* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3451394
This shows build failures for the packages:
* ROL (@trilinos/rol)
* MiniTensor (@lxmota)
* Zoltan (@trilinos/zoltan)
* TrilinosCouplings (???)
* ShyLU_DD (@trilinos/shylu )
* Domi (@trilinos/domi)
And it shows test failures for the packages:
* Stokhos (@trilinos/stokhos)
* ROL
* Zoltan
* TrilinosCouplings
* Piro (@trilinos/piro)
* FEI (@trilinos/fei)
## Steps to Reproduce
Anyone should be able to reproduce any of these package failures on 'white`'(SON) or 'ride' (SRN) as:
```
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnvSettungs.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_<PackageName>=ON \
$TRILINOS_DIR
$ ninja -j16 -k 999999
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
```
For more details, see:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2597Elevate all packages and SE packages used by ATDM APPs to Primary Tested2019-03-29T19:34:21ZJames WillenbringElevate all packages and SE packages used by ATDM APPs to Primary Tested*Created by: bartlettroscoe*
**CC:** @trilinos/framework, @fryeguy52
**Blocked By:** #4219, #4272
## Description:
Currently, it seems that some of the subpackages used by the ATDM EMPIRE APP are labeled Secondary Tested (ST)....*Created by: bartlettroscoe*
**CC:** @trilinos/framework, @fryeguy52
**Blocked By:** #4219, #4272
## Description:
Currently, it seems that some of the subpackages used by the ATDM EMPIRE APP are labeled Secondary Tested (ST). Since ST is enabled by default with Trilinos in the EMPIRE Trilinos configuration, this makes sense that that would have happened. However, by definition, anything being used by ATDM APPs should be treated as Primary Tested (see [here](http://trac.trilinos.org/wiki/TribitsLifecycleModelOverview#test_categories)).
Therefore, this story is to identify which Trilinos subpackages are really being used by ATDM that are currently categorized as ST and elevate them to PT.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2594Framework: Allow commits to non-code directories w/o autotesting 2018-11-02T19:06:04ZJames WillenbringFramework: Allow commits to non-code directories w/o autotesting *Created by: csiefer2*
I'm thinking specifically of cmake/ctest/drivers, which contains stuff for driving nightly tests and not code that the autotester can actually execute.*Created by: csiefer2*
I'm thinking specifically of cmake/ctest/drivers, which contains stuff for driving nightly tests and not code that the autotester can actually execute.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2578Framework: Configure does not issue warning if package is both ON and OFF2018-04-18T17:06:19ZJames WillenbringFramework: Configure does not issue warning if package is both ON and OFF*Created by: csiefer2*
If you explicitly turn a package both ON and OFF during configure, cmake happily lets you do that without warning you that this might not be what you intended.
Can this be fixed in TriBITS / Trilinos?
@jwill...*Created by: csiefer2*
If you explicitly turn a package both ON and OFF during configure, cmake happily lets you do that without warning you that this might not be what you intended.
Can this be fixed in TriBITS / Trilinos?
@jwillenbring @bartlettroscoe https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2563Delete old autotester messages2018-04-17T15:43:41ZJames WillenbringDelete old autotester messages*Created by: cgcgcg*
@trilinos/framework
## Expectations
It would be great if the autotester would delete old messages that are not useful anymore. In particular, once a build is done, the message about the start of the build is ...*Created by: cgcgcg*
@trilinos/framework
## Expectations
It would be great if the autotester would delete old messages that are not useful anymore. In particular, once a build is done, the message about the start of the build is superfluous.
More generally, all messages could be condensed into a single message that gets updated. This would avoid having pull requests getting flooded with a bunch of autotester runs..https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2550Pull request testing uses really old OpenMPI 1.6.5 w/ build warnings2018-06-19T17:37:07ZJames WillenbringPull request testing uses really old OpenMPI 1.6.5 w/ build warnings*Created by: mhoemmen*
Here is an example of the resulting build warnings:
https://testing-vm.sandia.gov/cdash/viewBuildError.php?type=1&buildid=3418378
```
In file included from /projects/sems/install/rhel6-x86_64/sems/compiler/...*Created by: mhoemmen*
Here is an example of the resulting build warnings:
https://testing-vm.sandia.gov/cdash/viewBuildError.php?type=1&buildid=3418378
```
In file included from /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.9.3/openmpi/1.6.5/include/mpi.h:253:0,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Comm/ml_comm.h:26,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Main/ml_1level.h:31,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Smoother/ml_smoother.h:40,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Main/ml_struct.h:31,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Include/ml_include.h:18,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/test/RefMaxwell/cxx_main.cpp:24:
[CTest: warning matched] /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.9.3/openmpi/1.6.5/include/mpi_portable_platform.h:374:34: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
_STRINGIFY(__GNUC__)"."_STRINGIFY(__GNUC_MINOR__)"."_STRINGIFY(__GNUC_PATCHLEVEL__)
^
[CTest: warning matched] /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.9.3/openmpi/1.6.5/include/mpi_portable_platform.h:374:63: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
_STRINGIFY(__GNUC__)"."_STRINGIFY(__GNUC_MINOR__)"."_STRINGIFY(__GNUC_PATCHLEVEL__)
^
```
OpenMPI calls their own version 1.6.5 "retired" and the next older one is called "ancient." Sierra uses 1.10.x.
@trilinos/framework https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2502Consistent Documentation2018-04-03T21:49:21ZJames WillenbringConsistent Documentation*Created by: jmgate*
@trilinos/framework
## Expectations
It'd be great if there was a single, comprehensive set of instructions for interacting with Trilinos.
## Current Behavior
At the moment, we have [README.md](https://githu...*Created by: jmgate*
@trilinos/framework
## Expectations
It'd be great if there was a single, comprehensive set of instructions for interacting with Trilinos.
## Current Behavior
At the moment, we have [README.md](https://github.com/trilinos/Trilinos/blob/master/README.md), which people will see rendered when they first navigate to [Trilinos on GitHub](https://github.com/trilinos/Trilinos). This file then links to [CONTRIBUTING.md](https://github.com/trilinos/Trilinos/blob/master/CONTRIBUTING.md), which also shows up linked on the new issue or new pull request creation pages. For instance, on the right-hand-side, at the bottom, you should see:
<img width="237" alt="screen shot 2018-04-03 at 3 43 19 pm" src="https://user-images.githubusercontent.com/20327215/38277489-ceb53f16-3755-11e8-94ad-c2c876e1e153.png">
In addition to these sources, we also have information spread out across the [wiki](https://github.com/trilinos/Trilinos/wiki). For instance, [here](https://github.com/trilinos/Trilinos/wiki/Pull-Request-Workflow) and [here](https://github.com/trilinos/Trilinos/wiki/Committing-Changes-to-the-Trilinos-Repository).
## Motivation and Context
At the very least, we need to make sure all these sources of information stay in sync. From what I've seen of other open source software projects, everything a potential user should need to know should show up in the README.md or CONTRIBUTING.md file. If either of those gets too lengthy and it seems prudent to break the information up over the wiki, everything should still be linked back into those files.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2434Building pytrilinos with GCC 6.3.0 fails with "execvp: /bin/sh: Argument list...2018-08-24T22:27:28ZJames WillenbringBuilding pytrilinos with GCC 6.3.0 fails with "execvp: /bin/sh: Argument list too long"*Created by: samfux84*
@trilinos/pytrilinos
I am trying to build Trilinos 12.10.1 with python support. When compiling, it crashes with the error
```
make[2]: execvp: /bin/sh: Argument list too long
packages/PyTrilinos/src/CMakeF...*Created by: samfux84*
@trilinos/pytrilinos
I am trying to build Trilinos 12.10.1 with python support. When compiling, it crashes with the error
```
make[2]: execvp: /bin/sh: Argument list too long
packages/PyTrilinos/src/CMakeFiles/PyTrilinos_LOCA_Epetra_Interface.dir/build.make:192: recipe for target 'packages/PyTrilinos/src/LOCA.Epetra.InterfacePYTHON_wrap.cpp' failed
make[2]: *** [packages/PyTrilinos/src/LOCA.Epetra.InterfacePYTHON_wrap.cpp] Error 127
```
The command (38741 characters) that causes this error is:
```
cd /scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/muelu/src && /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/openmpi-2.1.0-sjd6nhdb3d4y6hhk2qtx23r3ypdi7coy/bin/mpic++ -Dmuelu_EXPORTS -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/BrickAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/UncoupledAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/UserAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Headers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Interface -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Interface/FacadeClasses -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Misc -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/MueCentral -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Rebalancing -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Smoothers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/BaseClass -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/BlockedTransfers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Generic -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Energy-Minimization -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Energy-Minimization/Solvers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Geometric-Interpolation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Smoothed-Aggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Petrov-Galerkin-SA -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/User -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/SemiCoarsen -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Utils/ForwardDeclaration -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/../adapters/epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/../adapters/tpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/muelu/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teko/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/NS -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/Epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/Tpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/stratimikos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/belos/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/stratimikos/adapters/belos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/aztecoo/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/ml/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/ifpack/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/tpetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/tpetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/epetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/aztecoo/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/aztecoo/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/triutils/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/triutils/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/kokkoscomm/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/kokkoscomm/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/kokkoscompat/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/kokkoscompat/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/parameterlist/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/core/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/core/src -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/boost-1.63.0-w3vrb2dy7hu4x5n5mgqxyw7xjxjnmhtk/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/comm/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/remainder/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/remainder/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/numerics/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/tpetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_vector/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_vector/extended -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_vector/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_vector/adapter_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_solve/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_solve/extended -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_solve/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/nonlinear/model_evaluator/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/nonlinear/model_evaluator/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/nonlinear/solvers/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/nonlinear/solvers/client_support -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/thyra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/example/operator_vector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/interfaces -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/ops_lib -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/lapack -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/rtop/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/ext -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/ext -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/inout -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/inout -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/src/kokkos_refactor -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/tsqr/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/tsqr/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/LinAlg -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/NodeAPI -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/classic/NodeAPI -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/classic/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/kernels/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/kernels/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/kernels/src/impl -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/algorithms/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/algorithms/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/containers/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/containers/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetraext/src/model_evaluator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetraext/src/transformer -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/epetraext/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/transform -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/inout -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/coloring -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/model_evaluator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/block -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/restrict -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/hypre -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/python-2.7.13-kz2b5ll34ua4qdjbflypwhtk3ujlitr4/include/python2.7 -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/hypre-2.11.1-nqgf2yzso76zjo53ad7yf7jrls2teaqc/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/suite-sparse-4.5.5-rlv2pziqccmce2ppuztg5rybahktvih7/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/hdf5-1.10.1-cn37gr2d2rhtcw3ep5neqy7ech3n7oit/include -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/ml/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Comm -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Coarsen -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/FEGrid -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Main -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Smoother -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Krylov -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Operator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/RefMaxwell -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/MatrixFree -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/MLAPI -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/LevelWrap -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/ifpack/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/../amesos/src/SuiteSparse/AMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/../amesos/src/SuiteSparse/UFconfig -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/AMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/COLAMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CHOLMOD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/UFconfig -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/KLU/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/BTF/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CAMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CCOLAMD/Include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/mumps-5.0.1-cp5ucvw44w6y5izcns3mgcw6l4qmb3pf/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/superlu-dist-5.1.1-l766mdbscb27rd5y2si6nhy57gnipm6f/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/parmetis-4.0.3-kshbkdzjzhfp324mzyqu5dcyv7aacis2/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-xpetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src-epetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-xpetra/../src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src/Headers -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/Maps -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/CRSMatrices -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/FiniteElements -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Matrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/StridedMap -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/BlockedCrsMatrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/BlockedMultiVector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Cloner -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Utils -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/xpetra/sup -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/CrsGraph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/CrsMatrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/DistObject -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Export -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Headers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Import -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Map -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/MultiVector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Operator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Platform -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/RowGraph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpe-12-10-1/packages/xpetra/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Vector -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Tril-release-12-10-1/spack-build/packages/isorropia/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6kgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/isorropia/src/epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko7stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/zoltan/src -I/cluster/apps/spack/var/spack/stage/trilinos-ack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/all -I/cluster/apps/spac-I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan//packages/zoltan/src/ha -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-rilinos-release-12-10-1/packages/zoltan/src/hsfc -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Tzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/matrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/par -I/cluster/apps/spack/var/spack/stage/t/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/tpls -I/cluster/ahg -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zolt1/packages/zoltan/src/reftree -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-reilinos-trilinos-release-12-10-1/packages/zoltan/src/timer -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwis-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/Utilities/DDirectory -I/cluster/apps/spac/Timer -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/os-release-12-10-1/packages/zoltan/src/zz -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/metis-5.1.0-s3cgl2pdmunmqnqksn4l7eyklibma7y/packages/ifpack2/adapters/thyra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packagee-12-10-1/packages/ifpack2/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/-1 -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/amesos2/src -I/cluster/apps/luster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos2/src-release-12-10-1/packages/amesos2/src/SuiteSparse/KLU2/Source -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzar/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos2/src/../../amesos/src/S-release-12-10-1/packages/amesos2/src/../../amesos/src/SuiteSparse/BTF/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77r/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src -I/sspack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms zoltan2/src/algorithms/partition -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinosyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms/color -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms/map -I/cln2/src/algorithms/zoltan -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-releaserilinos-trilinos-release-12-10-1/packages/zoltan2/src/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrz7bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/models -I/cluster/apps/spack/var/spack/stage/trilinos/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/util -I/scratch/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/anasazi/tpetra/src -I/scrat-I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/anasazi/anasazi/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-1ackages/anasazi/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-1release-12-10-1/packages/anasazi/src/../thyra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/G -fPIC -o CMakeFiles/muelu.dir/Interface/MueLu_ML2MueLuParameterTranslator.cpp.o -c /cluster/apps/spack/var/spack/stage/trilinos-12.10..cpp
```
Is this a known problem and is there any solution for it ?