Trilinos issueshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues2018-08-24T22:27:28Zhttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2434Building pytrilinos with GCC 6.3.0 fails with "execvp: /bin/sh: Argument list...2018-08-24T22:27:28ZJames WillenbringBuilding pytrilinos with GCC 6.3.0 fails with "execvp: /bin/sh: Argument list too long"*Created by: samfux84*
@trilinos/pytrilinos
I am trying to build Trilinos 12.10.1 with python support. When compiling, it crashes with the error
```
make[2]: execvp: /bin/sh: Argument list too long
packages/PyTrilinos/src/CMakeF...*Created by: samfux84*
@trilinos/pytrilinos
I am trying to build Trilinos 12.10.1 with python support. When compiling, it crashes with the error
```
make[2]: execvp: /bin/sh: Argument list too long
packages/PyTrilinos/src/CMakeFiles/PyTrilinos_LOCA_Epetra_Interface.dir/build.make:192: recipe for target 'packages/PyTrilinos/src/LOCA.Epetra.InterfacePYTHON_wrap.cpp' failed
make[2]: *** [packages/PyTrilinos/src/LOCA.Epetra.InterfacePYTHON_wrap.cpp] Error 127
```
The command (38741 characters) that causes this error is:
```
cd /scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/muelu/src && /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/openmpi-2.1.0-sjd6nhdb3d4y6hhk2qtx23r3ypdi7coy/bin/mpic++ -Dmuelu_EXPORTS -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/BrickAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/UncoupledAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Graph/UserAggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Headers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Interface -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Interface/FacadeClasses -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Misc -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/MueCentral -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Rebalancing -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Smoothers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/BaseClass -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/BlockedTransfers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Generic -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Energy-Minimization -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Energy-Minimization/Solvers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Geometric-Interpolation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Smoothed-Aggregation -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/Petrov-Galerkin-SA -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/User -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Transfers/SemiCoarsen -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/Utils/ForwardDeclaration -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/../adapters/epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/muelu/src/../adapters/tpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/muelu/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teko/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/NS -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/Epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teko/src/Tpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/stratimikos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/belos/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/stratimikos/adapters/belos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/aztecoo/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/ml/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/stratimikos/adapters/ifpack/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/tpetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/tpetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/epetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/belos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/belos/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/aztecoo/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/aztecoo/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/triutils/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/triutils/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetra/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/kokkoscomm/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/kokkoscomm/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/kokkoscompat/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/kokkoscompat/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/parameterlist/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/core/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/core/src -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/boost-1.63.0-w3vrb2dy7hu4x5n5mgqxyw7xjxjnmhtk/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/comm/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/remainder/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/teuchos/remainder/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/teuchos/numerics/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/tpetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_vector/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_vector/extended -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_vector/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_vector/adapter_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_solve/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/operator_solve/extended -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/operator_solve/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/nonlinear/model_evaluator/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/nonlinear/model_evaluator/client_support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/interfaces/nonlinear/solvers/fundamental -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/src/support/nonlinear/solvers/client_support -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/thyra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/core/example/operator_vector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/interfaces -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/support -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/ops_lib -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/rtop/src/lapack -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/rtop/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/ext -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/ext -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/inout -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/inout -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/core/src/kokkos_refactor -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/core/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/tsqr/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/tsqr/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/LinAlg -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/NodeAPI -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/classic/NodeAPI -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/classic/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/classic/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/tpetra/kernels/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/kernels/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/tpetra/kernels/src/impl -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/algorithms/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/algorithms/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/kokkos/containers/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/kokkos/containers/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetraext/src/model_evaluator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/thyra/adapters/epetraext/src/transformer -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/epetraext/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/transform -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/inout -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/coloring -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/model_evaluator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/block -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/restrict -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/hypre -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/python-2.7.13-kz2b5ll34ua4qdjbflypwhtk3ujlitr4/include/python2.7 -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/hypre-2.11.1-nqgf2yzso76zjo53ad7yf7jrls2teaqc/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/suite-sparse-4.5.5-rlv2pziqccmce2ppuztg5rybahktvih7/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/hdf5-1.10.1-cn37gr2d2rhtcw3ep5neqy7ech3n7oit/include -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/ml/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Comm -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Coarsen -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/FEGrid -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Main -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Smoother -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Krylov -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Operator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/RefMaxwell -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/MatrixFree -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/MLAPI -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ml/src/LevelWrap -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/ifpack/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/../amesos/src/SuiteSparse/AMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/ifpack/../amesos/src/SuiteSparse/UFconfig -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/AMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/COLAMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CHOLMOD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/UFconfig -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/KLU/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/BTF/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CAMD/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos/src/SuiteSparse/CCOLAMD/Include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/mumps-5.0.1-cp5ucvw44w6y5izcns3mgcw6l4qmb3pf/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/superlu-dist-5.1.1-l766mdbscb27rd5y2si6nhy57gnipm6f/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/parmetis-4.0.3-kshbkdzjzhfp324mzyqu5dcyv7aacis2/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-xpetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src-epetra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-xpetra/../src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-xpetra/../src/Headers -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/galeri/src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/Maps -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/CRSMatrices -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/galeri/src-epetra/FiniteElements -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Matrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/StridedMap -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/BlockedCrsMatrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/BlockedMultiVector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Cloner -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/sup/Utils -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/xpetra/sup -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/CrsGraph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/CrsMatrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/DistObject -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Export -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Headers -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Import -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Map -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/MultiVector -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Operator -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Platform -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/RowGraph -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpe-12-10-1/packages/xpetra/src/Utils -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/xpetra/src/Vector -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Tril-release-12-10-1/spack-build/packages/isorropia/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6kgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/isorropia/src/epetra -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko7stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/zoltan/src -I/cluster/apps/spack/var/spack/stage/trilinos-ack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/all -I/cluster/apps/spac-I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan//packages/zoltan/src/ha -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-rilinos-release-12-10-1/packages/zoltan/src/hsfc -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Tzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/matrix -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/par -I/cluster/apps/spack/var/spack/stage/t/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/tpls -I/cluster/ahg -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zolt1/packages/zoltan/src/reftree -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-reilinos-trilinos-release-12-10-1/packages/zoltan/src/timer -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwis-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan/src/Utilities/DDirectory -I/cluster/apps/spac/Timer -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/os-release-12-10-1/packages/zoltan/src/zz -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/metis-5.1.0-s3cgl2pdmunmqnqksn4l7eyklibma7y/packages/ifpack2/adapters/thyra -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packagee-12-10-1/packages/ifpack2/src -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/-1 -I/scratch/spackapps/spack-stage/spack-stage-Qws5tO/Trilinos-trilinos-release-12-10-1/spack-build/packages/amesos2/src -I/cluster/apps/luster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos2/src-release-12-10-1/packages/amesos2/src/SuiteSparse/KLU2/Source -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzar/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/amesos2/src/../../amesos/src/S-release-12-10-1/packages/amesos2/src/../../amesos/src/SuiteSparse/BTF/Include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77r/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src -I/sspack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms zoltan2/src/algorithms/partition -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinosyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms/color -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/algorithms/map -I/cln2/src/algorithms/zoltan -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-releaserilinos-trilinos-release-12-10-1/packages/zoltan2/src/include -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrz7bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/models -I/cluster/apps/spack/var/spack/stage/trilinos/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/zoltan2/src/util -I/scratch/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/anasazi/tpetra/src -I/scrat-I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-10-1/packages/anasazi/anasazi/epetra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-1ackages/anasazi/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/Trilinos-trilinos-release-12-1release-12-10-1/packages/anasazi/src/../thyra/src -I/cluster/apps/spack/var/spack/stage/trilinos-12.10.1-ko77bb2zh5cmh6khrkgrzyqwiea6jc6a/G -fPIC -o CMakeFiles/muelu.dir/Interface/MueLu_ML2MueLuParameterTranslator.cpp.o -c /cluster/apps/spack/var/spack/stage/trilinos-12.10..cpp
```
Is this a known problem and is there any solution for it ?
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2100Configuration fails: CMake Error: CMake can not determine linker language for...2018-08-22T15:19:35ZJames WillenbringConfiguration fails: CMake Error: CMake can not determine linker language for target: blotlib*Created by: yurivict*
This error occurs when ```-DTPL_ENABLE_Netcdf=ON```:
```
-- Configuring done
CMake Error: CMake can not determine linker language for target: blotlib
CMake Error: CMake can not determine linker language for ...*Created by: yurivict*
This error occurs when ```-DTPL_ENABLE_Netcdf=ON```:
```
-- Configuring done
CMake Error: CMake can not determine linker language for target: blotlib
CMake Error: CMake can not determine linker language for target: blotlib
CMake Error in packages/seacas/applications/blot/CMakeLists.txt:
Exporting the target "blotlib" is not allowed since its linker language
cannot be determined
```
Version: 12-12-1 on FreeBSD 11.1
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/440Switch to 'develop'/'master' branch in all repos listed in ExtraRepositoriesL...2018-08-15T01:35:32ZJames WillenbringSwitch to 'develop'/'master' branch in all repos listed in ExtraRepositoriesList.cmake*Created by: bartlettroscoe*
## Next Action Status:
???
**Blocked By:** #370, #176, #452
## Description:
As described in [this comment](https://github.com/trilinos/Trilinos/issues/370#issuecomment-224391677) in #370, the cu...*Created by: bartlettroscoe*
## Next Action Status:
???
**Blocked By:** #370, #176, #452
## Description:
As described in [this comment](https://github.com/trilinos/Trilinos/issues/370#issuecomment-224391677) in #370, the current implementation of the `TRIBITS_CTEST_DRIVER()` function will clone and use the 'master' branch of the extra repos even if `Trilinos_BRANCH=develop` or `trilinos-release-X-Y-branch`, etc. That is not consistent with treating this set of repos as "one big repo".
Therefore, this story is to switch all of the extra repos listed in Trilinos/cmake/ExtraRepositoriesList.cmake over to the 'develop'/'master' workflow that will be released or used by outside users. Any extra repo that is not going to switch over to the 'develop'/'master' workflow should not be listed in that file and should not be part of automated testing of Trilinos or Trilinos releases. More can be discussed about this but that basics are simple.
## Tasks:
1. Implement TriBITSPub/TriBITS#130 in TriBITS (This will result in `Trilinos_BRANCH=<branch>` getting checked out in each extra repo as well as the base Trilinos repo.) **[DONE]**
2. Determine what extra repos currently listed in ExtraRepositoriesList.cmake should continue being tested and perhaps released and which should not.
3. For those repos listed in ExtraRepositoriesList.cmake (i.e. that will continue to be tested and perhaps released), add a 'develop' branch.
4. Transition to the 'develop'/'master' workflow for all of the extra repos listed in ExtraRepositoriesList.cmake (in one push to the Trilinos 'develop' branch):
a. Update 'develop' branch from 'master' branch in all these extra repos and then lock down 'master' branch like for the main Trilinos git repo in #370.
b. Update the process that updates the Trilinos 'develop' branch to the 'master' branch also update from the 'develop' to the 'master' branches in these extra repos as well.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3263Test ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1 randomly failing in CI and...2018-08-09T19:59:42ZJames WillenbringTest ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1 randomly failing in CI and PR GCC 4.8.4 + OpenMP builds*Created by: bartlettroscoe*
@trilinos/shylu, @trilinos/framework, @srajama1 (Trilinos Linear Solvers Product Lead)
## Expectations
A test should not fail unless a changes is made to break it. A test should not randomly fail.
...*Created by: bartlettroscoe*
@trilinos/shylu, @trilinos/framework, @srajama1 (Trilinos Linear Solvers Product Lead)
## Expectations
A test should not fail unless a changes is made to break it. A test should not randomly fail.
## Current Behavior
Looking at the four most recent failures of the test `ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1` in [this query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=testname&compare1=61&value1=ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1&field2=status&compare2=62&value2=passed&field3=status&compare3=62&value3=notrun&field4=buildstarttime&compare4=84&value4=now) the test appears to be randomly failing in the GCC 4.8.4 OpenMPI builds. In the most recent case, this test broke the auto PR GCC 4.8.4 + OpenMP build in PR #3260. IN each of the last for failures of this test dating back to 6/28/2018, they all fail showing:
```
....
[ RUN ] CrsMatrixBase.matrixmarket
unknown file: Failure
C++ exception with description "View bounds error of view ap ( 13 < 13 )
Traceback functionality not available
" thrown in the test body.
[ FAILED ] CrsMatrixBase.matrixmarket (23 ms)
...
[ FAILED ] 1 test, listed below:
[ FAILED ] CrsMatrixBase.matrixmarket
1 FAILED TEST
```
<!---
Tell us how the current behavior fails to meet your expectations in some way.
-->
## Motivation and Context
<!---
How has this expectation failure affected you? What are you trying to
accomplish? Why do we need to address this? What does it have to do with
anything? Providing context helps us come up with a solution that is most
useful in the real world.
-->
## Definition of Done
The test `ShyLU_NodeTacho_Tacho_TestSerial_double_MPI_1` is fixed to make it so that it does not randomly fail or is removed for CI and auto PR testing.
## Possible Solution
Fix it so that it does not randomly fail or remove it from CI and auto PR testing.
## Steps to Reproduce
See https://github.com/trilinos/Trilinos/wiki/Reproducing-PR-Testing-Errors.
## Your Environment
Standard SEMS GCC 4.8.4 auto PR build env (see above).
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3272package_subproject_list.cmake static in Trilinos PR tester?2018-08-09T19:58:47ZJames Willenbringpackage_subproject_list.cmake static in Trilinos PR tester?*Created by: bartlettroscoe*
@trilinos/framework,
Is the file package_subproject_list.cmake used at:
https://github.com/trilinos/Trilinos/blob/0df591fd6d9b9ccd91bd2695cc22665aa468886d/cmake/std/PullRequestLinuxDriver.sh#L206
st...*Created by: bartlettroscoe*
@trilinos/framework,
Is the file package_subproject_list.cmake used at:
https://github.com/trilinos/Trilinos/blob/0df591fd6d9b9ccd91bd2695cc22665aa468886d/cmake/std/PullRequestLinuxDriver.sh#L206
statically generated and require manual updating? If so, that would explain why the PR builds are not testing the new PT package TrilinosFrameworkTests.
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1155Implement automated pull request testing for Trilinos2018-08-01T20:51:58ZJames WillenbringImplement automated pull request testing for Trilinos*Created by: jwillenbring*
@trilinos/framework
@allevin is planning to develop a capability that will allow for testing of Trilinos pull request prior to having the changes applied to the develop branch. This story will track that w...*Created by: jwillenbring*
@trilinos/framework
@allevin is planning to develop a capability that will allow for testing of Trilinos pull request prior to having the changes applied to the develop branch. This story will track that work. A rough outline for the work is as follows:
1) Adapt some existing scripts that test the pull requests for another project to be more general and serve the Trilinos case.
2) Deploy on a limited, opt-in basis.
3) Deploy on a full basis for all Trilinos develop branch modifications.Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2788Replace existing GCC 4.8.4 auto PR serial Kokkos build with updated GCC 4.8.4...2018-08-01T20:50:44ZJames WillenbringReplace existing GCC 4.8.4 auto PR serial Kokkos build with updated GCC 4.8.4 + OpenMPI 1.10.1 + OpenMP configuration*Created by: bartlettroscoe*
CC: @trilinos/framework
## Description
This story is to solicit a review of the GCC 4.8.4 + OpenMPI 1.10.1 + OpenMP build which is currently submitting to CDash shown at:
* https://testing-vm.sandi...*Created by: bartlettroscoe*
CC: @trilinos/framework
## Description
This story is to solicit a review of the GCC 4.8.4 + OpenMPI 1.10.1 + OpenMP build which is currently submitting to CDash shown at:
* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&date=2018-05-20&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=61&value1=GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP&field2=buildstarttime&compare2=84&value2=now
merged in PRs #2694 and #2851 and then to consider swapping out the existing GCC 4.8.4 auto PR build with this build.
This build 100% matches the agreed-upon GCC 4.8.4 PR build described in #2317 and #2462 . And this build also uses ninja as well, and therefore adds that testing to PR testing (and greatly speeds up the build). This would be a PR build that tests Trilinos with OpenMP enabled and an updated version of OpenMPI compared to the existing auto PR builds.
The key files to review and be used in the auto PR tester are:
* [GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP_env.sh](https://github.com/trilinos/Trilinos/blob/develop/cmake/std/GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP_env.sh): Just source this to load env
* [GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP.cmake](https://github.com/trilinos/Trilinos/blob/develop/cmake/std/GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP.cmake): Just include this in the CMake configure `-C <base-path>` and enable any packages you want
Hopefully the auto PR testing infrastructure will allow these two files to be plugged in pretty easily.
The other files that are also of some interest (but don't need to be used in auto PR testing) are:
* [ctest_std_driver.cmake](https://github.com/trilinos/Trilinos/blob/develop/cmake/ctest/drivers/sems_ci/ctest_std_driver.cmake): Generic ctest -S driver for any build using this model
* [ctest_std_driver.sh](https://github.com/trilinos/Trilinos/blob/develop/cmake/ctest/drivers/sems_ci/ctest_std_driver.sh): Generic bash/Jenkins driver for any build using this model
* [ctest_GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP.sh](https://github.com/trilinos/Trilinos/blob/develop/cmake/ctest/drivers/sems_ci/ctest_GCC-4.8.4-OpenMPI-1.10.1-MpiReleaseDebugSharedPtOpenMP.sh): Specific Jenkins driver
These files shows how easy it is to set up a system based on `<build_name>` with a env script `<build_name>_env.sh` and set of CMake settings `<build_name>.cmake` and then drive everything from local builds to ctest -S/Jenkins drivers. Hopefully the auto PR system can be set up this way to make it easy to add new auto PR builds and also make it easy for developer to reproduce these builds.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/3106Failing ShyLU_DD tests in GCC 4.8.4 OpenMP CI and PR builds somehow pushed to...2018-07-24T20:04:19ZJames WillenbringFailing ShyLU_DD tests in GCC 4.8.4 OpenMP CI and PR builds somehow pushed to 'develop' branch*Created by: bartlettroscoe*
@trilinos/framework, @trilinos/shylu, @searhein, @srajama1 (Trilinos Linear Solvers Product Lead)
The post-push CI build `Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI` shown [here](https://testin...*Created by: bartlettroscoe*
@trilinos/framework, @trilinos/shylu, @searhein, @srajama1 (Trilinos Linear Solvers Product Lead)
The post-push CI build `Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI` shown [here](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=-MPI_RELEASE_DEBUG_SHARED_PT&field2=groupname&compare2=61&value2=Continuous&field3=buildstarttime&compare3=84&value3=now&field4=buildstarttime&compare4=83&value4=4%20weeks%20ago) failed this morning for the first time in [this CI iteration](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3719977&filtercount=4&showfilters=1&field1=groupname&compare1=61&value1=Continuous&field2=buildstarttime&compare2=84&value2=now&field3=buildstarttime&compare3=83&value3=4%20weeks%20ago&filtercombine=and) due to the failing tests shown [here](https://testing-vm.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=3719977) which were:
|Test Name | Status | Test Time | Details |
| :-- | :-- | :-- | :-- |
| ShyLU_DDFROSch_test_laplacian_epetra_2d_gdsw_MPI_4 | Failed | 1s 170ms | 4s 680ms | Completed (Failed) |
| ShyLU_DDFROSch_test_laplacian_epetra_2d_rgdsw_MPI_4 | Failed | 1s 170ms | 4s 680ms | Completed (Failed) |
| ShyLU_DDFROSch_test_interfacesets_2D_MPI_4 | Failed | 710ms | 2s 840ms | Completed (Failed) |
|ShyLU_DDFROSch_test_interfacepartitionofunity_MPI_4 | Failed | 700ms | 2s 800ms | Completed (Failed) |
Looking at the commits pulled this first CI iteration [here](https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3720229#!#note2) it seems likely these failures are due to one of the commits by `Alexander Heinlein <alexander.heinlein@uni-köln.de>` (@searhein).
These failing ShyLU_DD tests did not only impact the post-push CI OpenMP build above but it also impacted the auto PR OpenMP build `PR-<pr-id>-test-Trilinos_pullrequest_gcc_4.8.4-<build-num>`. For example, you can see this in the PR builds:
* [PR-3098-test-Trilinos_pullrequest_gcc_4.8.4-781](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3719867)
* [PR-3053-test-Trilinos_pullrequest_gcc_4.8.4-782](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3720319)
If you look, it is the same 4 failing tests. If you look at those PR #3098 the changes in that branch don't impact ShyLU_DD or the PR builds in any way.
https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3720229#!#note2https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3175Framework: Windows Testing is Down Again2018-07-24T14:39:28ZJames WillenbringFramework: Windows Testing is Down Again*Created by: csiefer2*
That.*Created by: csiefer2*
That.https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3118Kokkos CMake error (fixed in kokkos:develop)2018-07-16T15:37:16ZJames WillenbringKokkos CMake error (fixed in kokkos:develop)*Created by: mhoemmen*
@trilinos/kokkos @trilinos/framework
## Current Behavior
At least one Trilinos build on the Dashboard shows the following CMake error:
```
CMake Error at packages/kokkos/cmake/kokkos_functions.cmake:50 (s...*Created by: mhoemmen*
@trilinos/kokkos @trilinos/framework
## Current Behavior
At least one Trilinos build on the Dashboard shows the following CMake error:
```
CMake Error at packages/kokkos/cmake/kokkos_functions.cmake:50 (string):
Syntax error in cmake code at
/home/jenkins/slave/workspace/Trilinos_apollo_gcc_4.9.3_cuda_8.0.44/MPI_RELEASE_DEV_DownStream_ETI_SERIAL-ON_OPENMP-OFF_PTHREAD-OFF_CUDA-ON_COMPLEX-OFF/Trilinos/packages/kokkos/cmake/kokkos_functions.cmake:50
when parsing string
[0-9]+\.[0-9]+\.[0-9]+$
Invalid escape sequence \.
Call Stack (most recent call first):
cmake/ProjectCompilerPostConfig.cmake:34 (set_kokkos_cxx_compiler)
/home/jenkins/slave/workspace/Trilinos_apollo_gcc_4.9.3_cuda_8.0.44/Trilinos/cmake/tribits/core/package_arch/TribitsGlobalMacros.cmake:1847 (INCLUDE)
/home/jenkins/slave/workspace/Trilinos_apollo_gcc_4.9.3_cuda_8.0.44/Trilinos/cmake/tribits/core/package_arch/TribitsProjectImpl.cmake:188 (TRIBITS_SETUP_ENV)
/home/jenkins/slave/workspace/Trilinos_apollo_gcc_4.9.3_cuda_8.0.44/Trilinos/cmake/tribits/core/package_arch/TribitsProject.cmake:93 (TRIBITS_PROJECT_IMPL)
CMakeLists.txt:90 (TRIBITS_PROJECT)
```
This is https://github.com/kokkos/kokkos/issues/1661 , which has been fixed in kokkos:develop by Kokkos PR https://github.com/kokkos/kokkos/pull/1662 .
## Possible Solution
Wait until the next Kokkos promotion into Trilinos.
## Related Issues
* Related to https://github.com/kokkos/kokkos/issues/1661https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3105Random timeouts in auto PR builds2018-07-13T14:33:40ZJames WillenbringRandom timeouts in auto PR builds*Created by: bartlettroscoe*
@trilinos/framework
## Expectations
Tests will not fail or timeout unless changes in a PR branch cause the failures or timeouts.
## Current Behavior
Several auto PR build iterations have been fa...*Created by: bartlettroscoe*
@trilinos/framework
## Expectations
Tests will not fail or timeout unless changes in a PR branch cause the failures or timeouts.
## Current Behavior
Several auto PR build iterations have been failing for a while due to random timeouts in various auto PR builds. For example, the first PR testing iteration in my PR #3104 failed last night due to random timeouts as shown [here](https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=PR-3104-test&field2=buildstarttime&compare2=84&value2=NOW). It is impossible for that one change to have impacted these test timeouts in any way.
You can see timeouts impacting other PR testing iterations as well in just the last 12 days in [this query](https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-07-12&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildstarttime&compare1=83&value1=2018-07-01&field2=groupname&compare2=61&value2=Pull%20Request&field3=details&compare3=63&value3=timeout). This shows 12 tests timing out in 3 different PRs over the last 12 days. Looking at the numbering of the builds, it seems likely that up to 4 or 5 PR testing iterations failed due to this issue. (It is hard to know how many PR testing iterations failed due to these random timeouts due to the naming of the PR builds since it is hard to differentiate different PR testing iterations in the same PR or to match up which builds of different compilers match up to the same PR testing iteration). Also, it is possible that some of these timeouts were due to changes in the PR branch.
## Motivation and Context
We want PR testing iterations to only fail if they are triggered by changes in the specific topic branch being tested.
While 4 or 5 failed PR testing iterations over 12 days may not seem like a lot, when combined with randomly failing tests (see #3103) and randomly failures in the Jenkins jobs (git pull fails, etc.), these add up to make the auto PR testing pretty unstable.
## Definition of Done
* Eliminate randomly failing tests.
## Possible Solution
The likely cause is that the Jenkins build farm machines are being overloaded. The very setup of the Jenkins site allows for this to occur because jobs can use more cores (i.e. "executors") than they declare and therefore overload the machine. Many of these timeouts occur late at night or in the early morning when the Jenkins build farm machines are likely to be processing nightly jobs.
## Steps to Reproduce
Hard to reproduce because these are random failures. I wish we had load statistics on these Jenkins build farm machine so that we could know when these we see timeouts like this.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/1976Get Trilinos working with CUDA-9.02018-07-09T21:32:57ZJames WillenbringGet Trilinos working with CUDA-9.0*Created by: bartlettroscoe*
**CC**: @trilinos/framework, @crtrott, @hcedwar, @nmhamster, @bathmatt, @micahahoward
## Next Action Status
Done as part of #2706.
## Description
This is a high-level issue to coordinate the eff...*Created by: bartlettroscoe*
**CC**: @trilinos/framework, @crtrott, @hcedwar, @nmhamster, @bathmatt, @micahahoward
## Next Action Status
Done as part of #2706.
## Description
This is a high-level issue to coordinate the efforts to get Trilinos working with CUDA 9.0 (CUDA9.0, CUDA-9.0). There is a driver for this from ATDM application developers (i.e. @bathmatt and @micahahoward).
The Coordinated DevOps for ATDM Story tracking this is:
* https://software-srn.sandia.gov/jira/browse/CDOFA-22
but as many details as possible will be tracked in this issue.
NOTE: Other versions of CUDA 9.x should not be discussed.
## Tasks:
1. [ ] Verify that Kokkos is supporting CUDA 9.0
2. [ ] Identify one or more computer systems at SNL where a CUDA 9.0 env can be accessed for Trilinos developers and to set up automated testing with Trilinos
3. [ ] Produce a build of Trilinos with target CUDA 9.0 env and post to experimental build to CDash
4. [ ] Verify that there exists (or create) Trilinos GitHub issues for the current set of problems with this CUDA 9.0 build (along with clear and easy reproducability instructions).
## Related Issues
* Blocks
* Is blocked by
* Follows
* Precedes
* **Related to:** #1928
* Part of
* Composed of
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/3029Framework: Windows build didn't post again last night2018-06-28T21:05:18ZJames WillenbringFramework: Windows build didn't post again last night*Created by: csiefer2*
... but it did the night before*Created by: csiefer2*
... but it did the night beforehttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/1751Discuss and document workflow for directly snapshotting SEACAS into Trilinos ...2018-06-26T12:49:47ZJames WillenbringDiscuss and document workflow for directly snapshotting SEACAS into Trilinos to better serve ATDM*Created by: bartlettroscoe*
**Next Action Status:**
@gdsjaar to implement agreed to workflow on SEACAS github site then start doing new workflow ...
**CC:** @trilinos/framework, @gsjaardema
**Description:**
This story is t...*Created by: bartlettroscoe*
**Next Action Status:**
@gdsjaar to implement agreed to workflow on SEACAS github site then start doing new workflow ...
**CC:** @trilinos/framework, @gsjaardema
**Description:**
This story is to document a discussion and the final workflow for allowing @gsjaardema to directly SEACAS directly into Trilinos/packages/seacas/ in addition to shapshotting SEACAS into Sierra.base/seacas/ and then having that snapshotted from ther into Trilinos/packages/seacas/. The nature of the snapshotting process (i.e. never a merge) eliminates the potential for merge conflicts that might make the snapshotting from Sierra.base/seacas/ to Trilinos/packages/seacas/ more difficult. And this would allow @gsjaardema control to address issues for ATDM customers very quickly and simplify the version control and configuration of Trilinos for ATDM customers. For more background and discussion, see:
* https://software-srn.sandia.gov/jira/browse/SPAR-277
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2995Framework: Autotester may be busted?2018-06-21T13:54:55ZJames WillenbringFramework: Autotester may be busted?*Created by: csiefer2*
Been waiting 5 hours now for the testing to finish... See #2988
@trilinos/framework *Created by: csiefer2*
Been waiting 5 hours now for the testing to finish... See #2988
@trilinos/framework https://gitlab.osti.gov/jmwille/Trilinos/-/issues/2974Trilinos GCC 4.8.4 PR build and running of tests is broken2018-06-19T20:25:41ZJames WillenbringTrilinos GCC 4.8.4 PR build and running of tests is broken*Created by: bartlettroscoe*
@trilinos/framework, @eric-c-cyr, @rppawlo, @trilinos/nox
## Expectations
The GCC 4.8.4 auto PR build should work unless the PR code itself is broken.
## Current Behavior
The GCC 4.8.4 auto PR bu...*Created by: bartlettroscoe*
@trilinos/framework, @eric-c-cyr, @rppawlo, @trilinos/nox
## Expectations
The GCC 4.8.4 auto PR build should work unless the PR code itself is broken.
## Current Behavior
The GCC 4.8.4 auto PR build has broken NOX test builds and all of the tests in every package fail, starting in the auto PR testing iteration build last night:
* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3627737&filtercount=3&showfilters=1&field1=groupname&compare1=61&value1=Pull%20Request&field2=buildstarttime&compare2=84&value2=NOW&field3=buildstarttime&compare3=83&value3=1%20week%20ago&filtercombine=and
YOu can see the NOX failures at:
* https://testing-vm.sandia.gov/cdash/viewBuildError.php?buildid=3627737
And all of the tests fail because `--bind-to none` is being used with `mpiexec` with OpenMP 1.6.5 as shown, for example, at:
* https://testing-vm.sandia.gov/cdash/testDetails.php?test=48792061&build=3627756
which shows the commandline:
```
"/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.8.4/openmpi/1.6.5/bin/mpiexec \"--bind-to\" \"none\" \"-np\" \"1\" \"/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/pull_request_test/packages/kokkos-kernels/unit_test/KokkosKernels_blas_serial.exe\" \"--gtest_filter=-serial.gemm_double\"
```
As described in #2462 and #2788, you can't use those options with OpenMPI 1.6.5. OpenMPI 1.10.1 is the only version in the SEMS env where those options work (see #2462 and #2788).
It is important to fix this ASAP since every PR build will fail and therefore no one will be able to merge any PRs until this gets fixed.
## Motivation and Context
I can't merge my PR #2964.
## Definition of Done
PR builds are working unless code changes break them.
## Possible Solution
Use the source and *.cmake scripts in #2788. That will fix it.
## Steps to Reproduce
## Your Environment
N.A. This is the auto PR testing having this issue.
## Related Issues
* Blocks: #2964
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2550Pull request testing uses really old OpenMPI 1.6.5 w/ build warnings2018-06-19T17:37:07ZJames WillenbringPull request testing uses really old OpenMPI 1.6.5 w/ build warnings*Created by: mhoemmen*
Here is an example of the resulting build warnings:
https://testing-vm.sandia.gov/cdash/viewBuildError.php?type=1&buildid=3418378
```
In file included from /projects/sems/install/rhel6-x86_64/sems/compiler/...*Created by: mhoemmen*
Here is an example of the resulting build warnings:
https://testing-vm.sandia.gov/cdash/viewBuildError.php?type=1&buildid=3418378
```
In file included from /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.9.3/openmpi/1.6.5/include/mpi.h:253:0,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Comm/ml_comm.h:26,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Main/ml_1level.h:31,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Smoother/ml_smoother.h:40,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Main/ml_struct.h:31,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/src/Include/ml_include.h:18,
from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3@2/Trilinos/packages/ml/test/RefMaxwell/cxx_main.cpp:24:
[CTest: warning matched] /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.9.3/openmpi/1.6.5/include/mpi_portable_platform.h:374:34: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
_STRINGIFY(__GNUC__)"."_STRINGIFY(__GNUC_MINOR__)"."_STRINGIFY(__GNUC_PATCHLEVEL__)
^
[CTest: warning matched] /projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.9.3/openmpi/1.6.5/include/mpi_portable_platform.h:374:63: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
_STRINGIFY(__GNUC__)"."_STRINGIFY(__GNUC_MINOR__)"."_STRINGIFY(__GNUC_PATCHLEVEL__)
^
```
OpenMPI calls their own version 1.6.5 "retired" and the next older one is called "ancient." Sierra uses 1.10.x.
@trilinos/framework https://gitlab.osti.gov/jmwille/Trilinos/-/issues/1304Address basic stability of Trilinos 'develop' branch short-term2018-06-14T14:45:43ZJames WillenbringAddress basic stability of Trilinos 'develop' branch short-term*Created by: bartlettroscoe*
**Related to:** #1362
**Next Action Status:**
The auto PR testing process (#1155) is deployed and is working fairly well to stabilize 'develop' (at least as good or better than the checkin-test-sems....*Created by: bartlettroscoe*
**Related to:** #1362
**Next Action Status:**
The auto PR testing process (#1155) is deployed and is working fairly well to stabilize 'develop' (at least as good or better than the checkin-test-sems.sh script did). Further improvements will be worked in other issues.
**Description:**
This story is to discuss and decide how to address stability problems of the Trilinos 'develop' branch in the short term. I know there is a long-term plan to use a PR model (see #1155) but since there are no updates or ETA on that, we need to address stability issues faster than that.
Currently there have been a good deal of stability problems of the Trilinos 'develop' branch, even with the basic CI build linked to from:
* https://github.com/trilinos/Trilinos/wiki/Policies--%7C-Testing#post_push_ci_testing
and the "Clean" builds shown here:
* http://testing.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=2&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=Clean&field2=buildstarttime&compare2=84&value2=now
The "Clean" builds have never been clean in the entire history of the track.
Some very recent examples of failures causing this are described in #1290 and #1301. These have broken the standard CI build and the "Clean" builds continuously since May 4 (and it is still broken as I type this).
We need a strategy to improve stability right now. I have been helping people set up to use the [checkin-test-sems.sh](https://github.com/trilinos/Trilinos/wiki/Policies-%7C-Safe-Checkin-Testing) script to test and push their changes. I would estimate that a large percentage of the failures (and 100% of the CI failures) seen on CDash would be avoided by usage of the checkin-test-sems.sh script.
**CC:** @trilinos/framework Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2864Auto PR build failures due to Intel License server problems2018-06-01T18:11:28ZJames WillenbringAuto PR build failures due to Intel License server problems*Created by: bartlettroscoe*
CC: @trilinos/framework
## Expectations
Auto PR builds should be robust and not fail unless there is a failure in code itself.
## Current Behavior
The new Intel auto PR build fails randomly due ...*Created by: bartlettroscoe*
CC: @trilinos/framework
## Expectations
Auto PR builds should be robust and not fail unless there is a failure in code itself.
## Current Behavior
The new Intel auto PR build fails randomly due to Intel license server problems such as shown at:
* https://github.com/trilinos/Trilinos/pull/2860#issuecomment-393742327
which showed the build failure:
* https://testing-vm.sandia.gov/cdash/viewBuildError.php?buildid=3564874
which showed:
```
Error: A license for Comp-CL is not available now (-15,570,115).
A connection to the license server could not be made. You should
make sure that your license daemon process is running: both an
lmgrd process and an INTEL process should be running
if your license limits you to a specified number of licenses in use
at a time. Also, check to see if the wrong port@host or the wrong
license file is being used, or if the port or hostname in the license
file has changed.
License file(s) used were (in this order):
1. Trusted Storage
** 2. /projects/sems/install/rhel6-x86_64/sems/compiler/intel/17.0.1/base/Licenses/intel-Linux-SRN.lic
** 3. /projects/sems/install/rhel6-x86_64/sems/compiler/intel/17.0.1/base/compilers_and_libraries_2017.1.132/linux/bin/intel64/../../Licenses
** 4. /ascldap/users/trilinos/Licenses
```
The reason that I noticed this was because it happened in my PR #2860.
Looking at the current PR build history at:
* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=buildname&compare1=63&value1=-test-Trilinos_pullrequest_intel_17.0.1&field2=buildstarttime&compare2=84&value2=2018-06-02&field3=buildstarttime&compare3=83&value3=2018-05-14
out of 10 builds, it failed twice with these Intel license server problems. That is an only 80% success rate so far. That is not robust enough for an auto PR build.
## Motivation and Context
Auto PR builds block what goes into the 'develop' branch and long delays make things harder.
## Definition of Done
Auto PR builds should only fail due to non-code issues very infrequently.
## Possible Solution
Don't know.
## Steps to Reproduce
Don't know.
## Your Environment
N.A. This is the auto PR builds that define their own env.
Improve productivity, stability, and quality of Trilinoshttps://gitlab.osti.gov/jmwille/Trilinos/-/issues/2292Trilinos "Clean" and auto PR builds need a Trilinos_ENABLE_DEBUG=ON build2018-05-30T18:37:02ZJames WillenbringTrilinos "Clean" and auto PR builds need a Trilinos_ENABLE_DEBUG=ON build*Created by: bartlettroscoe*
**CC:** @trilinos/framework, @maherou, @rppawlo
## Description
It would seem that all of the current "Clean" builds of Trilinos shown for example yesterday at:
* https://testing.sandia.gov/cdash/in...*Created by: bartlettroscoe*
**CC:** @trilinos/framework, @maherou, @rppawlo
## Description
It would seem that all of the current "Clean" builds of Trilinos shown for example yesterday at:
* https://testing.sandia.gov/cdash/index.php?project=Trilinos&date=2018-02-21&filtercount=1&showfilters=1&field1=groupname&compare1=61&value1=Clean
all have `Trilinos_ENABLE_DEBUG=OFF` set. You can see this, for example, by looking at the uploaded CMakeCache.txt files for these three builds at:
* https://testing.sandia.gov/cdash/viewNotes.php?buildid=3398366##note0
* https://testing.sandia.gov/cdash/viewNotes.php?buildid=3398237##note3
* https://testing.sandia.gov/cdash/viewNotes.php?buildid=3398203##note3
which all show:
```
Trilinos_ENABLE_DEBUG:BOOL=OFF
```
This is not a good thing because there are a lot of run-time checks turned on with you configure Trilinos with `-DTrilinos_ENABLE_DEBUG=ON`. It catches a lot of undefined and otherwise illegal behavior that a `Trilinos_ENABLE_DEBUG=OFF` does not catch.
Because none of the "Clean" builds have `-DTrilinos_ENABLE_DEBUG=ON` one would assume that none of the auto PR builds have it set either. Therefore, can that auto PR builds have at least one build that has this turned on. And from looking at recent PRs like:
* https://github.com/trilinos/Trilinos/pull/2289#issuecomment-367782068
it looks like the auto PR tester is now only running one build. If that is the case, it is critical that this one build set `-DTrilinos_ENABLE_DEBUG=ON`.
This is a big issue for supporting developers and users of Trilinos and especially for ATDM builds of Trilinos that set `-DTrilinos_ENABLE_DEBUG=ON`. For example, this let skip through defects like #2270.