Test Teko_testdriver_tpetra_MPI_1 failing in new ATDM Trilinos GCC 6.1.0 RHEL6 builds
Created by: bartlettroscoe
CC: @trilinos/teko, @fryeguy52
Next Action Status
PR #2656 merged on 4/28/2018 which disabled just this one unit test for just these RHEL6 builds. This test has not failed since then for the RHEL6 builds as of 5/3/2018!
Description
As shown in the query:
The Teko test Teko_testdriver_tpetra_MPI_1 is failing in all of the new RHEL6 ATDM Trilinos builds:
- Trilinos-atdm-rhel6-gnu-debug-openmp
- Trilinos-atdm-rhel6-gnu-debug-serial
- Trilinos-atdm-rhel6-gnu-opt-openmp
- Trilinos-atdm-rhel6-gnu-opt-serial
- Trilinos-atdm-rhel6-intel-opt-openmp
These builds are using the SEMS GCC 6.1.0 compiler, which looks to be the only GCC 6x builds going up to the Trilinos CDash site.
The failure shown in the output shows:
...
Teko: LSCPrecFact::buildPO TotalTime = 0.0128801
"strategy" ... FAILED ( PID = 0 )
Test "LSCStabilized_tpetra" completed ... FAILED (1)
...
Teko: Building LSC strategy "The Cat"
LSC Construction failed: Strategy "The Cat" could not be constructed
Teko: Begin debug MSG
Looked up "NS LSC"
Built Teuchos::RCP<Teko::PreconditionerFactory>{ptr=0x5f1a7d8,node=0x5eafbe0,strong_count=1,weak_count=0}
Teko: End debug MSG
LSC Construction failed: Strategy "The Cat" requires a "Strategy Settings" sublist
...
Tests Passed: 85, Tests Failed: 1
(Incidently, you want no failures)
Teko tests failedFull test output (click to expand)
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name hansel.sandia.gov and rank 0!
Running test "SIMPLEPreconditionerFactory_tpetra"
"createPrec" ... PASSED
Teko: Begin debug MSG
SIMPLE Parameters:
inv type = ""
inv v type = "Ifpack2"
inv p type = "Ifpack2"
alpha = 1
use mass = 0
vel scaling = Lumped
SIMPLE Parameter list:
Explicit Velocity Inverse Type = Lumped
Inverse Pressure Type = Ifpack2
Inverse Velocity Type = Ifpack2
Teko: End debug MSG
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
"initializePrec(lumped)" ... PASSED
Teko: Begin debug MSG
SIMPLE Parameters:
inv type = ""
inv v type = "Ifpack2"
inv p type = "Ifpack2"
alpha = 1
use mass = 0
vel scaling = Diagonal
SIMPLE Parameter list:
Explicit Velocity Inverse Type = Diagonal
Inverse Pressure Type = Ifpack2
Inverse Velocity Type = Ifpack2
Teko: End debug MSG
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
"initializePrec(diag)" ... PASSED
Teko: Begin debug MSG
SIMPLE Parameters:
inv type = ""
inv v type = "Ifpack2"
inv p type = "Ifpack2"
alpha = 1
use mass = 0
vel scaling = AbsRowSum
SIMPLE Parameter list:
Explicit Velocity Inverse Type = AbsRowSum
Inverse Pressure Type = Ifpack2
Inverse Velocity Type = Ifpack2
Teko: End debug MSG
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
"initializePrec(absrowsum)" ... PASSED
"uninitializePrec" ... PASSED
"isCompatable" ... PASSED
"diagonal(diag)" ... PASSED
"result(diag)" ... PASSED
Test "SIMPLEPreconditionerFactory_tpetra" completed ... PASSED (8)
Running test "DiagonalPreconditionerFactory_tpetra"
"createPrec" ... PASSED
"initializePrec" ... PASSED
||Z-Y||/||Z|| = 0
"canApply" ... PASSED
Test "DiagonalPreconditionerFactory_tpetra" completed ... PASSED (3)
Running test "LU2x2PreconditionerFactory_tpetra"
"createPrec" ... PASSED
"initializePrec" ... PASSED
"uninitializePrec" ... PASSED
"isCompatable" ... PASSED
"identity" ... PASSED
"diagonal" ... PASSED
"result" ... PASSED
"alphabeta" ... PASSED
Test "LU2x2PreconditionerFactory_tpetra" completed ... PASSED (8)
Running test "LSCStablePreconditionerFactory_tpetra"
"createPrec" ... PASSED
Teko: LSCPrecFact::buildPO BuildStateTime = 9.53674e-07
Teko: LSCPrecFact::buildPO GetInvTime = 4.05312e-06
Teko: LSCPrecFact::buildPO TotalTime = 5.79357e-05
"initializePrec" ... PASSED
"uninitializePrec" ... PASSED
"isCompatable" ... PASSED
Teko: LSCPrecFact::buildPO BuildStateTime = 0
Teko: LSCPrecFact::buildPO GetInvTime = 2.86102e-06
Teko: LSCPrecFact::buildPO TotalTime = 5.00679e-05
"identity" ... PASSED
Teko: LSCPrecFact::buildPO BuildStateTime = 0
Teko: LSCPrecFact::buildPO GetInvTime = 1.90735e-06
Teko: LSCPrecFact::buildPO TotalTime = 5.6982e-05
"diagonal" ... PASSED
Teko: LSCPrecFact::buildPO BuildStateTime = 0
Teko: LSCPrecFact::buildPO GetInvTime = 9.53674e-07
Teko: LSCPrecFact::buildPO TotalTime = 5.48363e-05
"result" ... PASSED
Test "LSCStablePreconditionerFactory_tpetra" completed ... PASSED (7)
Running test "LSCStabilized_tpetra"
Teko: LSCPrecFact::buildPO BuildStateTime = 9.53674e-07
Teko: LSCPrecFact::buildPO GetInvTime = 3.09944e-06
Teko: LSCPrecFact::buildPO TotalTime = 6.50883e-05
"diagonal" ... PASSED
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: LSC::initializeState Build Scaling <mass> type "Diagonal"
Teko: Begin debug MSG
LSC Gamma Parameter = 0.237413
LSC Alpha Parameter = 0.667104
Teko: End debug MSG
Teko: LSC::buildState BuildOpsTime = 0.012218
Teko: LSC::computeInverses Building inv(F)
Teko: LSC::computeInverses GetInvF = 0.000306845
Teko: LSC::computeInverses Building inv(BQBtmC)
Teko: LSC::computeInverses GetInvBQBt = 0.000249863
Teko: LSC::buildState BuildInvTime = 0.000586033
Teko: LSCPrecFact::buildPO BuildStateTime = 0.0128162
Teko: LSCPrecFact::buildPO GetInvTime = 3.09944e-06
Teko: LSCPrecFact::buildPO TotalTime = 0.0128801
"strategy" ... FAILED ( PID = 0 )
Test "LSCStabilized_tpetra" completed ... FAILED (1)
Running test "Jacobi2x2PreconditionerFactory_tpetra"
"createPrec" ... PASSED
"initializePrec" ... PASSED
"uninitializePrec" ... PASSED
"isCompatable" ... PASSED
"identity" ... PASSED
"diagonal" ... PASSED
"result" ... PASSED
Teko: Begin debug MSG
Looked up "Block Jacobi"
Built Teuchos::RCP<Teko::PreconditionerFactory>{ptr=0x5dcf4d8,node=0x5d438d0,strong_count=1,weak_count=0}
Teko: End debug MSG
Teko: JacobiPrecFact: Building default inverse "Ifpack2"
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: Begin debug MSG
Looked up "Block Jacobi"
Built Teuchos::RCP<Teko::PreconditionerFactory>{ptr=0x5d8e0a8,node=0x5d438d0,strong_count=1,weak_count=0}
Teko: End debug MSG
Teko: JacobiPrecFact: Building default inverse "ML"
Teko: Inverse "ML" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: JacobiPrecFact: Building inverse 1 "Amesos"
Teko: Inverse "Amesos" is of type strat prec = 0, strat solv = 1, block prec = 0
Teko: JacobiPrecFact: Building inverse 3 "Ifpack"
Teko: Inverse "Ifpack" is of type strat prec = 1, strat solv = 0, block prec = 0
"initializeFromParameterList" ... PASSED
Test "Jacobi2x2PreconditionerFactory_tpetra" completed ... PASSED (8)
Running test "BlockJacobiPreconditionerFactory_tpetra"
"createPrec" ... PASSED
"initializePrec" ... PASSED
"uninitializePrec" ... PASSED
"isCompatible" ... PASSED
Test "BlockJacobiPreconditionerFactory_tpetra" completed ... PASSED (4)
Running test "BlockUpperTriInverseOp_tpetra"
"apply" ... PASSED
"alphabeta" ... PASSED
Test "BlockUpperTriInverseOp_tpetra" completed ... PASSED (2)
Running test "BlockLowerTriInverseOp_tpetra"
"apply" ... PASSED
"alphabeta" ... PASSED
Test "BlockLowerTriInverseOp_tpetra" completed ... PASSED (2)
Running test "tTpetraOperatorWrapper"
"functionality" ... PASSED
Test "tTpetraOperatorWrapper" completed ... PASSED (1)
Running test "InterlacedTpetra"
"buildSubMaps_num" ... PASSED
"buildSubMaps_vec" ... PASSED
"buildMaps" ... PASSED
"one2many" ... PASSED
"many2one" ... PASSED
Test "InterlacedTpetra" completed ... PASSED (5)
Running test "BlockingTpetra"
"buildMaps" ... PASSED
"one2many" ... PASSED
"many2one" ... PASSED
"buildSubBlock" ... PASSED
Test "BlockingTpetra" completed ... PASSED (4)
Running test "TpetraThyraConverter"
"blockThyraToTpetra" ... PASSED
"single_blockThyraToTpetra" ... PASSED
"blockTpetraToThyra" ... PASSED
"single_blockTpetraToThyra" ... PASSED
Test "TpetraThyraConverter" completed ... PASSED (4)
Running test "tGraphLaplacian_tpetra"
"single_array" ... PASSED
"multi_array" ... PASSED
Test "tGraphLaplacian_tpetra" completed ... PASSED (2)
Running test "tParallelInverse_tpetra"
Teko: Inverse "Belos" is of type strat prec = 0, strat solv = 1, block prec = 0
"inverse" ... PASSED
Test "tParallelInverse_tpetra" completed ... PASSED (1)
Running test "tExplicitOps_tpetra"
"mult_diagScaleMatProd" ... PASSED
"mult_diagScaling" ... PASSED
"add" ... PASSED
"mult_modScaleMatProd" ... PASSED
"add_mod" ... PASSED
Test "tExplicitOps_tpetra" completed ... PASSED (5)
Running test "LSCHIntegrationTest_tpetra"
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: LSC::initializeState Build Scaling <mass> type "Diagonal"
Teko: LSC::buildState BuildOpsTime = 0.000967979
Teko: LSC::computeInverses Building inv(F)
Teko: LSC::computeInverses GetInvF = 0.000282049
Teko: LSC::computeInverses Building inv(BQBtmC)
Teko: LSC::computeInverses GetInvBQBt = 0.000233173
Teko: LSC::computeInverses Building inv(BHBtmC)
Teko: LSC::computeInverses GetInvBHBt = 0.000244856
Teko: LSC::buildState BuildInvTime = 0.000801086
Teko: LSCPrecFact::buildPO BuildStateTime = 0.00178909
Teko: LSCPrecFact::buildPO GetInvTime = 2.86102e-06
Teko: LSCPrecFact::buildPO TotalTime = 0.00185704
"hScaling" ... PASSED
Test "LSCHIntegrationTest_tpetra" completed ... PASSED (1)
Running test "Lumping_tpetra"
"lumping" ... PASSED
"invLumping" ... PASSED
Test "Lumping_tpetra" completed ... PASSED (2)
Running test "AbsRowSum_tpetra"
"absRowSum" ... PASSED
"invAbsRowSum" ... PASSED
Test "AbsRowSum_tpetra" completed ... PASSED (2)
Running test "NeumannSeries_tpetra"
Teko: Inverse "Neumann" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: Inverse "Neumann" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
"test_simpleOp" ... PASSED
Teko: Inverse "Neumann" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
"test_scaledOp" ... PASSED
Test "NeumannSeries_tpetra" completed ... PASSED (2)
Running test "PCDStrategy_tpetra"
Teko: Inverse "Ifpack2" is of type strat prec = 1, strat solv = 0, block prec = 0
"PCDStrategy" ... PASSED
Test "PCDStrategy_tpetra" completed ... PASSED (1)
Running test "LSCIntegrationTest_tpetra"
Teko: LSC::initializeState Build Scaling <mass> type "Diagonal"
Teko: LSC::buildState BuildOpsTime = 0.0380211
Teko: LSC::computeInverses Building inv(F)
Teko: LSC::computeInverses GetInvF = 0.108474
Teko: LSC::computeInverses Building inv(BQBtmC)
Teko: LSC::computeInverses GetInvBQBt = 0.0198691
Teko: LSC::buildState BuildInvTime = 0.128404
Teko: LSCPrecFact::buildPO BuildStateTime = 0.166463
Teko: LSCPrecFact::buildPO GetInvTime = 4.05312e-06
Teko: LSCPrecFact::buildPO TotalTime = 0.166535
"withmassStable" ... PASSED
Teko: LSC::initializeState Build Scaling <F> type "Diagonal"
Teko: LSC::buildState BuildOpsTime = 0.0350621
Teko: LSC::computeInverses Building inv(F)
Teko: LSC::computeInverses GetInvF = 0.101603
Teko: LSC::computeInverses Building inv(BQBtmC)
Teko: LSC::computeInverses GetInvBQBt = 0.018666
Teko: LSC::buildState BuildInvTime = 0.120294
Teko: LSCPrecFact::buildPO BuildStateTime = 0.15537
Teko: LSCPrecFact::buildPO GetInvTime = 1.90735e-06
Teko: LSCPrecFact::buildPO TotalTime = 0.155421
"nomassStable" ... PASSED
Teko: Begin debug MSG
Looked up "NS LSC"
Built Teuchos::RCP<Teko::PreconditionerFactory>{ptr=0x6278158,node=0x5eafbe0,strong_count=1,weak_count=0}
Teko: End debug MSG
Teko: Building LSC strategy "Basic Inverse"
Teko: Begin debug MSG
LSC Inverse Strategy Parameters:
inv type = "Amesos"
inv v type = "Ifpack"
inv p type = "Ifpack"
bndry rows = 1
use ldu = 1
use mass = 0
use w-scaling = 0
assume stable = 0
scale type = Diagonal
LSC Inverse Strategy Parameter list:
Inverse Type = Amesos
Inverse Velocity Type = Ifpack
Inverse Pressure Type = Ifpack
Ignore Boundary Rows = 1
Use LDU = 1
Teko: End debug MSG
Teko: Inverse "Ifpack" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: Begin debug MSG
Looked up "NS LSC"
Built Teuchos::RCP<Teko::PreconditionerFactory>{ptr=0x5f22be8,node=0x5dfeb80,strong_count=1,weak_count=0}
Teko: End debug MSG
Teko: Building LSC strategy "Basic Inverse"
Teko: Begin debug MSG
LSC Inverse Strategy Parameters:
inv type = "Amesos"
inv v type = "Ifpack"
inv p type = "Ifpack"
bndry rows = 1
use ldu = 1
use mass = 0
use w-scaling = 0
assume stable = 0
scale type = Diagonal
LSC Inverse Strategy Parameter list:
Inverse Type = Amesos
Inverse Velocity Type = Ifpack
Inverse Pressure Type = Ifpack
Ignore Boundary Rows = 1
Use LDU = 1
Teko: End debug MSG
Teko: Inverse "Ifpack" is of type strat prec = 1, strat solv = 0, block prec = 0
Teko: Begin debug MSG
Looked up "NS LSC"
Built Teuchos::RCP<Teko::PreconditionerFactory>{ptr=0x5e06f88,node=0x5eafbe0,strong_count=1,weak_count=0}
Teko: End debug MSG
Teko: Building LSC strategy "The Cat"
LSC Construction failed: Strategy "The Cat" could not be constructed
Teko: Begin debug MSG
Looked up "NS LSC"
Built Teuchos::RCP<Teko::PreconditionerFactory>{ptr=0x5f1a7d8,node=0x5eafbe0,strong_count=1,weak_count=0}
Teko: End debug MSG
LSC Construction failed: Strategy "The Cat" requires a "Strategy Settings" sublist
"plConstruction" ... PASSED
Test "LSCIntegrationTest_tpetra" completed ... PASSED (3)
Running test "tStridedTpetraOperator"
"numvars_constr" ... PASSED
"vector_constr" ... PASSED
"reorder(flat reorder)" ... PASSED
"reorder(composite reorder = 1)" ... PASSED
"reorder(composite reorder = 2)" ... PASSED
Test "tStridedTpetraOperator" completed ... PASSED (5)
Running test "tBlockedTpetraOperator"
"vector_constr" ... PASSED
"reorder(flat reorder)" ... PASSED
"reorder(composite reorder = 1)" ... PASSED
"reorder(composite reorder = 2)" ... PASSED
Test "tBlockedTpetraOperator" completed ... PASSED (4)
Tests Passed: 85, Tests Failed: 1
(Incidently, you want no failures)
Teko tests failed
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[21024,1],0]
Exit code: 255
--------------------------------------------------------------------------Steps to Reproduce
One should be able to reproduce this on any SNL COE RHEL6 machine that has the SEMS env as described in:
For example, on any SNL COE RHEL6 SEMS machine, one should be able to reproduce this failing test with:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh gnu-debug-serial
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j16