Jumbled multi-process output causing random failure of test FEI_fei_ubase_MPI_4 in Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_CI build
Created by: bartlettroscoe
CC: @trilinos/framework, @trilinos/fei
Description:
The test FEI_fei_ubase_MPI_4
failed in the CI build Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_CI
shown at:
The detailed test output at:
showed:
Sorting tests by group name then by the order they were added ... (time = 0.000271)
Running unit tests ...
...
41. fei_UBase_slaveMatrix_UnitTest ... test Eqns_unit.feiInitSlave only runs on 2 procs. returning.
[Passed] (4.77e-06 sec)
...
Total Time: 0.00719 sec
Summary: total = 54, run = 54, passed = 54, failed = 0
End Result: TESTTeuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1113 and rank 1!
skipping test of fei::DirichletBCManager::finalizeBCEqn, which only runs on 1 proc.
test Eqns_unit.feiInitSlave only runs on 2 procs. returning.
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1113 and rank 2!
skipping test of fei::DirichletBCManager::finalizeBCEqn, which only runs on 1 proc.
test Eqns_unit.feiInitSlave only runs on 2 procs. returning.
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ceerws1113 and rank 3!
skipping test of fei::DirichletBCManager::finalizeBCEqn, which only runs on 1 proc.
test Eqns_unit.feiInitSlave only runs on 2 procs. returning.
PASSED
The problem is the jumbled output from the other MPI processes that broke up "End Result: PASSED" that is looked for in the regex for pass/fail for the test.
There are now changes reported for this CI iteration shown at:
that would explain this failure.
Therefore, I suspect that is a random fluke which may not happen again for a while. However, it might be nice to fix the test code to avoid printing on multiple processes other than the root process or manage the parallel outputting or something.
Related Issues
- Related to: #1304 (closed)