Teuchos: VerboseObject fix filestreams
Created by: jjellio
Currently, VerboseObject provides an option to allow output to be redirected to a file. Unfortunately, this feature results in every MPI process opening/writing the file, which is not typically what an OStream is used to accomplish. For file output, this implementation leads to extreme performance degradation with large parallel jobs.
@trilinos/teuchos @bartlettroscoe @mhoemmen
Expectations
VerboseObject should write to a single file (and be as performant at stdout.
Current Behavior
Currently, we open multiple files and typically write exactly the same data to the same file. This gives the illusion of working as intended. Since the file will actually be correct at the end of the run. The issue is that every MPI process is writing the data, and this is not intended.
Motivation and Context
ATDM apps would like to use this functionality. We uncovered/identified this issue by running with VerboseObject output to files with >64 nodes.
Possible Solution
This fix is straightforward, and is addressed in PR#4629
(This is an issue to map to the PR... I should create the issues first in the future ... woops!)