Switch from OpenMPI 3.1.0 to 2.1.2 on 'waterman' (TRIL-213)
Created by: bartlettroscoe
CC: @fryeguy52, @mhoemmen, @kddevin, @rppawlo
Description
Switches from OpenMPI 3.1.0 to OpenMPI 2.1.2 env on Power9 'waterman'. We were told by @nmhamster today that OpenMPI 3.1 really does not work on the Power9 (and is a known issue apparently) and to use the OpenMPI 2.1.2 env instead.
Motivation and Context
This appears to fix a bunch of failing tests including those in #3344 (closed), #3331 (closed) and perhaps others.
How Has This Been Tested?
On 'white' I ran:
$ bsub -x -Is -n 20 \
./checkin-test-atdm.sh cuda-opt-Power9-Volta70 \
--enable-packages=Kokkos,Teuchos,Zoltan2,Ifpack2,Tpetra,SEACAS,Panzer \
--local-do-all
and it returned:
99% tests passed, 2 tests failed out of 645
Subproject Time Summary:
Ifpack2 = 646.49 sec*proc (36 tests)
Kokkos = 475.50 sec*proc (27 tests)
Panzer = 7880.59 sec*proc (158 tests)
SEACAS = 24.01 sec*proc (20 tests)
Teuchos = 207.66 sec*proc (129 tests)
Tpetra = 2326.59 sec*proc (173 tests)
Zoltan2 = 1269.06 sec*proc (102 tests)
Total Test time (real) = 1888.47 sec
The following tests FAILED:
619 - PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 (Failed)
623 - PanzerAdaptersSTK_MixedPoissonExample-ConvTest-Hex-Order-3 (Timeout)
This is worth trying on the full ATDM Trilinos build.