KokkosAlgorithms_UnitTest_MPI_1 Fails on POWER8 with GCC 4.9 and CUDA 7.5
Created by: nmhamster
@crtrott @hcedwar I am getting a test failure in Kokkos Algorithms XOR Random on POWER8 with GCC4.9 and CUDA 7.5 builds.
$ ctest -VV -R KokkosAlgorithms_UnitTest_MPI_1
UpdateCTestConfiguration from :/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/DartConfiguration.tcl
Parse Config file:/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/DartConfiguration.tcl
Add coverage exclude regular expressions.
SetCTestConfiguration:CMakeCommand:/home/projects/pwr8-rhel72/cmake/3.4.3/bin/cmake
UpdateCTestConfiguration from :/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/DartConfiguration.tcl
Parse Config file:/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/DartConfiguration.tcl
Test project /ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 10
Start 10: KokkosAlgorithms_UnitTest_MPI_1
10: Test command: /ascldap/users/projects/pwr8-rhel72/openmpi/1.10.2/gcc/4.9.2/cuda/7.5.7/bin/mpiexec "-np" "1" "/ascldap/users/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-757-test-ramdisk/packages/kokkos/algorithms/unit_tests/KokkosAlgorithms_UnitTest.exe"
10: Test timeout computed to be: 1500
10: [==========] Running 9 tests from 3 test cases.
10: [----------] Global test environment set-up.
10: [----------] 3 tests from cuda
10: Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 3.5 on device with compute capability 3.7 , this will likely reduce potential performance.
10: [ RUN ] cuda.Random_XorShift64
10: Test Scalar=int
10: -- Testing randomness properties
10: Pass: 1 1 -2.82471e-06 -3.40352e-05 -0.000104384 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.43252e-07 -0.0022995 -0.00836735 || 0.051031 27960 29418 || 28740.5 28674.4 || -115.67 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 -0.0169805 0.0015321 || 0.051031 1e+64 -1e+64
10: Test Scalar=unsigned int
10: -- Testing randomness properties
10: Pass: 1 1 -3.41644e-05 7.42232e-05 -3.04288e-05 || 0.000173985
10: -- Testing 1-D histogram
10: Density 1D: 6.43252e-07 0.0193418 0.0128208 || 0.051031 28059 29295 || 28130.3 28674.4 || 177.234 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 -0.0150665 0.00138121 || 0.051031 1e+64 -1e+64
10: Test Scalar=int64_t
10: -- Testing randomness properties
10: Mean: 2.867646e+04
10: Pass: 1 1 -3.6109e-05 -8.63585e-06 4.63777e-06 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.43252e-07 0.0115433 0.00712385 || 0.051031 28067 29309 || 28347.2 28674.4 || 98.4801 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 0.0200474 0.00931228 || 0.051031 1e+64 -1e+64
10: Test Scalar=uint64_t
10: -- Testing randomness properties
10: Pass: 1 1 2.42433e-05 -4.91036e-05 -1.69406e-05 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.43252e-07 0.0161715 -0.0122164 || 0.051031 27984 29362 || 28218.1 28674.4 || -168.88 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 0.0144323 -0.00292772 || 0.051031 1e+64 -1e+64
10: Test Scalar=float
10: -- Testing randomness properties
10: Pass: 1 1 5.75692e-05 9.05598e-11 -6.08529e-05 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.68478e-07 -0.00302028 0.0181608 || 0.051031 28019 29426 || 28761.3 28674.4 || 251.054 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 7.18929e-07 0.0221568 0.00457695 || 0.051031 1e+64 -1e+64
10: Test Scalar=double
10: -- Testing randomness properties
10: Pass: 1 1 1.06213e-05 5.18704e-05 3.61577e-07 || 0.000173985
10: -- Testing 1-D histogram
10: Mean: 2.867646e+04
10: Density 1D: 6.43252e-07 0.0144485 -0.0405824 || 0.051031 28009 29477 || 28266 28674.4 || -561.011 -2.0744
10: -- Testing 3-D histogram
10: Density 3D: 6.43252e-07 0.0018761 -0.00990339 || 0.051031 1e+64 -1e+64
10: [ OK ] cuda.Random_XorShift64 (33438 ms)
10: [ RUN ] cuda.Random_XorShift1024
10: Test Scalar=int
10: -- Testing randomness properties
10: Pass: 1 1 -3.73188e-05 8.34822e-05 -0.000165954 || 0.000276214
10: -- Testing 1-D histogram
10: Mean: 1.137778e+04
10: /ascldap/users/sdhammo/git/trilinos-github-repo/packages/kokkos/algorithms/unit_tests/TestRandom.hpp:396: Failure
10: Value of: 1
10: Expected: test_int.pass_hist1d_var
10: Which is: 0
10: [ FAILED ] cuda.Random_XorShift1024 (2396 ms)
10: [ RUN ] cuda.SortUnsigned
10: Density 1D: 2.47955e-07 -0.0611763 -0.000749968 || 0.051031 10969 11798 || 12118.3 11377 || -10.3676 -0.823045
10: -- Testing 3-D histogram
10: Density 3D: 2.47955e-07 0.0076215 -0.000964689 || 0.051031 1e+64 -1e+64
10: --------------------------------------------------------------------------
10: mpiexec noticed that process rank 0 with PID 48917 on node white26 exited on signal 11 (Segmentation fault).
10: --------------------------------------------------------------------------
1/1 Test #10: KokkosAlgorithms_UnitTest_MPI_1 ...***Failed Error regular expression found in output. Regex=[ FAILED ] 46.83 sec
0% tests passed, 1 tests failed out of 1
Label Time Summary:
Kokkos = 46.83 sec (1 test)
Total Test time (real) = 47.29 sec
The following tests FAILED:
10 - KokkosAlgorithms_UnitTest_MPI_1 (Failed)
Errors while running CTest