TpetraCore tests failing in ATDM sems-rhel7+cuda+complex build
Created by: fryeguy52
CC: @trilinos/tpetra, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52
??: Add label "ATDM"> ??: Add label "bug"?> ??: Add label for affected packages (e.g. "MueLu", "Tpetra", "Kokkos", etc.)> ??: Add milestone "Initial cleanup of new ATDM builds of Trilinos" or "Keep promoted ATDM builds of Trilinos clean"> ??: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files> ??: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>Next Action Status
Description
As shown in this query the build:
- Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
has several failing TpetraCore tests. The following are failing everyday (CDash)
- TpetraCore_gemm_m_eq_1_MPI_1
- TpetraCore_gemm_m_eq_1_MPI_2
- TpetraCore_gemm_m_eq_1_MPI_5
- TpetraCore_gemm_m_eq_1_MPI_13
While these have failed randomly over the last couple weeks (CDash):
- TpetraCore_MultiVector_MicroBenchmark_MPI_1
- TpetraCore_Map_Bug5822_2_MPI_2
- TpetraCore_getEntryOnHost_MPI_1
- TpetraCore_gemv_MPI_1
- TpetraCore_deep_copy_MultiVector_to_SerialDenseMatrix_MPI_1
- TpetraCore_createMirrorView_MPI_1
several more tests have failed over the last 2 weeks in similar complex shared build shown here:
- Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-shared-release-debug
Those tests are:
- TpetraCore_MatrixMatrix_UnitTests_MPI_4
- TpetraCore_Issue_607_MPI_4
- TpetraCore_Issue_114_MPI_4
- TpetraCore_Issue601_MPI_4
- TpetraCore_ImportExport2_UnitTests_MPI_4
- TpetraCore_ImportBug5430_MPI_4
- TpetraCore_Import_Union_MPI_4
- TpetraCore_CrsMatrix_NonlocalSumInto_Ignore_MPI_4
- TpetraCore_CrsMatrix_gaussSeidel_MPI_4
- TpetraCore_CrsGraph_getNumDiags_MPI_4
- TpetraCore_AddProfiling_UnitTests_MPI_4
Current Status on CDash
Failed TpetraCore tests for the current testing day
Steps to Reproduce
One should be able to reproduce this failure on with a sems rhel6 environment as described in:
More specifically, the commands given for with a sems rhel6 environment are provided at:
The exact commands to reproduce this issue should be:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-sems-rhel7-cuda-9.2-Volta70-complex-static-release-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Tpetra=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j8