Address expensive PT pre-push BASIC Trilinos tests
Created by: bartlettroscoe
CC: @trilinos/framework, @trilinos/rol, @trilinos/teko, @trilinos/zoltan
Description
The initial PT pre-push CI test suite (see #482 (closed)) currently has some pretty expensive tests. Looking a the first post-push CI iteration this morning, we see the 2259 tests and their times at:
We see that there are 9 tests that take over 1 minute to run and they drop down to under 30s once you get past the 18 most expensive tests. These tests are:
| Test Name | Status | Time |
|---|---|---|
| KokkosContainers_UnitTest_MPI_1 | Passed | 2m 45s 30ms |
| ROL_example_PDE-OPT_topo-opt_elasticity_example_02_MPI_4 | Passed | 2m 35s 300ms |
| ROL_example_PDE-OPT_0ld_adv-diff-react_example_02_MPI_4 | Passed | 1m 59s 280ms |
| Teko_testdriver_tpetra_MPI_1 | Passed | 1m 36s 880ms |
| Zoltan_hg_simple_zoltan_parallel | Passed | 1m 28s 370ms |
| ROL_example_PDE-OPT_0ld_poisson_example_01_MPI_4 | Passed | 1m 14s 460ms |
| Zoltan_ch_simple_zoltan_parallel | Passed | 1m 7s 150ms |
| ROL_example_PDE-OPT_topo-opt_elasticity_example_01_MPI_4 | Passed | 1m 7s 40ms |
| ROL_example_burgers-control_example_06_MPI_1 | Passed | 1m 4s 170ms |
| MueLu_ParameterListInterpreterTpetra_MPI_1 | Passed | 58s 70ms |
| ROL_test_sol_solSROMGenerator_MPI_1 | Passed | 55s 390ms |
| Teko_testdriver_tpetra_MPI_4 | Passed | 54s 950ms |
| KokkosAlgorithms_UnitTest_MPI_1 | Passed | 48s 850ms |
| Rythmos_IntegratorBuilder_ConvergenceTest_MPI_1 | Passed | 44s 730ms |
| ROL_example_PDE-OPT_ginzburg-landau_example_02_MPI_4 | Passed | 35s 810ms |
| KokkosCore_UnitTest_Serial_MPI_1 | Passed | 35s 220ms |
| ROL_example_parabolic-control_example_03_MPI_1 | Passed | 31s 70ms |
| Rythmos_BackwardEuler_ConvergenceTest_MPI_1 | Passed | 29s 700ms |
| ... | ... | ... |
After that, all of the remaining tests 2241 of the 2259 tests take less than 30s to run in this build.
What I would like to propose is that we make all of these tests that take longer than 1 minute and either do some work to make them run in under 1 minute or we change their category from the default BASIC to CONTINUOUS using the argument CATEGORY CONTINUOUS.
And I would like to make it a policy that we try to keep the test times for pre-push BASIC tests to under 1 minute. If we can do that, then we can make the pre-push test suite run faster. If you have just a single test that takes 2m45s to run, then no matter how many processes your throw at it, the test suite will never run faster than 2m45s. That is the major motivation here for focusing on individual test times.
As part of this, we can consider how long each package's test suite takes to run by looking at the package-by-package summary this morning at:
which when sorted by Test Time looks like:
| Package | Test time |
|---|---|
| ROL | 6m 34s |
| Kokkos | 2m 45s |
| Zoltan | 2m 1s |
| Teko | 1m 37s |
| Panzer | 1m 28s |
| MueLu | 1m 25s |
| Zoltan2 | 55s |
| Tpetra | 54s |
| Rythmos | 51s |
| ... | ... |
If we could cut down on these tests that take over 1 minute, I think it would also help to shorten the test time for each package as well.
Does anyone from any of these packages have any objections to relegating these tests from BASIC to CONTINUOUS using the argument CATEGORY CONTINUOUS? If so, is it possible to make these run a little faster for pre-push testing?
Note that changing from BASIC to CONTINUOUS does not impact what tests get enabled by default when you configure Trilinos manually since the default test category enabled is NIGHTLY. See: