GitLab

Created by: bartlettroscoe

CC: @trilinos/seacas, @fryeguy52

Description

The following line in the SEACAS file:

packages/seacas/libraries/aprepro_lib/apr_scanner.cc:974:#line 1 "/scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll"

is causing the ninja CUDA build for Trilinos with SEACAS to rebuild the same files every time, even when everything should be up-to-date.

The cause for this is very strange. There is a defect with the nvcc program (at least CUDA 8.0.61) when running nvcc -M -Wno-deprecated-gpu-targets on a file that has #line 1 <some-file> in the file results in nvcc incorrectly spitting out a dependency on <some-file>. This results in nvcc when run under nvcc_wrapper run under ninja to produce a dependency on the file <some-file>. Apparently this does not happen for any number other than 1. For example, lines with #line 2 <some-file> do not cause nvcc to print out a bad dependency.

Kitware reported this NVIDIA as bug number 2080680 just today. (However, their issue tracker is not public).

The workaround for this is to make the single one-line change:

diff --git a/packages/seacas/libraries/aprepro_lib/apr_scanner.cc b/packages/seacas/libraries/aprepro_lib/apr_scanner.cc
index 747470a..a2512cd 100644
--- a/packages/seacas/libraries/aprepro_lib/apr_scanner.cc
+++ b/packages/seacas/libraries/aprepro_lib/apr_scanner.cc
@@ -971,7 +971,6 @@ static yyconst flex_int16_t yy_rule_linenum[102] = {
 #define yymore() yymore_used_but_not_detected
 #define YY_MORE_ADJ 0
 #define YY_RESTORE_YY_MORE_OFFSET
-#line 1 "/scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll"
 /* -*- Mode: c++ -*- */
 #line 39 "/scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll"

Also, changing #line 1 <file-name> to #line 2 <file-name> also fixes the problem.

Steps to Reproduce

This can be produced, for example, with one of the ATDM Trilinos CUDA configurations described at:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

For example, just log onto hansen or shiller or white or ride, clone Trilinos somewhere, and then configure and build with:

 cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-opt

$ cmake \
  -GNinja \
  -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
  -DTrilinos_ENABLE_Panzer=ON -DPanzer_ENABLE_TESTS=ON \
  $TRILINOS_DIR

$ ninja -j16  # Builds everything the first time

$ ninja -j16  # Rebuilds a bunch of targets!

You can see what is triggering the rebuild using:

$ ninja -d explain
...
ninja explain: output /scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll of phony edge with no inputs doesn't exist
ninja explain: /scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll is dirty
ninja explain: packages/seacas/libraries/aprepro_lib/CMakeFiles/aprepro_lib.dir/apr_scanner.cc.o is dirty
...
$ ninja -t deps packages/seacas/libraries/aprepro_lib/CMakeFiles/aprepro_lib.dir/apr_scanner.cc.o
...
    /scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll
...

See, that file /scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll is not on this system obviously.

Then to fix, just apply the patch shown above, rebuild once and then rebuild again, and you will get:

$ time ninja
ninja: no work to do.

real    0m2.744s
user    0m0.275s
sys     0m0.218s

(Yes, that is ninja evaluating all of the targets to figure out if it needs to build any of the Panzer tests in under 3 sec! That is why we want to be able to use ninja.)