'#line 1' from flex generated apr_scanner.cpp SEACAS file causing unnecessary rebuilds when using Ninja + CUDA-8
Created by: bartlettroscoe
CC: @trilinos/seacas, @fryeguy52
Description
The following line in the SEACAS file:
packages/seacas/libraries/aprepro_lib/apr_scanner.cc:974:#line 1 "/scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll"
is causing the ninja CUDA build for Trilinos with SEACAS to rebuild the same files every time, even when everything should be up-to-date.
The cause for this is very strange. There is a defect with the nvcc
program (at least CUDA 8.0.61) when running nvcc -M -Wno-deprecated-gpu-targets
on a file that has #line 1 <some-file>
in the file results in nvcc
incorrectly spitting out a dependency on <some-file>
. This results in nvcc
when run under nvcc_wrapper
run under ninja to produce a dependency on the file <some-file>
. Apparently this does not happen for any number other than 1
. For example, lines with #line 2 <some-file>
do not cause nvcc
to print out a bad dependency.
Kitware reported this NVIDIA as bug number 2080680 just today. (However, their issue tracker is not public).
The workaround for this is to make the single one-line change:
diff --git a/packages/seacas/libraries/aprepro_lib/apr_scanner.cc b/packages/seacas/libraries/aprepro_lib/apr_scanner.cc
index 747470a..a2512cd 100644
--- a/packages/seacas/libraries/aprepro_lib/apr_scanner.cc
+++ b/packages/seacas/libraries/aprepro_lib/apr_scanner.cc
@@ -971,7 +971,6 @@ static yyconst flex_int16_t yy_rule_linenum[102] = {
#define yymore() yymore_used_but_not_detected
#define YY_MORE_ADJ 0
#define YY_RESTORE_YY_MORE_OFFSET
-#line 1 "/scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll"
/* -*- Mode: c++ -*- */
#line 39 "/scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll"
Also, changing #line 1 <file-name>
to #line 2 <file-name>
also fixes the problem.
Steps to Reproduce
This can be produced, for example, with one of the ATDM Trilinos CUDA configurations described at:
For example, just log onto hansen
or shiller
or white
or ride
, clone Trilinos somewhere, and then configure and build with:
cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-opt
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_Panzer=ON -DPanzer_ENABLE_TESTS=ON \
$TRILINOS_DIR
$ ninja -j16 # Builds everything the first time
$ ninja -j16 # Rebuilds a bunch of targets!
You can see what is triggering the rebuild using:
$ ninja -d explain
...
ninja explain: output /scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll of phony edge with no inputs doesn't exist
ninja explain: /scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll is dirty
ninja explain: packages/seacas/libraries/aprepro_lib/CMakeFiles/aprepro_lib.dir/apr_scanner.cc.o is dirty
...
$ ninja -t deps packages/seacas/libraries/aprepro_lib/CMakeFiles/aprepro_lib.dir/apr_scanner.cc.o
...
/scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll
...
See, that file /scratch/gdsjaar/seacas/packages/seacas/libraries/aprepro_lib/aprepro.ll
is not on this system obviously.
Then to fix, just apply the patch shown above, rebuild once and then rebuild again, and you will get:
$ time ninja
ninja: no work to do.
real 0m2.744s
user 0m0.275s
sys 0m0.218s
(Yes, that is ninja evaluating all of the targets to figure out if it needs to build any of the Panzer tests in under 3 sec! That is why we want to be able to use ninja.)