Address "Error 127" when trying to build libs with ETI and COMPLEX enabled
Created by: bartlettroscoe
CC: @trilinos/framework, @trilinos/kokkos, @trilinos/kokkos-kernels
Next Action Status
Waiting to resolve all of the issues with CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON
(e.g. #2115 (closed)) before trying to automate turning this on ...
Description
This issue is to address the problem with the library creation command failing with "Error 127" with KokkosKernels with CUDA, ETI, and COMPLEX enabled described in kokkos/kokkos-kernels#141. The issue is that Kokkos and KokkosKernels create a bunch of *.cpp files, one for each function, instantiation set, and scalar type. This quickly explodes to hundreds of *.cpp files per library and therefore hundreds of *.o files when creating a library. It seems that some compilers choke on that many object files and just return "Error 127". Christian found the CMake option CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS
that when set to TRUE
will put the list of all of the object files into a *.rsp
file that is then passed to the command to create the library. This was reported to solve the problem described in kokkos/kokkos-kernels#141.
It turns out that the option CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS
is not documented in the official CMake documentation. There are just a few references to it when you Google. Apparently, this option is only meant to be set in a compiler toolchain file for a given system. However, Google searches mention this options working with Linux GCC compilers, Windows Visual Studio compilers, and MAC OS-X. Therefore, we might hope that it is somewhat portable.
The proposed solution that we will attempt to implement in this Issue is to put in automatic logic to set CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS
when ETI is enabled, COMPLEX is enabled, and KokkosKernels is enabled. I think this case works right now and only fails when CUDA is also enabled which doubles the number of *.o files (one for the host CPU, and one for the GPU). But talking with @crtrott, it seems likely that as KokkosKernels is continued to be developed that we will hit this "Error 127" soon even without CUDA enabled. Therefore, it seems like a good idea to go ahead and turn this on by default when just ETI, COMPLEX and KokkosKernels are enabled.
I will add a NOTE:
to the CMake output remarking that this var is being set and then just cross our fingers that the target system will support this feature. Since there are fairly few clients that will enable ETI, COMPLEX and KokkosKernels, hopefully this will not cause a lot of trouble.
I guess to be safe, we could add a configure-time test to make sure that CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON
works for the selected compilers and other options but that would be a good bit of work (since we would have to create a separate dummy CMake project to try this out and verify that a *.rsp file is getting used). In fact, if we do that, we should just always set CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON
if we determine that response files are supported for the selected compilers and system. Since this will take a bit more work we can avoid doing this for now. But If we discover that this people are getting broken builds due to this logic, we can go ahead and implement the more sophisticated configure-time check and be done with this once and for all.
Related Issues
- Blocks
- Is blocked by
- Follows
- Precedes
- Related to: kokkos/kokkos-kernels#141, #2115 (closed)
- Part of
- Composed of