This MR changes our build infrastructure to use Spack:
An overview of the configuration:
-
setup/spack.yaml
: Contains the system independent configuration. This is mostly a shim around thepy-spinifel
Spack package. Do not add new dependencies here. -
setup/spack-repo/packages/py-spinifel/package.py
: Contains the guts of our Spinifel package. Note that this is just a "bundle" package which means it contains dependencies only and no real build step. This is where new dependencies should be added. -
setup/machines/${MACHINE}.yaml
: Contains the system dependent configuration. Should be limited tocompilers
,packages
andmodules
sections only. (E.g., this is where we tell it what compiler and MPI to use, where we enable CUDA/ROCm, etc.) -
setup/machines/${MACHINE}.lock
: The concretized environment for a given machine. Should contain the exact versions of all dependencies (including transitive dependencies). In theory, if we can build it once, we can reproduce this any time. -
setup/spack
: The repository of Spack itself. Currently pinned to a recentdevelop
branch commit because we need very recent changes. (Edit: moved to a fork temporarily because we need changes not yet merged upstream.)
Some other notable features of this change:
- I ripped out a lot of machine-specific configuration. To the extent possible, I want everything to be done with Spack now. We can move even more into Spack later, but this is a start.
- In particular, we do NOT use the system-specific FFTW. This has been moved into Spack. (This means giving up on the Cray FFT library. I think this is fine in practice because we will never run on a Cray machine that does not have GPUs.)
- I removed a bunch of obsolete machines (e.g., Cori and Cori GPU) that do not exist anymore.
- In order to make (re)builds fast, I assume that a build cache has been established on every machine. This is just a directory in a shared location (like a project home directory) that is group-writable and special guid bit set.
The build has been confirmed to work on Perlmutter, Crusher and Frontier. CI is passing on Crusher. There are issues on Summit/Ascent, but that machine is going away soon so I don't plan to fix them.