spinifel merge requestshttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests2022-10-29T01:18:46Zhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/51Reference volume from MRC or PDB file2022-10-29T01:18:46ZAriana PeckReference volume from MRC or PDB fileTo address a numba-related incompatibility on Crusher, the code was adjusted so that either a PDB or MRC file can be provided as the reference. If the latter, the density array is simply loaded rather than computed from a coordinates fil...To address a numba-related incompatibility on Crusher, the code was adjusted so that either a PDB or MRC file can be provided as the reference. If the latter, the density array is simply loaded rather than computed from a coordinates file using skopi, which depends on numba. The relevant entry in the toml file has been changed from `pdb_path` to `ref_path`. This was tested on Cori GPU as follows:
```
srun -n 1 -G 1 python -m spinifel --default-settings=cgpu_mpi.toml --mode=mpi
```
with the `chk_convergence` flag set to true.
@yoon82, is there a Crusher-specific toml file? I wasn't sure which toml files should use an MRC rather than a PDB file; for now, I've only updated the cgpu_mpi.toml to use the former.https://gitlab.osti.gov/mtip/spinifel/-/merge_requests/44Install dependencies with Spack2023-07-25T06:35:45ZElliott SlaughterInstall dependencies with SpackThis MR changes our build infrastructure to use Spack:
An overview of the configuration:
* `setup/spack.yaml`: Contains the *system independent* configuration. This is mostly a shim around the `py-spinifel` Spack package. *Do not add ...This MR changes our build infrastructure to use Spack:
An overview of the configuration:
* `setup/spack.yaml`: Contains the *system independent* configuration. This is mostly a shim around the `py-spinifel` Spack package. *Do not add new dependencies here.*
* `setup/spack-repo/packages/py-spinifel/package.py`: Contains the guts of our Spinifel package. Note that this is just a "bundle" package which means it contains dependencies only and no real build step. *This is where new dependencies should be added.*
* `setup/machines/${MACHINE}.yaml`: Contains the *system dependent* configuration. Should be limited to `compilers`, `packages` and `modules` sections only. (E.g., this is where we tell it what compiler and MPI to use, where we enable CUDA/ROCm, etc.)
* `setup/machines/${MACHINE}.lock`: The concretized environment for a given machine. Should contain the exact versions of all dependencies (including transitive dependencies). In theory, if we can build it once, we can reproduce this any time.
* `setup/spack`: The repository of Spack itself. Currently pinned to a recent `develop` branch commit because we need very recent changes. (Edit: moved to a fork temporarily because we need changes not yet merged upstream.)
Some other notable features of this change:
* I ripped out a lot of machine-specific configuration. To the extent possible, I want everything to be done with Spack now. We can move even more into Spack later, but this is a start.
* In particular, we do **NOT** use the system-specific FFTW. This has been moved into Spack. (This means giving up on the Cray FFT library. I think this is fine in practice because we will never run on a Cray machine that does not have GPUs.)
* I removed a bunch of obsolete machines (e.g., Cori and Cori GPU) that do not exist anymore.
* In order to make (re)builds fast, I assume that a build cache has been established on every machine. This is just a directory in a shared location (like a project home directory) that is group-writable and special guid bit set.
The build has been confirmed to work on Perlmutter, Crusher and Frontier. CI is passing on Crusher. There are issues on Summit/Ascent, but that machine is going away soon so I don't plan to fix them.https://gitlab.osti.gov/mtip/spinifel/-/merge_requests/36Start working on updating how we do device management2022-03-07T16:48:45ZJohannes Paul BlaschkeStart working on updating how we do device managementA lot of the C++ stuff (in `spinfel/device`) will likely move to a seperate PyPI package soon. This MR makes two changes:
1. Get the device count from the CUDA API. I checked and this respects `CUDA_VISIBLE_DEVICES` on Cori GPU, Perlmutt...A lot of the C++ stuff (in `spinfel/device`) will likely move to a seperate PyPI package soon. This MR makes two changes:
1. Get the device count from the CUDA API. I checked and this respects `CUDA_VISIBLE_DEVICES` on Cori GPU, Perlmutter, and Summit.
2. Remove the `gpu.devices_per_node` setting -- this is controlled by `srun` or `CUDA_VISIBLE_DEVICES` now.
RE point 2 above: you need to remove the
```
[gpu]
devices_per_node = ...
```
section of your own tomls.
This also fixed a bug where the orientation matching code would use the setting rather than `context.dev_id`Johannes Paul BlaschkeJohannes Paul Blaschkehttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/33adding log parser2022-02-11T01:28:44ZVinay Bharadwaj Ramakrishnaiahadding log parserVinay Bharadwaj RamakrishnaiahVinay Bharadwaj Ramakrishnaiahhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/29Upgrade to FINUFFT v2.0.32022-02-02T19:25:57ZElliott SlaughterUpgrade to FINUFFT v2.0.3Note: this doesn't work on Cori GPU because it does not supply a single-precision FFTW at the moment. If you want me to fix that, we can build our own FFTW, just let me know.Note: this doesn't work on Cori GPU because it does not supply a single-precision FFTW at the moment. If you want me to fix that, we can build our own FFTW, just let me know.https://gitlab.osti.gov/mtip/spinifel/-/merge_requests/23remove NERSC_HOST in setting target2021-10-15T21:31:15ZChun Hong Yoonremove NERSC_HOST in setting targetMain motivation here is to not set SPINIFEL_TARGET variable on cori gpu.
$NERSC_HOST is not needed, and produces wrong target if the user forgets to set SPINIFEL_TARGET.
If SPINIFEL_TARGET is needed on other machines we can keep it, oth...Main motivation here is to not set SPINIFEL_TARGET variable on cori gpu.
$NERSC_HOST is not needed, and produces wrong target if the user forgets to set SPINIFEL_TARGET.
If SPINIFEL_TARGET is needed on other machines we can keep it, otherwise we could remove this variable too.Johannes Paul BlaschkeJohannes Paul Blaschke