spinifel merge requestshttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests2023-05-08T18:03:09Zhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/61Legion FSC related opts + code cleanup2023-05-08T18:03:09ZSeema MirchandaneyLegion FSC related opts + code cleanup- replace futures with regions
- check for early convergence- replace futures with regions
- check for early convergencehttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/59Legion: multiple conformations + misc improvements2023-05-02T21:56:16ZSeema MirchandaneyLegion: multiple conformations + misc improvements- Improve legion related CI tests
- Support multiple conformations (non algorithm related) for legion (psana2, hdf5, streaming, multirank)
- Misc cleanup- Improve legion related CI tests
- Support multiple conformations (non algorithm related) for legion (psana2, hdf5, streaming, multirank)
- Misc cleanupSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/48Add psana2 with streaming2022-09-26T16:27:39ZMonarin UervirojnangkoornAdd psana2 with streamingThis merge fixes some issues with the current mpi-psana2 (new comm_compute is added) and add psana2 streaming option (with mode=psana2).
The reference check gives following result at Generation 20 when running on corigpu (following Qui...This merge fixes some issues with the current mpi-psana2 (new comm_compute is added) and add psana2 streaming option (with mode=psana2).
The reference check gives following result at Generation 20 when running on corigpu (following QuickStart guideline).
###########################
##### Generation 20/20 #####
###########################.
Match tot:158.65s. slice=83.71s. match=73.88s. slice_oh=0.00s. match_oh=1.05s.
Orientations matched in 161.10s.
WARNING: CG did not converge at rlambda = 0.009174451535658204.
summary = [(0, 0.009174451535658204, 0.38474797725707166, 0.28914907118689465)].
Keeping result from rank 0: v1=0.38474797725707166 and v2=0.28914907118689465.
Rank 0 got AC in 2055 iterations.
AC recovered in 98.38s.
heights: [439.].
widths: [0].
skews: [1.0156696].
Keeping result from rank 0: skew=1.02.
Problem phased in 91.75s.
Alignment CC after rotation is: 0.789.
Map had to be inverted.
Final CC between unzoomed / unfiltered volumes is: 0.789.
Estimated resolution from FSC: 21.8 Angstrom.
Results saved in /global/cscratch1/sd/monarin/spinifel_output.
Successfully completed in 7274.91s.
There are some issues seen on Ascent that may need to be addressed.
- MPI Call after Finalize (issue#55) https://gitlab.osti.gov/mtip/spinifel/-/issues/55
- pytest fails when call modules that import mpi4py (issue#56 - only on Ascent/Summit) https://gitlab.osti.gov/mtip/spinifel/-/issues/56Chun Hong YoonChun Hong Yoonhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/42Draft: Fsc optimizations2022-08-08T16:16:34ZNiteya ShahDraft: Fsc optimizationsCurrently, I have modified the existing implementation by monkey patching in my changes as there is clear way to create a flag that says enable cupy. Unfortunately, this makes the code both clumsy and brittle. As it is currently implemen...Currently, I have modified the existing implementation by monkey patching in my changes as there is clear way to create a flag that says enable cupy. Unfortunately, this makes the code both clumsy and brittle. As it is currently implemented, I see no good way to make the changes I wish for as the Spinifel Settings, which decide if cupy can be used requires parameters not needed for FSC calculation. Also, running FSC calculations separately, as requested, is not a clean change either. I could again do it in the way I have made these changes if required.
The best way I think FSC should be handled is that it becomes a callable module, much like spinifel, with the option of running it in a headless mode for when we want to run it as part of our core iterations.
Another way is that FSC starts depending on a configuration file for its parameters. This would be less intrusive at least for running it separately but running it as part of the core iteration would require us to add more parameters to spinifel settings to support FSC.Niteya ShahNiteya Shahhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/40Build on Crusher2023-07-10T21:48:30ZDarren Jason HsuBuild on CrusherThis merge request is to enable building of Spinifel on Crusher. To enable so, I made the following changes:
In `.gitmodules`:
1. changed dependency on `pycuda` to `pybindgpu` by @jpblaschke ~(currently this points to a fork branch tha...This merge request is to enable building of Spinifel on Crusher. To enable so, I made the following changes:
In `.gitmodules`:
1. changed dependency on `pycuda` to `pybindgpu` by @jpblaschke ~(currently this points to a fork branch that can be installed with `pip install -e .`)~
1. changed `cufinufft` submodule pointer ~to my branch~ that also has the `pycuda` dependency replaced with `pybindgpu` (now pulled back into eds/hip)
In `setup/`:
1. added in build-from-scratch script instruction to build `pybindgpu` and `cufinufft` on Crusher/Frontier
1. ~changed module use path in `mk_env.sh` to use a newer llvm compiler as well as force both cupy and cufinufft to use the same `hipfft.so` on Cursher~ updated to ROCm 5.4.3 so this is no longer necessary
In `settings/`:
1. added test settings for Crusher in settings
In `./`:
1. added Crusher submission scriptMonarin UervirojnangkoornMonarin Uervirojnangkoornhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/39Optimized version of the development branch2022-07-27T22:45:52ZNiteya ShahOptimized version of the development branchThis branch uses the GPU more aggressively, has a more optimized orientation algorithm and fixes some of the resolution problems seen in the development branch.This branch uses the GPU more aggressively, has a more optimized orientation algorithm and fixes some of the resolution problems seen in the development branch.Niteya ShahNiteya Shahhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/31Chy/development2022-02-01T23:02:50ZChun Hong YoonChy/developmentAdding functions from cmtip. Verified convergence 2 out of 5 times for high resolution 3iyf dataset.Adding functions from cmtip. Verified convergence 2 out of 5 times for high resolution 3iyf dataset.Iris ChangIris Changhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/17Offload phasing to GPU using Cupy2021-07-13T19:16:25ZIris ChangOffload phasing to GPU using CupyThis is for offloading phasing to GPU using Cupy.
Changes include: (1) add `pip install --no-cache-dir cupy` in `setup/build_from_scratch.sh`; (2) add `use_cupy` in `spinifel/settings.py`; (3) add `use_cupy` under [runtime] in toml file;...This is for offloading phasing to GPU using Cupy.
Changes include: (1) add `pip install --no-cache-dir cupy` in `setup/build_from_scratch.sh`; (2) add `use_cupy` in `spinifel/settings.py`; (3) add `use_cupy` under [runtime] in toml file; (3) replace NumPy with Cupy in `sequential/phasing.py`.
Note that one needs to `export CUPY_CACHE_DIR=$PWD/setup/cupy` before the jsrun command.https://gitlab.osti.gov/mtip/spinifel/-/merge_requests/14Development2021-06-29T18:09:51ZMonarin UervirojnangkoornDevelopmentLooks like CI is happy.Looks like CI is happy.Monarin UervirojnangkoornMonarin Uervirojnangkoornhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/10Chy/settings2021-06-08T06:06:37ZChun Hong YoonChy/settingsgeneral cleaning of codegeneral cleaning of codeChun Hong YoonChun Hong Yoonhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/6Fixes required for Legion to be able to run cufinufft2021-07-06T22:07:54ZElliott SlaughterFixes required for Legion to be able to run cufinufftThere are two changes here:
1. pycuda stores the context in a thread-local. Since Legion executes each task in a thread, this must be set up each time we use the GPU.
2. Some precision changes to match MPI.There are two changes here:
1. pycuda stores the context in a thread-local. Since Legion executes each task in a thread, this must be set up each time we use the GPU.
2. Some precision changes to match MPI.https://gitlab.osti.gov/mtip/spinifel/-/merge_requests/3Development2021-04-22T20:26:08ZMonarin UervirojnangkoornDevelopmentThis is currently tested with the the short ./scripts/run_summit.sh -s -e test and the full test (see readme.md) and seems like there's no problem.This is currently tested with the the short ./scripts/run_summit.sh -s -e test and the full test (see readme.md) and seems like there's no problem.Monarin UervirojnangkoornMonarin Uervirojnangkoorn