spinifel merge requestshttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests2023-02-09T23:19:58Zhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/55Legion improvements for soft reset + support multiple conformations in legion...2023-02-09T23:19:58ZSeema MirchandaneyLegion improvements for soft reset + support multiple conformations in legion mode (non-streaming)Seema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/70New merge algorithm - in Legion2023-10-06T21:29:25ZSeema MirchandaneyNew merge algorithm - in LegionUpdate Legion to support the new merge algorithm that allows for convergence in the case of a small # of images per rankUpdate Legion to support the new merge algorithm that allows for convergence in the case of a small # of images per rankSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/69Legion fix for issue #452023-08-11T23:41:56ZSeema MirchandaneyLegion fix for issue #45results.results.Seema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/68support show_image option in legion2023-07-26T00:12:16ZSeema Mirchandaneysupport show_image option in legionsupport show_image option in legionsupport show_image option in legionSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/67Frontier related - contains legion updates and merged djh/crusher branch2023-07-10T19:51:37ZSeema MirchandaneyFrontier related - contains legion updates and merged djh/crusher branchFrontier changes for legion + merged djh/crusher branchFrontier changes for legion + merged djh/crusher branchSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/66Support F-ordering on Crusher/Frontier2023-07-06T20:08:19ZMonarin UervirojnangkoornSupport F-ordering on Crusher/FrontierMonarin UervirojnangkoornMonarin Uervirojnangkoornhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/65Update min dist for conformations that have converged2023-07-17T18:20:17ZSeema MirchandaneyUpdate min dist for conformations that have converged@yoon82 please review@yoon82 please reviewSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/64Support Legion based multiple conformations with mode=max_likelihood and mode...2023-06-12T16:12:08ZSeema MirchandaneySupport Legion based multiple conformations with mode=max_likelihood and mode=test_debughttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/63Add intensity clipping with arbitrary input hdf5 with known orientations2023-05-15T23:57:16ZChun Hong YoonAdd intensity clipping with arbitrary input hdf5 with known orientations+ apply intensity clipping to help with orientation matching convergence (do not use by default)
+ known orientations from general hdf5 file
@Seemah we may need this change in legion version
CI passed+ apply intensity clipping to help with orientation matching convergence (do not use by default)
+ known orientations from general hdf5 file
@Seemah we may need this change in legion version
CI passedChun Hong YoonChun Hong Yoonhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/62Legion FSC related opts + code cleanup2023-05-08T21:05:16ZSeema MirchandaneyLegion FSC related opts + code cleanup- replace futures with regions
- check for early convergence
- code cleanup- replace futures with regions
- check for early convergence
- code cleanupSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/61Legion FSC related opts + code cleanup2023-05-08T18:03:09ZSeema MirchandaneyLegion FSC related opts + code cleanup- replace futures with regions
- check for early convergence- replace futures with regions
- check for early convergencehttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/60legion: multiple conformations support (algorithm changes needed) + improve C...2023-05-03T02:46:11ZSeema Mirchandaneylegion: multiple conformations support (algorithm changes needed) + improve CI tests + misc cleanup- add legion support for multiple conformations for all modes (psana2, streaming, hdf5)
- improve legion CI tests
- misc cleanup- add legion support for multiple conformations for all modes (psana2, streaming, hdf5)
- improve legion CI tests
- misc cleanupSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/59Legion: multiple conformations + misc improvements2023-05-02T21:56:16ZSeema MirchandaneyLegion: multiple conformations + misc improvements- Improve legion related CI tests
- Support multiple conformations (non algorithm related) for legion (psana2, hdf5, streaming, multirank)
- Misc cleanup- Improve legion related CI tests
- Support multiple conformations (non algorithm related) for legion (psana2, hdf5, streaming, multirank)
- Misc cleanupSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/58remove SPINIFEL_TEST_FLAG2023-04-28T19:04:06ZChun Hong Yoonremove SPINIFEL_TEST_FLAG@monarin @Seemah
removing SPINIFEL_TEST_FLAG@monarin @Seemah
removing SPINIFEL_TEST_FLAGChun Hong YoonChun Hong Yoonhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/57Sm/legion fixes2023-04-13T21:49:19ZSeema MirchandaneySm/legion fixesVerbosity support in legion (remove verbose)Verbosity support in legion (remove verbose)Seema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/56fix over-fetching slices2023-03-29T23:58:30ZChun Hong Yoonfix over-fetching slicesI was getting an error when hdf5 file only has 10k images with following toml setting.
I'm using 3 ranks in mpi mode: -n3 -G3
N_images_per_rank = 1500
N_batch_size = 500
N_images_max = 3000
Fetching logic was somewhat complex, but I ho...I was getting an error when hdf5 file only has 10k images with following toml setting.
I'm using 3 ranks in mpi mode: -n3 -G3
N_images_per_rank = 1500
N_batch_size = 500
N_images_max = 3000
Fetching logic was somewhat complex, but I hope this fixes it.Chun Hong YoonChun Hong Yoonhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/54Fix CI by pinning all packages2023-01-13T18:52:17ZElliott SlaughterFix CI by pinning all packagesFixes https://gitlab.osti.gov/mtip/spinifel/-/issues/61Fixes https://gitlab.osti.gov/mtip/spinifel/-/issues/61https://gitlab.osti.gov/mtip/spinifel/-/merge_requests/53FY22 demo development into master2022-11-08T18:22:08ZChun Hong YoonFY22 demo development into mastercontains FY22 legion and psana2 streaming modecontains FY22 legion and psana2 streaming modeChun Hong YoonChun Hong Yoonhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/52scripts for demo + track mem usage + flexible streaming in Legion2022-11-03T23:24:58ZSeema Mirchandaneyscripts for demo + track mem usage + flexible streaming in LegionDemo scripts + track mem usage + flexible streaming in LegionDemo scripts + track mem usage + flexible streaming in LegionSeema MirchandaneySeema Mirchandaneyhttps://gitlab.osti.gov/mtip/spinifel/-/merge_requests/51Reference volume from MRC or PDB file2022-10-29T01:18:46ZAriana PeckReference volume from MRC or PDB fileTo address a numba-related incompatibility on Crusher, the code was adjusted so that either a PDB or MRC file can be provided as the reference. If the latter, the density array is simply loaded rather than computed from a coordinates fil...To address a numba-related incompatibility on Crusher, the code was adjusted so that either a PDB or MRC file can be provided as the reference. If the latter, the density array is simply loaded rather than computed from a coordinates file using skopi, which depends on numba. The relevant entry in the toml file has been changed from `pdb_path` to `ref_path`. This was tested on Cori GPU as follows:
```
srun -n 1 -G 1 python -m spinifel --default-settings=cgpu_mpi.toml --mode=mpi
```
with the `chk_convergence` flag set to true.
@yoon82, is there a Crusher-specific toml file? I wasn't sure which toml files should use an MRC rather than a PDB file; for now, I've only updated the cgpu_mpi.toml to use the former.