CHECK_FOR_ERRORS: 
USING_MPI: 1
root_dir: /gpfs/alpine/chm137/scratch/cahrens/devel/spinifel-devel-070821
NRESOURCESETS: 6
NTASKS_PER_RS: 1
DEVICES_PER_RS: 1
NRSS_PER_NODE: 6
OMP_NUM_THREADS: 1
LAUNCH_SCRIPT: spinifel
MPI run
JSRUN_COMMAND: jsrun -n 6 -a 1 -c 1 -g 1 -r 6 -b rs -d packed python -m spinifel --default-settings=summit_quickstart.toml --mode=mpi runtime.n_images_per_rank=1000 data.out_dir=/gpfs/alpine/scratch/cahrens/chm137/spinifel_data/spinifel_output_2021-07-16T1316-0400_TRIAL_080/nodes_1
WARNING! The environment variable VERBOSE supersedes all other inputs for this setting. If this is unintensional unset VERBOSE.
WARNING! The environment variable DATA_DIR supersedes all other inputs for this setting. If this is unintensional unset DATA_DIR.
WARNING! The environment variable DATA_FILENAME supersedes all other inputs for this setting. If this is unintensional unset DATA_FILENAME.
WARNING! The environment variable OUT_DIR supersedes all other inputs for this setting. If this is unintensional unset OUT_DIR.
WARNING! The environment variable DEVICES_PER_RS supersedes all other inputs for this setting. If this is unintensional unset DEVICES_PER_RS.
SpinifelSettings:
  + chk_convergence=False
    source: runtime.chk_convergence, description: if false, no check if output density converges
  + data_dir=/gpfs/alpine/chm137/proj-shared/data/spi
    source: data.in_dir, description: data dir
  + data_filename=2CEX-10k-2.h5
    source: data.name, description: data file name
  + devices_per_node=1
    source: gpu.devices_per_node, description: gpu-device count per node/resource set
  + n_images_per_rank=1000
    source: runtime.n_images_per_rank, description: no. of images per rank
  + out_dir=/gpfs/alpine/scratch/cahrens/chm137/spinifel_data/spinifel_output_2021-07-16T1316-0400_TRIAL_080/nodes_1
    source: data.out_dir, description: output dir
  + ps_eb_nodes=1
    source: psana.ps_eb_nodes, description: no. of eventbuilder cores
  + ps_smd_n_events=10000
    source: psana.ps_smd_n_events, description: no. of events to be sent to an EventBuilder core
  + ps_srv_nodes=0
    source: psana.ps_srv_nodes, description: no. of server cores
  + small_problem=False
    source: runtime.small_problem, description: run in small problem mode
  + test=Quickstart settings for Summit
    source: debug.test, description: test field used for debugging
  + use_callmonitor=False
    source: debug.use_callmonitor, description: enable call-monitor
  + use_cufinufft=True
    source: runtime.use_cufinufft, description: use cufinufft for nufft support
  + use_psana=False
    source: psana.enable, description: enable PSANA
  + use_single_prec=False
    source: runtime.use_single_prec, description: if true, spinifel will use single-precision floating point
  + using_cuda=True
    source: runtime.using_cuda, description: use cuda wherever possible
  + verbose=True
    source: debug.verbose, description: is verbosity > 0
  + verbosity=1
    source: debug.verbosity, description: reporting verbosity

Runtime MODE = mpi
MPI has been initialized on rank 0
Psana functionality is not available.
Rank 0 assigned to device 0
Orientation Matching: USING_CUDA
++++++++++++++++++++: USING_CUFINUFFT
pyCudaKNearestNeighbors is available: True
In MPI main
comm.size : 6
#workers  : 6
writerrank: 0
batch_size: 100
max_events: 6000
parms.N_binning: 3
parms.nER: 50
parms.nHIO: 25
parms.N_phase_loops: 10
parms.N_clipping: 0
parms.N_binning: 3
parms.N_orientations: 2000
parms.N_batch_size: 100
parms.Mquat: 20
get_slices rank=0 st=0 en=1000
Loaded in 12.00s.
###########################
##### Generation 0/10 #####
###########################
WARNING! The environment variable VERBOSE supersedes all other inputs for this setting. If this is unintensional unset VERBOSE.
WARNING! The environment variable DATA_DIR supersedes all other inputs for this setting. If this is unintensional unset DATA_DIR.
WARNING! The environment variable DATA_FILENAME supersedes all other inputs for this setting. If this is unintensional unset DATA_FILENAME.
WARNING! The environment variable OUT_DIR supersedes all other inputs for this setting. If this is unintensional unset OUT_DIR.
WARNING! The environment variable DEVICES_PER_RS supersedes all other inputs for this setting. If this is unintensional unset DEVICES_PER_RS.
SpinifelSettings:
  + chk_convergence=False
    source: runtime.chk_convergence, description: if false, no check if output density converges
  + data_dir=/gpfs/alpine/chm137/proj-shared/data/spi
    source: data.in_dir, description: data dir
  + data_filename=2CEX-10k-2.h5
    source: data.name, description: data file name
  + devices_per_node=1
    source: gpu.devices_per_node, description: gpu-device count per node/resource set
  + n_images_per_rank=1000
    source: runtime.n_images_per_rank, description: no. of images per rank
  + out_dir=/gpfs/alpine/scratch/cahrens/chm137/spinifel_data/spinifel_output_2021-07-16T1316-0400_TRIAL_080/nodes_1
    source: data.out_dir, description: output dir
  + ps_eb_nodes=1
    source: psana.ps_eb_nodes, description: no. of eventbuilder cores
  + ps_smd_n_events=10000
    source: psana.ps_smd_n_events, description: no. of events to be sent to an EventBuilder core
  + ps_srv_nodes=0
    source: psana.ps_srv_nodes, description: no. of server cores
  + small_problem=False
    source: runtime.small_problem, description: run in small problem mode
  + test=Quickstart settings for Summit
    source: debug.test, description: test field used for debugging
  + use_callmonitor=False
    source: debug.use_callmonitor, description: enable call-monitor
  + use_cufinufft=True
    source: runtime.use_cufinufft, description: use cufinufft for nufft support
  + use_psana=False
    source: psana.enable, description: enable PSANA
  + use_single_prec=False
    source: runtime.use_single_prec, description: if true, spinifel will use single-precision floating point
  + using_cuda=True
    source: runtime.using_cuda, description: use cuda wherever possible
  + verbose=True
    source: debug.verbose, description: is verbosity > 0
  + verbosity=1
    source: debug.verbosity, description: reporting verbosity

Runtime MODE = mpi
MPI has been initialized on rank 2
Psana functionality is not available.
Rank 2 assigned to device 0
Orientation Matching: USING_CUDA
++++++++++++++++++++: USING_CUFINUFFT
pyCudaKNearestNeighbors is available: True
get_slices rank=2 st=2000 en=3000
N_images = 1000
N = 16384000
Ntot = 98304000
N_pixels_per_image = 16384.0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
info = 0
summary = None
Rank 2 got AC in 44 iterations.
WARNING! The environment variable VERBOSE supersedes all other inputs for this setting. If this is unintensional unset VERBOSE.
WARNING! The environment variable DATA_DIR supersedes all other inputs for this setting. If this is unintensional unset DATA_DIR.
WARNING! The environment variable DATA_FILENAME supersedes all other inputs for this setting. If this is unintensional unset DATA_FILENAME.
WARNING! The environment variable OUT_DIR supersedes all other inputs for this setting. If this is unintensional unset OUT_DIR.
WARNING! The environment variable DEVICES_PER_RS supersedes all other inputs for this setting. If this is unintensional unset DEVICES_PER_RS.
SpinifelSettings:
  + chk_convergence=False
    source: runtime.chk_convergence, description: if false, no check if output density converges
  + data_dir=/gpfs/alpine/chm137/proj-shared/data/spi
    source: data.in_dir, description: data dir
  + data_filename=2CEX-10k-2.h5
    source: data.name, description: data file name
  + devices_per_node=1
    source: gpu.devices_per_node, description: gpu-device count per node/resource set
  + n_images_per_rank=1000
    source: runtime.n_images_per_rank, description: no. of images per rank
  + out_dir=/gpfs/alpine/scratch/cahrens/chm137/spinifel_data/spinifel_output_2021-07-16T1316-0400_TRIAL_080/nodes_1
    source: data.out_dir, description: output dir
  + ps_eb_nodes=1
    source: psana.ps_eb_nodes, description: no. of eventbuilder cores
  + ps_smd_n_events=10000
    source: psana.ps_smd_n_events, description: no. of events to be sent to an EventBuilder core
  + ps_srv_nodes=0
    source: psana.ps_srv_nodes, description: no. of server cores
  + small_problem=False
    source: runtime.small_problem, description: run in small problem mode
  + test=Quickstart settings for Summit
    source: debug.test, description: test field used for debugging
  + use_callmonitor=False
    source: debug.use_callmonitor, description: enable call-monitor
  + use_cufinufft=True
    source: runtime.use_cufinufft, description: use cufinufft for nufft support
  + use_psana=False
    source: psana.enable, description: enable PSANA
  + use_single_prec=False
    source: runtime.use_single_prec, description: if true, spinifel will use single-precision floating point
  + using_cuda=True
    source: runtime.using_cuda, description: use cuda wherever possible
  + verbose=True
    source: debug.verbose, description: is verbosity > 0
  + verbosity=1
    source: debug.verbosity, description: reporting verbosity

Runtime MODE = mpi
MPI has been initialized on rank 5
Psana functionality is not available.
Rank 5 assigned to device 0
Orientation Matching: USING_CUDA
++++++++++++++++++++: USING_CUFINUFFT
pyCudaKNearestNeighbors is available: True
get_slices rank=5 st=5000 en=6000
N_images = 1000
N = 16384000
Ntot = 98304000
N_pixels_per_image = 16384.0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
info = 0
summary = None
Rank 5 got AC in 39 iterations.
WARNING! The environment variable VERBOSE supersedes all other inputs for this setting. If this is unintensional unset VERBOSE.
WARNING! The environment variable DATA_DIR supersedes all other inputs for this setting. If this is unintensional unset DATA_DIR.
WARNING! The environment variable DATA_FILENAME supersedes all other inputs for this setting. If this is unintensional unset DATA_FILENAME.
WARNING! The environment variable OUT_DIR supersedes all other inputs for this setting. If this is unintensional unset OUT_DIR.
WARNING! The environment variable DEVICES_PER_RS supersedes all other inputs for this setting. If this is unintensional unset DEVICES_PER_RS.
SpinifelSettings:
  + chk_convergence=False
    source: runtime.chk_convergence, description: if false, no check if output density converges
  + data_dir=/gpfs/alpine/chm137/proj-shared/data/spi
    source: data.in_dir, description: data dir
  + data_filename=2CEX-10k-2.h5
    source: data.name, description: data file name
  + devices_per_node=1
    source: gpu.devices_per_node, description: gpu-device count per node/resource set
  + n_images_per_rank=1000
    source: runtime.n_images_per_rank, description: no. of images per rank
  + out_dir=/gpfs/alpine/scratch/cahrens/chm137/spinifel_data/spinifel_output_2021-07-16T1316-0400_TRIAL_080/nodes_1
    source: data.out_dir, description: output dir
  + ps_eb_nodes=1
    source: psana.ps_eb_nodes, description: no. of eventbuilder cores
  + ps_smd_n_events=10000
    source: psana.ps_smd_n_events, description: no. of events to be sent to an EventBuilder core
  + ps_srv_nodes=0
    source: psana.ps_srv_nodes, description: no. of server cores
  + small_problem=False
    source: runtime.small_problem, description: run in small problem mode
  + test=Quickstart settings for Summit
    source: debug.test, description: test field used for debugging
  + use_callmonitor=False
    source: debug.use_callmonitor, description: enable call-monitor
  + use_cufinufft=True
    source: runtime.use_cufinufft, description: use cufinufft for nufft support
  + use_psana=False
    source: psana.enable, description: enable PSANA
  + use_single_prec=False
    source: runtime.use_single_prec, description: if true, spinifel will use single-precision floating point
  + using_cuda=True
    source: runtime.using_cuda, description: use cuda wherever possible
  + verbose=True
    source: debug.verbose, description: is verbosity > 0
  + verbosity=1
    source: debug.verbosity, description: reporting verbosity

Runtime MODE = mpi
MPI has been initialized on rank 3
Psana functionality is not available.
Rank 3 assigned to device 0
Orientation Matching: USING_CUDA
++++++++++++++++++++: USING_CUFINUFFT
pyCudaKNearestNeighbors is available: True
get_slices rank=3 st=3000 en=4000
N_images = 1000
N = 16384000
Ntot = 98304000
N_pixels_per_image = 16384.0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
info = 0
summary = None
Rank 3 got AC in 44 iterations.
WARNING! The environment variable VERBOSE supersedes all other inputs for this setting. If this is unintensional unset VERBOSE.
WARNING! The environment variable DATA_DIR supersedes all other inputs for this setting. If this is unintensional unset DATA_DIR.
WARNING! The environment variable DATA_FILENAME supersedes all other inputs for this setting. If this is unintensional unset DATA_FILENAME.
WARNING! The environment variable OUT_DIR supersedes all other inputs for this setting. If this is unintensional unset OUT_DIR.
WARNING! The environment variable DEVICES_PER_RS supersedes all other inputs for this setting. If this is unintensional unset DEVICES_PER_RS.
SpinifelSettings:
  + chk_convergence=False
    source: runtime.chk_convergence, description: if false, no check if output density converges
  + data_dir=/gpfs/alpine/chm137/proj-shared/data/spi
    source: data.in_dir, description: data dir
  + data_filename=2CEX-10k-2.h5
    source: data.name, description: data file name
  + devices_per_node=1
    source: gpu.devices_per_node, description: gpu-device count per node/resource set
  + n_images_per_rank=1000
    source: runtime.n_images_per_rank, description: no. of images per rank
  + out_dir=/gpfs/alpine/scratch/cahrens/chm137/spinifel_data/spinifel_output_2021-07-16T1316-0400_TRIAL_080/nodes_1
    source: data.out_dir, description: output dir
  + ps_eb_nodes=1
    source: psana.ps_eb_nodes, description: no. of eventbuilder cores
  + ps_smd_n_events=10000
    source: psana.ps_smd_n_events, description: no. of events to be sent to an EventBuilder core
  + ps_srv_nodes=0
    source: psana.ps_srv_nodes, description: no. of server cores
  + small_problem=False
    source: runtime.small_problem, description: run in small problem mode
  + test=Quickstart settings for Summit
    source: debug.test, description: test field used for debugging
  + use_callmonitor=False
    source: debug.use_callmonitor, description: enable call-monitor
  + use_cufinufft=True
    source: runtime.use_cufinufft, description: use cufinufft for nufft support
  + use_psana=False
    source: psana.enable, description: enable PSANA
  + use_single_prec=False
    source: runtime.use_single_prec, description: if true, spinifel will use single-precision floating point
  + using_cuda=True
    source: runtime.using_cuda, description: use cuda wherever possible
  + verbose=True
    source: debug.verbose, description: is verbosity > 0
  + verbosity=1
    source: debug.verbosity, description: reporting verbosity

Runtime MODE = mpi
MPI has been initialized on rank 4
Psana functionality is not available.
Rank 4 assigned to device 0
Orientation Matching: USING_CUDA
++++++++++++++++++++: USING_CUFINUFFT
pyCudaKNearestNeighbors is available: True
get_slices rank=4 st=4000 en=5000
N_images = 1000
N = 16384000
Ntot = 98304000
N_pixels_per_image = 16384.0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
info = 0
summary = None
Rank 4 got AC in 44 iterations.
WARNING! The environment variable VERBOSE supersedes all other inputs for this setting. If this is unintensional unset VERBOSE.
WARNING! The environment variable DATA_DIR supersedes all other inputs for this setting. If this is unintensional unset DATA_DIR.
WARNING! The environment variable DATA_FILENAME supersedes all other inputs for this setting. If this is unintensional unset DATA_FILENAME.
WARNING! The environment variable OUT_DIR supersedes all other inputs for this setting. If this is unintensional unset OUT_DIR.
WARNING! The environment variable DEVICES_PER_RS supersedes all other inputs for this setting. If this is unintensional unset DEVICES_PER_RS.
SpinifelSettings:
  + chk_convergence=False
    source: runtime.chk_convergence, description: if false, no check if output density converges
  + data_dir=/gpfs/alpine/chm137/proj-shared/data/spi
    source: data.in_dir, description: data dir
  + data_filename=2CEX-10k-2.h5
    source: data.name, description: data file name
  + devices_per_node=1
    source: gpu.devices_per_node, description: gpu-device count per node/resource set
  + n_images_per_rank=1000
    source: runtime.n_images_per_rank, description: no. of images per rank
  + out_dir=/gpfs/alpine/scratch/cahrens/chm137/spinifel_data/spinifel_output_2021-07-16T1316-0400_TRIAL_080/nodes_1
    source: data.out_dir, description: output dir
  + ps_eb_nodes=1
    source: psana.ps_eb_nodes, description: no. of eventbuilder cores
  + ps_smd_n_events=10000
    source: psana.ps_smd_n_events, description: no. of events to be sent to an EventBuilder core
  + ps_srv_nodes=0
    source: psana.ps_srv_nodes, description: no. of server cores
  + small_problem=False
    source: runtime.small_problem, description: run in small problem mode
  + test=Quickstart settings for Summit
    source: debug.test, description: test field used for debugging
  + use_callmonitor=False
    source: debug.use_callmonitor, description: enable call-monitor
  + use_cufinufft=True
    source: runtime.use_cufinufft, description: use cufinufft for nufft support
  + use_psana=False
    source: psana.enable, description: enable PSANA
  + use_single_prec=False
    source: runtime.use_single_prec, description: if true, spinifel will use single-precision floating point
  + using_cuda=True
    source: runtime.using_cuda, description: use cuda wherever possible
  + verbose=True
    source: debug.verbose, description: is verbosity > 0
  + verbosity=1
    source: debug.verbosity, description: reporting verbosity

Runtime MODE = mpi
MPI has been initialized on rank 1
Psana functionality is not available.
Rank 1 assigned to device 0
Orientation Matching: USING_CUDA
++++++++++++++++++++: USING_CUFINUFFT
pyCudaKNearestNeighbors is available: True
get_slices rank=1 st=1000 en=2000
N_images = 1000
N = 16384000
Ntot = 98304000
N_pixels_per_image = 16384.0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
info = 0
summary = None
Rank 1 got AC in 44 iterations.
N_images = 1000
N = 16384000
Ntot = 98304000
N_pixels_per_image = 16384.0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
Using v2 CUDA to solve the NUFFT 3D T1 on device 0
info = 0
summary = [(0, 1.0172526041666667e-11, 0, 4898882474945.297, array([44.17337768+0.j])), (1, 1.0172526041666667e-10, 0, 4898882364570.963, array([44.17337768+0.j])), (2, 1.0172526041666667e-09, 0, 4898881455476.693, array([44.17337768+0.j])), (3, 1.0172526041666666e-08, 0, 4898872576096.936, array([44.17337768+0.j])), (4, 1.0172526041666666e-07, 0, 4898787786364.569, array([44.17337773+0.j])), (5, 1.0172526041666667e-06, 0, 4897929127213.685, array([44.17338361+0.j]))]
ranks = [0 1 2 3 4 5]
lambdas = [1.0172526e-11 1.0172526e-10 1.0172526e-09 1.0172526e-08 1.0172526e-07
 1.0172526e-06]
solns = [4.89888247e+12 4.89888236e+12 4.89888146e+12 4.89887258e+12
 4.89878779e+12 4.89792913e+12]
resids = [44.17337768+0.j 44.17337768+0.j 44.17337768+0.j 44.17337768+0.j
 44.17337773+0.j 44.17338361+0.j]
iref = 4
Rank 0 got AC in 44 iterations.
Keeping result from rank 4.
AC recovered in 61.69s.
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
New Data/Model std ratio: 7.854830860666866e-08.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Heap sort function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of euDist: 1
[CUDA] Size of euDist: 2000000
[CUDA] Dimension of kIndex: 1
[CUDA] Size of kIndex: 1000
Problem phased in 188.48s.
###########################
##### Generation 1/10 #####
###########################
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
New Data/Model std ratio: 8.078870275703558e-08.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Heap sort function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of euDist: 1
[CUDA] Size of euDist: 2000000
[CUDA] Dimension of kIndex: 1
[CUDA] Size of kIndex: 1000
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
New Data/Model std ratio: 8.48849269360449e-08.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
New Data/Model std ratio: 7.93916056868541e-08.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Heap sort function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of euDist: 1
[CUDA] Size of euDist: 2000000
[CUDA] Dimension of kIndex: 1
[CUDA] Size of kIndex: 1000
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Heap sort function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of euDist: 1
[CUDA] Size of euDist: 2000000
[CUDA] Dimension of kIndex: 1
[CUDA] Size of kIndex: 1000
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
New Data/Model std ratio: 8.332022453819056e-08.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
Using v2 CUDA to solve the NUFFT 3D T2 on device 0
New Data/Model std ratio: 7.244406723084928e-08.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Heap sort function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of euDist: 1
[CUDA] Size of euDist: 2000000
[CUDA] Dimension of kIndex: 1
[CUDA] Size of kIndex: 1000
Using CUDA  to calculate Euclidean distance and heap sort (batch_size=100)
Rank 0 using deviceId 0
Match tot:5.76s. slice=4.25s. match=0.84s. slice_oh=0.47s. match_oh=0.20s.
Orientations matched in 5.76s.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Euclidean distance function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of dataImgs: 1 and refImgs: 1
[CUDA] Size of dataImgs: 16384000 and refImgs: 1638400
[CUDA] Copy data from host to device.
[CUDA] Copy data from device to host.
[CUDA] In CUDA Heap sort function.
[CUDA] Using deviceId: 0
[CUDA] Dimension of euDist: 1
[CUDA] Size of euDist: 2000000
[CUDA] Dimension of kIndex: 1
[CUDA] Size of kIndex: 1000

------------------------------------------------------------
Sender: LSF System <lsfadmin@batch5>
Subject: Job 1162887: <RunSpinifel_080_1> in cluster <summit> Exited

Job <RunSpinifel_080_1> was submitted from host <login3> by user <cahrens> in cluster <summit> at Fri Jul 16 13:16:18 2021
Job was executed on host(s) <1*batch5>, in queue <batch>, as user <cahrens> in cluster <summit> at Fri Jul 16 13:16:55 2021
                            <42*h35n11>
</ccs/home/cahrens> was used as the home directory.
</gpfs/alpine/chm137/scratch/cahrens/devel/spinifel-devel-070821> was used as the working directory.
Started at Fri Jul 16 13:16:55 2021
Terminated at Fri Jul 16 15:17:19 2021
Results reported at Fri Jul 16 15:17:19 2021

The output (if any) is above this job summary.



PS:

Read file </gpfs/alpine/scratch/cahrens/chm137/spinifel_data/spinifel_output_2021-07-16T1316-0400_TRIAL_080/error.1162887.log_nodes_1> for stderr output of this job.