7fffea830000-7fffea870000 rw-p 00000000 00:00 0 [stack]
Size: 256 kB
KernelPageSize: 64 kB
MMUPageSize: 64 kB
Rss: 256 kB
Pss: 256 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 256 kB
Referenced: 256 kB
Anonymous: 256 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
VmFlags: rd wr mr mw me gd ac
VmFlags: rd wr mr mw me gd ac
MemTotal: 630978176 kB
MemFree: 552515008 kB
MemAvailable: 582776320 kB
Buffers: 640 kB
Cached: 43211136 kB
SwapCached: 0 kB
Active: 13815872 kB
Inactive: 33122304 kB
Active(anon): 13233280 kB
Inactive(anon): 1677760 kB
Active(file): 582592 kB
Inactive(file): 31444544 kB
Unevictable: 16777216 kB
Mlocked: 16777216 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 2624 kB
Writeback: 0 kB
AnonPages: 20521024 kB
Mapped: 1839424 kB
Shmem: 11180736 kB
KReclaimable: 514368 kB
Slab: 5428544 kB
SReclaimable: 514368 kB
SUnreclaim: 4914176 kB
KernelStack: 37056 kB
PageTables: 22912 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 315489088 kB
Committed_AS: 74855168 kB
VmallocTotal: 549755813888 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 596992 kB
HardwareCorrupted: 128 kB
AnonHugePages: 393216 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 26853376 kB
CmaFree: 26853376 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
Hugetlb: 0 kB
ibv_reg_mr returns 13, addr=0x160d0000, bytes=262144
python: /__SMPI_build_dir__________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/ibvdevice/IBVMRManager.h:230: pami_result_t PAMI::Device::IBV::IBVMRManager::create_mr(void*, size_t, PAMI::Device::IBV::IBVMRElem*): Assertion `mr[n] != __null' failed.
[g05n05:1401055] *** Process received signal ***
[g05n05:1401055] Signal: Aborted (6)
[g05n05:1401055] Signal code: (-6)
[g05n05:1401055] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000504d8]
[g05n05:1401055] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x200000353618]
[g05n05:1401055] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200000333a2c]
[g05n05:1401055] [ 3] /lib64/power9/libc.so.6(+0x36f70)[0x200000346f70]
[g05n05:1401055] [ 4] /lib64/power9/libc.so.6(__assert_fail+0x64)[0x200000347014]
[g05n05:1401055] [ 5] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libpami.so.3(_ZN4PAMI6Device3IBV12IBVMRManager9create_mrEPvmPNS1_9IBVMRElemE+0x2b4)[0x200061b08554]
[g05n05:1401055] [ 6] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libpami.so.3(_ZN4PAMI14MemregionCache15createMemregionEPNS_9MemregionEPmmPvm+0x930)[0x200061b31af0]
[g05n05:1401055] [ 7] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libpami.so.3(_ZN4PAMI7Context21memregion_create_implEPvmPmPA24_m+0x17c)[0x200061b323cc]
[g05n05:1401055] [ 8] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libpami.so.3(PAMI_Memregion_create+0x18)[0x200061adc1f8]
[g05n05:1401055] [ 9] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libpami.so.3(_ZN4PAMI8Protocol4Send9EagerRgetINS_6Device5Shmem11PacketModelINS3_11ShmemDeviceINS_4Fifo8WrapFifoINS7_10FifoPacketILj64ELj4096EEENS_7Counter15IndirectBoundedINS_6Atomic12NativeAtomicEEELj256EEENSB_8IndirectINSB_6NativeEEENS4_9CMAShaddrELj256ELj512EEEEENS4_8DmaModelISL_Lb0EEEE6simpleEP11pami_send_t+0x158)[0x200061b99858]
[g05n05:1401055] [10] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libpami.so.3(_ZN4PAMI8Protocol4Send9CompositeINS1_4SendES3_E6simpleEP11pami_send_t+0x40)[0x200061aec500]
[g05n05:1401055] [11] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libpami.so.3(PAMI_Send+0x58)[0x200061ad4a38]
[g05n05:1401055] [12] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libcollectives.so.3(_ZN7LibColl18NativeInterfaceP2PILb1ELb0EE19multicast_over_sendEPNS1_20p2p_multicast_send_tEm+0x25c)[0x2000823cde7c]
[g05n05:1401055] [13] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libcollectives.so.3(_ZN7LibColl18NativeInterfaceP2PILb1ELb0EE9multicastEPNS_14ni_multicast_tE+0x36c)[0x2000823ce3dc]
[g05n05:1401055] [14] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libcollectives.so.3(_ZN4CCMI8Executor9BroadcastINS_17ConnectionManager14CommSeqConnMgrE20_gather_cheader_dataE5startEv+0x78)[0x200082308798]
[g05n05:1401055] [15] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libcollectives.so.3(_ZN4CCMI9Protocols9Broadcast22AsyncBroadcastFactoryTINS1_15AsyncBroadcastTINS_17ConnectionManager14CommSeqConnMgrENS_8Executor9BroadcastIS5_20_gather_cheader_dataEELN7LibColl15topologyIndex_tE0ELj256EEEE8generateEPvSE_+0x53c)[0x20008235df3c]
[g05n05:1401055] [16] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libcollectives.so.3(_ZN7LibColl7Adapter20StaticCollSelAdviser10autoSelectE19libcoll_xfer_type_tP14libcoll_xfer_tb+0x1dc)[0x2000822de6fc]
[g05n05:1401055] [17] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/pami_port/libcollectives.so.3(LIBCOLL_AutoSelect+0x58)[0x2000822d9fe8]
[g05n05:1401055] [18] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/spectrum_mpi/mca_coll_ibm.so(start_libcoll_blocking_collective+0x478)[0x2000720001a8]
[g05n05:1401055] [19] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/container/../lib/spectrum_mpi/mca_coll_ibm.so(mca_coll_ibm_bcast+0x208)[0x2000720056c8]
[g05n05:1401055] [20] /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-8.3.1/spectrum-mpi-10.4.0.3-20210112-i64lccn3bi6ibfavxamdhjoaqvdvbo5b/lib/libmpi_ibm.so.3(MPI_Bcast+0x198)[0x200000651158]
[g05n05:1401055] [21] /autofs/nccs-svm1_home1/chuck/spinifel/setup/conda/envs/myenv/lib/python3.8/site-packages/mpi4py/MPI.cpython-38-powerpc64le-linux-gnu.so(+0x110eec)[0x200030ef0eec]
[g05n05:1401055] [22] python[0x10043a14]
[g05n05:1401055] [23] python(_PyObject_MakeTpCall+0xec)[0x100444cc]
[g05n05:1401055] [24] python(_PyEval_EvalFrameDefault+0x87b0)[0x1002cc40]
[g05n05:1401055] [25] python(PyEval_EvalFrameEx+0x34)[0x1012a9f4]
[g05n05:1401055] [26] python[0x10022cd4]
[g05n05:1401055] [27] python(PyVectorcall_Call+0xa0)[0x10046be0]
[g05n05:1401055] [28] python(PyObject_Call+0x1ec)[0x10046fbc]
[g05n05:1401055] [29] python(_PyEval_EvalFrameDefault+0x3314)[0x100277a4]
[g05n05:1401055] *** End of error message ***
ERROR: One or more process (first noticed rank 0) terminated with signal 6
PSJobCompleted TotalElapsed 22 1674153562 1674153584