[loadgen1:13441][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13439][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13443][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13447][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13440][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13445][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13442][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13441] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13439] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13447] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13443] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13445] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13442] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13440] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20143][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18261][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18259][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20141][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18257][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20144][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17991][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20120][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18258][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20148][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17986][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20118][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18471][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18317][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28923][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18260][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20145][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17988][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20122][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18477][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18315][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28925][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2918][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3829][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20398][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18226][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18256][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20140][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17984][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20121][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18472][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18319][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28927][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2925][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3833][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20399][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18222][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17973][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6151][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18263][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20142][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17989][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20124][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18473][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18318][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28924][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2922][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3831][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20403][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18224][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17971][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6153][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18259] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20143] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17987][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20117][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18475][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18321][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28929][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2923][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3828][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20401][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18223][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17969][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6149][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18261] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20141] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17985][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20119][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18474][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18314][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28922][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2921][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3830][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20400][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18225][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17975][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6155][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18257] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20144] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17986] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20122] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18470][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18316][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28926][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2919][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3832][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20405][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18221][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17972][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6152][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18258] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20148] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17991] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20118] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18471] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18319] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28923] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:2920][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3835][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20402][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18228][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17968][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6148][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18260] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20145] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17984] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20120] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18477] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18315] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28925] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02918] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03833] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20398] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18224] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17970][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6150][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18256] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20140] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17988] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20121] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18475] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18317] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28927] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02925] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03831] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20403] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18226] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17969] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06153] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18263] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20142] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17989] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20124] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18473] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18318] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28929] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02922] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03829] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20401] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18222] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17973] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06151] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17987] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20117] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18472] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18314] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28922] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02919] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03828] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20399] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18223] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17971] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06149] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17985] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20119] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18474] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18321] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28924] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02921] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03830] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20400] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18221] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17975] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06155] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18470] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18316] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28926] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02923] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03835] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20405] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18225] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17972] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06150] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02920] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03832] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20402] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18228] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17970] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06148] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17968] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06152] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 30 08:54:38 2019 Command line : /mnt/cephfs/io-500-dev/bin/ior -r -R -C -Q 1 -g -G 27 -k -e -t 4M -b 12000m -s 4 -F -o /mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/ior_easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/ior_easy/stonewall Machine : Linux loadgen1 TestID : 0 StartTime : Thu May 30 08:54:38 2019 Path : /mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/ior_easy FS : 93.2 TiB Used FS: 14.2% Inodes: 11.8 Mi Used Inodes: 100.0% Options: api : POSIX apiVersion : test filename : /mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/ior_easy/ior_file_easy access : file-per-process type : independent segments : 4 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 98 clients per node : 7 repetitions : 1 xfersize : 4 MiB blocksize : 11.72 GiB aggregate filesize : 4.49 TiB Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- WARNING: Expected aggregate file size = 4932501504000. WARNING: Stat() of aggregate file size = 3130083246080. WARNING: Using actual aggregate bytes moved = 3130083246080. read 2253.82 12288000 4096 6.45 1318.00 0.096765 1324.45 0 Max Read: 2253.82 MiB/sec (2363.30 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum read 2253.82 2253.82 2253.82 0.00 563.45 563.45 563.45 0.00 1324.45365 0 98 7 1 1 1 1 0 0 4 12582912000 4194304 2985080.0 POSIX 0 Finished : Thu May 30 09:16:43 2019