[loadgen1:13239][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13236][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13234][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13241][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13237][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13235][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13233][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13239] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13234] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13236] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13241] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13233] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13237] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13235] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17751][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17749][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19913][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18034][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19917][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18032][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18257][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19876][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19914][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18036][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18264][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19883][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18084][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20174][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3614][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19915][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18035][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18261][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19877][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18086][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20172][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3610][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28660][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2670][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17750][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5918][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19916][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18038][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18259][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19879][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18082][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20170][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3612][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28658][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2668][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17746][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5920][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18028][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17751][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19912][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18033][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18263][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19881][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18085][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20176][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3611][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28664][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2672][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17747][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5922][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18026][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17749][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19919][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18031][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18258][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19878][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18081][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20169][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3609][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28657][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2667][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17748][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5919][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18027][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17757][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19914] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18032] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18260][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19880][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18088][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20171][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3613][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28661][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2671][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17754][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5921][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18025][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17753][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19917] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18036] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18257] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19876] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18083][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20173][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3616][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28659][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2674][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17751] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:5917][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18032][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17750][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19915] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18034] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18264] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19881] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18084] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20172] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03612] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28662][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2669][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17749] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:5924][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18030][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17752][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19913] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18035] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18261] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19877] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18086] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20174] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03610] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28658] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02670] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17746] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05920] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18029][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17754][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19916] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18038] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18259] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19879] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18082] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20170] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03614] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28660] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02668] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17750] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05918] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18026] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17751] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19912] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18033] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18263] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19883] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18081] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20169] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03609] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28664] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02672] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17754] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05922] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18028] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17753] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19919] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18031] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18258] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19878] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18088] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20171] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03611] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28661] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02667] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17747] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05919] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18025] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17750] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18260] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19880] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18085] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20176] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03616] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28659] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02674] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17748] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05917] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18030] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17752] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18083] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20173] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03613] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28662] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02671] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05921] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18027] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17754] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28657] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02669] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05924] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18032] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17749] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18029] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17757] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed -- started at 05/30/2019 08:48:28 -- mdtest-1.9.3 was launched with 98 total task(s) on 14 node(s) Command line used: /mnt/cephfs/io-500-dev/bin/mdtest "-C" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "/mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/mdt_hard" "-n" "51614" "-x" "/mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/mdt_hard-stonewall" "-W" "300" Path: /mnt/cephfs/datafiles/io500.2019.05.30-08.24.51 FS: 93.3 TiB Used FS: 14.1% Inodes: 8.9 Mi Used Inodes: 100.0% 98 tasks, 5058172 files Continue stonewall hit min: 28349 max: 30667 avg: 29518.2 SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 9612.049 9611.348 9611.776 0.167 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 2699.143 2699.143 2699.143 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/30/2019 08:53:40 --