[loadgen1:13596][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13598][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13594][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13595][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13599][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13593][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13597][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13598] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13594] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13596] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13595] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13599] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13593] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13597] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18656][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18658][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18654][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18655][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:18180][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20332][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20312][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18657][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:18182][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20336][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20314][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18500][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18450][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:29090][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:3108][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18653][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:18178][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20334][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20311][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18498][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18454][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:29092][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:3106][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18412][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:4013][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6346][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20592][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:18167][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18659][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:18181][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20330][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20315][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18495][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18452][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:29094][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:3110][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18414][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:4015][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6348][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20594][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:18165][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18658] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18177][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20333][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20313][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18497][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18451][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:29089][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:3111][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18410][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:4011][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6344][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20596][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:18163][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18656] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18179][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20331][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20309][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18499][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18455][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:29091][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:3109][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18415][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:4016][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6349][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20593][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:18168][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18654] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18183][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:20335][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:20310][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:18494][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18453][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:29095][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:3107][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18411][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:4012][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6347][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20595][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:18164][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18655] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18180] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20334] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20312] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18496][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:18449][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:29093][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:3112][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:18409][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:4010][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6343][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20597][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:18166][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18659] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18182] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20332] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20314] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18495] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18454] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:29091] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:03110] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18413][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:4014][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:6345][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20591][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:18162][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18653] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18178] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20330] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20313] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18497] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18450] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:29095] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:03106] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18412] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:04015] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06346] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20594] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:18163] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18657] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18179] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20336] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20311] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18499] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18452] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:29089] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:03108] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18411] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:04013] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06348] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20596] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:18167] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18183] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20335] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20315] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18496] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18451] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:29093] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:03111] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18409] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:04011] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06347] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20592] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:18165] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18181] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20331] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20309] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18494] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18453] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:29092] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:03107] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18414] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:04012] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06349] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20593] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:18164] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:18177] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:20333] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:20310] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18500] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18455] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:29090] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:03109] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18413] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:04014] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06343] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20595] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:18166] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:18498] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:18449] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:29094] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:03112] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18415] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:04010] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06344] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20597] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:18162] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:18410] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:04016] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:06345] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20591] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:18168] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed -- started at 05/30/2019 09:16:45 -- mdtest-1.9.3 was launched with 98 total task(s) on 14 node(s) Command line used: /mnt/cephfs/io-500-dev/bin/mdtest "-T" "-F" "-d" "/mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/mdt_easy" "-n" "60000" "-u" "-L" "-x" "/mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/mdt_easy-stonewall" Path: /mnt/cephfs/datafiles/io500.2019.05.30-08.24.51 FS: 93.2 TiB Used FS: 14.2% Inodes: 11.8 Mi Used Inodes: 100.0% 98 tasks, 5880000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 37465.241 37462.495 37464.192 0.649 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/30/2019 09:19:22 --