[loadgen1:13135][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13129][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13127][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13131][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13130][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13132][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13128][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen1:13135] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13131] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13129] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13130] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13132] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13128] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen1:13127] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19745][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17617][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19753][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17619][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19781][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28540][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3478][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18134][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:17905][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19749][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17624][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19788][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28536][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3480][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18129][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:17901][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2537][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:17951][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:17884][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20043][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19751][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17618][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19785][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28538][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3482][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18127][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:17903][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2539][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:17953][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:17886][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20041][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17624][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5789][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19748][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17622][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19786][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28539][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3481][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18130][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:17907][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2541][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:17950][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:17888][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20039][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17622][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5791][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19746][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17620][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19782][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28537][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3484][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18128][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:17902][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2538][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:17957][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:17887][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20038][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17620][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5786][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19747][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen8:17621][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen2:19784][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28535][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3477][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18131][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:17904][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2540][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:17954][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:17890][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20045][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17623][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5793][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19746] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17624] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19783][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen12:28542][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen13:3479][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen7:18132][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen10:17900][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen16:2543][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:17952][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:17883][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20040][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen11:17619][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5788][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19751] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17617] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19785] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28540] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03482] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18134] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:17905] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:2536][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen9:17955][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen6:17885][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen4:20042][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator.[loadgen11:17621][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5790][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19748] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17620] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19781] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28536] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03480] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18129] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:17901] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02537] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:17951] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:17886] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17626][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen14:5787][coll_ml_mca.c:239:set_hcoll_device] COLL-ML You must specify a valid HCA device by setting: -x HCOLL_MAIN_IB= or -x MXM_RDMA_PORTS=. If no device was specified for HCOLL (or the calling library), automatic device detection will be run. In case of unfounded HCA device please contact your system administrator. [loadgen5:19745] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17618] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19786] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28538] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03478] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18127] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:17903] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02541] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:17953] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:17888] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20043] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17624] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05790] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19753] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17622] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19788] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28539] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03484] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18130] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:17902] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02539] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:17950] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:17884] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20038] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17622] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05786] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19747] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17619] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19783] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28537] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03481] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18128] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:17907] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02540] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:17954] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:17887] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20039] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17621] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05791] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen5:19749] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen8:17621] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19784] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28542] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03477] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18132] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:17904] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02538] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:17952] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:17890] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20045] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17623] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05789] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen2:19782] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen12:28535] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen13:03479] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen7:18131] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen10:17900] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02543] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:17955] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:17885] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20040] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17619] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05788] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen16:02536] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen9:17957] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen6:17883] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20041] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17620] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05787] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen4:20042] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen11:17626] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed [loadgen14:05793] Error: coll_hcoll_module.c:301 - mca_coll_hcoll_comm_query() Hcol library init failed IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 30 08:42:03 2019 Command line : /mnt/cephfs/io-500-dev/bin/ior -w -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -s 185192 -o /mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/ior_hard/IOR_file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300 Machine : Linux loadgen1 TestID : 0 StartTime : Thu May 30 08:42:03 2019 Path : /mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/ior_hard FS : 93.6 TiB Used FS: 13.3% Inodes: 8.7 Mi Used Inodes: 100.0% Options: api : POSIX apiVersion : test filename : /mnt/cephfs/datafiles/io500.2019.05.30-08.24.51/ior_hard/IOR_file access : single-shared-file type : independent segments : 185192 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 98 clients per node : 7 repetitions : 1 xfersize : 47008 bytes blocksize : 47008 bytes aggregate filesize : 794.55 GiB stonewallingTime : 300 stoneWallingWearOut : 1 Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- stonewalling pairs accessed min: 132935 max: 182350 -- min data: 5.8 GiB mean data: 7.0 GiB time: 300.1s WARNING: Expected aggregate file size = 853139542528. WARNING: Stat() of aggregate file size = 840047062400. WARNING: Using actual aggregate bytes moved = 840047062400. WARNING: maybe caused by deadlineForStonewalling write 2088.99 45.91 45.91 0.002681 383.48 0.028331 383.50 0 Max Write: 2088.99 MiB/sec (2190.46 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum write 2088.99 2088.99 2088.99 0.00 46597.69 46597.69 46597.69 0.00 383.50186 0 98 7 1 0 1 1 0 0 185192 47008 47008 801131.3 POSIX 0 Finished : Thu May 30 08:48:27 2019