[node1806.palmetto.clemson.edu:13343] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1881.palmetto.clemson.edu:64374] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node0145.palmetto.clemson.edu:58453] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node0121.palmetto.clemson.edu:11950] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1808.palmetto.clemson.edu:21560] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1814.palmetto.clemson.edu:78627] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1832.palmetto.clemson.edu:40726] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node0200.palmetto.clemson.edu:44658] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node0175.palmetto.clemson.edu:66734] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node0046.palmetto.clemson.edu:97238] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1816.palmetto.clemson.edu:24930] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1189.palmetto.clemson.edu:23437] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node0139.palmetto.clemson.edu:30378] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1805.palmetto.clemson.edu:81484] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1113.palmetto.clemson.edu:23107] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1870.palmetto.clemson.edu:98831] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1811.palmetto.clemson.edu:13064] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1813.palmetto.clemson.edu:105457] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1883.palmetto.clemson.edu:03126] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1815.palmetto.clemson.edu:66052] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1812.palmetto.clemson.edu:58258] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1819.palmetto.clemson.edu:86372] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node0653.palmetto.clemson.edu:104429] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1869.palmetto.clemson.edu:13479] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1820.palmetto.clemson.edu:89310] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1882.palmetto.clemson.edu:62145] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1810.palmetto.clemson.edu:06999] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1868.palmetto.clemson.edu:33926] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1818.palmetto.clemson.edu:119973] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1872.palmetto.clemson.edu:60956] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1873.palmetto.clemson.edu:11928] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) [node1871.palmetto.clemson.edu:68314] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu Oct 18 16:31:33 2018 Command line : /home/jburto2/io-500-dev/bin/ior -r -R -C -Q 1 -g -G 27 -k -e -t 2m -b 65536m -F -a MPIIO -o /scratch1/jburto2/io500/datafiles/io500.2018.10.18-15.54.56/ior_easy/ior_file_easy -O stoneWallingStatusFile=/scratch1/jburto2/io500/datafiles/io500.2018.10.18-15.54.56/ior_easy/stonewall Machine : Linux node0046.palmetto.clemson.edu TestID : 0 StartTime : Thu Oct 18 16:31:33 2018 Path : /scratch1/jburto2/io500/datafiles/io500.2018.10.18-15.54.56/ior_easy FS : 232.8 TiB Used FS: 15.2% Inodes: 4398046511104.0 Mi Used Inodes: 0.0% Options: api : MPIIO apiVersion : (3.0) test filename : /scratch1/jburto2/io500/datafiles/io500.2018.10.18-15.54.56/ior_easy/ior_file_easy access : file-per-process type : independent segments : 1 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 32 clients per node : 1 repetitions : 1 xfersize : 2 MiB blocksize : 64 GiB aggregate filesize : 2 TiB Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- [E 16:36:34.481665] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 169. [E 16:36:34.481758] fp_multiqueue_cancel: flow proto cancel called on 0x1677c68 [E 16:36:34.481767] fp_multiqueue_cancel: I/O error occurred [E 16:36:34.481775] handle_io_error: flow proto error cleanup started on 0x1677c68: Operation cancelled (possibly due to timeout) [E 16:36:34.481844] handle_io_error: flow proto 0x1677c68 canceled 1 operations, will clean up. [E 16:36:34.481856] bmi_to_mem_callback_fn: I/O error occurred [E 16:36:34.481861] handle_io_error: flow proto 0x1677c68 error cleanup finished: Operation cancelled (possibly due to timeout) [E 16:36:34.481871] io_datafile_complete_operations: flow failed, retrying from msgpair [E 16:41:09.323216] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 191365. [E 16:41:09.323352] fp_multiqueue_cancel: flow proto cancel called on 0x2cf9828 [E 16:41:09.323361] fp_multiqueue_cancel: I/O error occurred [E 16:41:09.323369] handle_io_error: flow proto error cleanup started on 0x2cf9828: Operation cancelled (possibly due to timeout) [E 16:41:09.323462] handle_io_error: flow proto 0x2cf9828 canceled 1 operations, will clean up. [E 16:41:09.323474] bmi_to_mem_callback_fn: I/O error occurred [E 16:41:09.323479] handle_io_error: flow proto 0x2cf9828 error cleanup finished: Operation cancelled (possibly due to timeout) [E 16:41:09.323490] io_datafile_complete_operations: flow failed, retrying from msgpair [E 16:41:34.698516] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 168. [E 16:41:34.698619] fp_multiqueue_cancel: flow proto cancel called on 0x2ec6ce8 [E 16:41:34.698628] fp_multiqueue_cancel: I/O error occurred [E 16:41:34.698636] handle_io_error: flow proto error cleanup started on 0x2ec6ce8: Operation cancelled (possibly due to timeout) [E 16:41:34.698710] handle_io_error: flow proto 0x2ec6ce8 canceled 1 operations, will clean up. [E 16:41:34.698723] bmi_to_mem_callback_fn: I/O error occurred [E 16:41:34.698727] handle_io_error: flow proto 0x2ec6ce8 error cleanup finished: Operation cancelled (possibly due to timeout) [E 16:41:34.698738] io_datafile_complete_operations: flow failed, retrying from msgpair WARNING: Expected aggregate file size = 2199023255552. WARNING: Stat() of aggregate file size = 2007897210880. WARNING: Using actual aggregate bytes moved = 2007897210880. read 2001.29 67108864 2048.00 0.847465 955.97 0.001980 956.82 0 Max Read: 2001.29 MiB/sec (2098.51 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum read 2001.29 2001.29 2001.29 0.00 1000.65 1000.65 1000.65 0.00 956.82178 0 32 1 1 1 1 1 0 0 1 68719476736 2097152 1914880.0 MPIIO 0 Finished : Thu Oct 18 16:47:31 2018