[node1113.palmetto.clemson.edu:24457] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0046.palmetto.clemson.edu:98655] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1821.palmetto.clemson.edu:20066] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1832.palmetto.clemson.edu:42279] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0145.palmetto.clemson.edu:59974] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1189.palmetto.clemson.edu:24807] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1875.palmetto.clemson.edu:119270] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0121.palmetto.clemson.edu:13396] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1868.palmetto.clemson.edu:35279] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1831.palmetto.clemson.edu:75085] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0139.palmetto.clemson.edu:31783] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1822.palmetto.clemson.edu:52276] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1813.palmetto.clemson.edu:106941] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1880.palmetto.clemson.edu:00896] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0175.palmetto.clemson.edu:68249] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1881.palmetto.clemson.edu:65754] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1806.palmetto.clemson.edu:14714] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1872.palmetto.clemson.edu:62318] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1878.palmetto.clemson.edu:22258] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1811.palmetto.clemson.edu:14414] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1823.palmetto.clemson.edu:01914] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1876.palmetto.clemson.edu:28155] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1882.palmetto.clemson.edu:63594] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1815.palmetto.clemson.edu:67419] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1812.palmetto.clemson.edu:59673] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1810.palmetto.clemson.edu:08376] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1883.palmetto.clemson.edu:04636] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1826.palmetto.clemson.edu:29852] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1869.palmetto.clemson.edu:14836] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1827.palmetto.clemson.edu:47535] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1819.palmetto.clemson.edu:87731] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1873.palmetto.clemson.edu:13348] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1871.palmetto.clemson.edu:69734] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1825.palmetto.clemson.edu:107067] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0200.palmetto.clemson.edu:46162] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1874.palmetto.clemson.edu:101661] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1870.palmetto.clemson.edu:100241] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1824.palmetto.clemson.edu:57607] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1814.palmetto.clemson.edu:79977] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1805.palmetto.clemson.edu:83007] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1830.palmetto.clemson.edu:33518] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1816.palmetto.clemson.edu:26280] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1820.palmetto.clemson.edu:90659] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node0653.palmetto.clemson.edu:105867] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1829.palmetto.clemson.edu:96046] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1877.palmetto.clemson.edu:118018] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1818.palmetto.clemson.edu:121322] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
[node1808.palmetto.clemson.edu:22945] mca: base: component_find: unable to open /software/openmpi/1.10.7_gcc/lib/openmpi/mca_btl_openib: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored)
IOR-3.2.0: MPI Coordinated Test of Parallel I/O
Began               : Thu Oct 18 17:48:26 2018
Command line        : /home/jburto2/io-500-dev/bin/ior -r -R -C -Q 1 -g -G 27 -k -e -t 2m -b 43690m -F -a MPIIO -o /scratch1/jburto2/io500/datafiles/io500.2018.10.18-17.05.34/ior_easy/ior_file_easy -O stoneWallingStatusFile=/scratch1/jburto2/io500/datafiles/io500.2018.10.18-17.05.34/ior_easy/stonewall
Machine             : Linux node0046.palmetto.clemson.edu
TestID              : 0
StartTime           : Thu Oct 18 17:48:26 2018
Path                : /scratch1/jburto2/io500/datafiles/io500.2018.10.18-17.05.34/ior_easy
FS                  : 232.8 TiB   Used FS: 16.9%   Inodes: 4398046511104.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.0)
test filename       : /scratch1/jburto2/io500/datafiles/io500.2018.10.18-17.05.34/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 48
clients per node    : 1
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 42.67 GiB
aggregate filesize  : 2.00 TiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ---------- ---------  --------   --------   --------   --------   ----
[E 17:53:27.670631] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 144.
[E 17:53:27.670714] fp_multiqueue_cancel: flow proto cancel called on 0x18f3f40
[E 17:53:27.670722] fp_multiqueue_cancel: I/O error occurred
[E 17:53:27.670730] handle_io_error: flow proto error cleanup started on 0x18f3f40: Operation cancelled (possibly due to timeout)
[E 17:53:27.670795] handle_io_error: flow proto 0x18f3f40 canceled 1 operations, will clean up.
[E 17:53:27.670808] bmi_to_mem_callback_fn: I/O error occurred
[E 17:53:27.670813] handle_io_error: flow proto 0x18f3f40 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 17:53:27.670823] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 17:53:27.996751] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 145.
[E 17:53:27.996847] fp_multiqueue_cancel: flow proto cancel called on 0x194acc0
[E 17:53:27.996855] fp_multiqueue_cancel: I/O error occurred
[E 17:53:27.996863] handle_io_error: flow proto error cleanup started on 0x194acc0: Operation cancelled (possibly due to timeout)
[E 17:53:27.996932] handle_io_error: flow proto 0x194acc0 canceled 1 operations, will clean up.
[E 17:53:27.996944] bmi_to_mem_callback_fn: I/O error occurred
[E 17:53:27.996949] handle_io_error: flow proto 0x194acc0 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 17:53:27.996960] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 17:53:59.727644] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 17966.
[E 17:53:59.727748] fp_multiqueue_cancel: flow proto cancel called on 0x1a71570
[E 17:53:59.727757] fp_multiqueue_cancel: I/O error occurred
[E 17:53:59.727766] handle_io_error: flow proto error cleanup started on 0x1a71570: Operation cancelled (possibly due to timeout)
[E 17:53:59.727833] handle_io_error: flow proto 0x1a71570 canceled 1 operations, will clean up.
[E 17:53:59.727845] bmi_to_mem_callback_fn: I/O error occurred
[E 17:53:59.727850] handle_io_error: flow proto 0x1a71570 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 17:53:59.727862] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 17:56:39.999579] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 56043.
[E 17:56:39.999680] fp_multiqueue_cancel: flow proto cancel called on 0x18ca918
[E 17:56:39.999689] fp_multiqueue_cancel: I/O error occurred
[E 17:56:39.999697] handle_io_error: flow proto error cleanup started on 0x18ca918: Operation cancelled (possibly due to timeout)
[E 17:56:39.999768] handle_io_error: flow proto 0x18ca918 canceled 1 operations, will clean up.
[E 17:56:39.999781] bmi_to_mem_callback_fn: I/O error occurred
[E 17:56:39.999786] handle_io_error: flow proto 0x18ca918 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 17:56:39.999796] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 17:59:55.721728] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 202548.
[E 17:59:55.721844] fp_multiqueue_cancel: flow proto cancel called on 0x2e1e940
[E 17:59:55.721854] fp_multiqueue_cancel: I/O error occurred
[E 17:59:55.721882] handle_io_error: flow proto error cleanup started on 0x2e1e940: Operation cancelled (possibly due to timeout)
[E 17:59:55.721944] handle_io_error: flow proto 0x2e1e940 canceled 1 operations, will clean up.
[E 17:59:55.721956] bmi_to_mem_callback_fn: I/O error occurred
[E 17:59:55.721960] handle_io_error: flow proto 0x2e1e940 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 17:59:55.721971] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 18:00:53.963549] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 77406.
[E 18:00:53.963661] fp_multiqueue_cancel: flow proto cancel called on 0x28042e0
[E 18:00:53.963670] fp_multiqueue_cancel: I/O error occurred
[E 18:00:53.963678] handle_io_error: flow proto error cleanup started on 0x28042e0: Operation cancelled (possibly due to timeout)
[E 18:00:53.963745] handle_io_error: flow proto 0x28042e0 canceled 1 operations, will clean up.
[E 18:00:53.963778] bmi_to_mem_callback_fn: I/O error occurred
[E 18:00:53.963784] handle_io_error: flow proto 0x28042e0 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 18:00:53.963796] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 18:01:13.929597] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 94803.
[E 18:01:13.929689] fp_multiqueue_cancel: flow proto cancel called on 0x16bd7e0
[E 18:01:13.929697] fp_multiqueue_cancel: I/O error occurred
[E 18:01:13.929705] handle_io_error: flow proto error cleanup started on 0x16bd7e0: Operation cancelled (possibly due to timeout)
[E 18:01:13.929780] handle_io_error: flow proto 0x16bd7e0 canceled 1 operations, will clean up.
[E 18:01:13.929792] bmi_to_mem_callback_fn: I/O error occurred
[E 18:01:13.929796] handle_io_error: flow proto 0x16bd7e0 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 18:01:13.929807] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 18:01:44.176982] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 142412.
[E 18:01:44.177074] fp_multiqueue_cancel: flow proto cancel called on 0x20c5a50
[E 18:01:44.177082] fp_multiqueue_cancel: I/O error occurred
[E 18:01:44.177089] handle_io_error: flow proto error cleanup started on 0x20c5a50: Operation cancelled (possibly due to timeout)
[E 18:01:44.177154] handle_io_error: flow proto 0x20c5a50 canceled 1 operations, will clean up.
[E 18:01:44.177166] bmi_to_mem_callback_fn: I/O error occurred
[E 18:01:44.177170] handle_io_error: flow proto 0x20c5a50 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 18:01:44.177180] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 18:02:41.642302] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 145814.
[E 18:02:41.642406] fp_multiqueue_cancel: flow proto cancel called on 0x2c6e860
[E 18:02:41.642415] fp_multiqueue_cancel: I/O error occurred
[E 18:02:41.642422] handle_io_error: flow proto error cleanup started on 0x2c6e860: Operation cancelled (possibly due to timeout)
[E 18:02:41.642489] handle_io_error: flow proto 0x2c6e860 canceled 1 operations, will clean up.
[E 18:02:41.642501] bmi_to_mem_callback_fn: I/O error occurred
[E 18:02:41.642506] handle_io_error: flow proto 0x2c6e860 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 18:02:41.642516] io_datafile_complete_operations: flow failed, retrying from msgpair
[E 18:05:57.886017] job_time_mgr_expire: job time out: cancelling flow operation, job_id: 2414.
[E 18:05:57.886109] fp_multiqueue_cancel: flow proto cancel called on 0x2931dc0
[E 18:05:57.886117] fp_multiqueue_cancel: I/O error occurred
[E 18:05:57.886125] handle_io_error: flow proto error cleanup started on 0x2931dc0: Operation cancelled (possibly due to timeout)
[E 18:05:57.886189] handle_io_error: flow proto 0x2931dc0 canceled 1 operations, will clean up.
[E 18:05:57.886202] bmi_to_mem_callback_fn: I/O error occurred
[E 18:05:57.886206] handle_io_error: flow proto 0x2931dc0 error cleanup finished: Operation cancelled (possibly due to timeout)
[E 18:05:57.886217] io_datafile_complete_operations: flow failed, retrying from msgpair
read      1878.75    44738560   2048.00    1.26       1114.97    0.003219   1116.23    0   
Max Read:  1878.75 MiB/sec (1970.01 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         1878.75    1878.75    1878.75       0.00     939.37     939.37     939.37       0.00 1116.23212     0     48   1    1   1     1        1         0    0      1 45812285440  2097152 2097120.0 MPIIO      0
Finished            : Thu Oct 18 18:07:03 2018