ior WARNING: fsync() not supported in selected backend.  Using value of 0.
[3181] 2019-11-14 08:17:55 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1a02380 172.30.140.201@verbs-ib C P0-peer00 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3181] 2019-11-14 08:17:55 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1a02380 172.30.140.201@verbs-ib C P0-peer00 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Nov 14 08:17:57 2019
Command line        : /cache/xg17i000/x10007/io500-sc-19/bin/ior -r -R -s 385000 -a IME -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/IOR_file -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/stonewall
Machine             : Linux c0468.ofp
TestID              : 0
StartTime           : Thu Nov 14 08:17:57 2019
Path                : /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard
FS                  : 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.9%
[1675] 2019-11-14 08:18:09 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x2c7b150 172.30.140.236@verbs-ib C P3-peer05 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1675] 2019-11-14 08:18:09 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x2c7b150 172.30.140.236@verbs-ib C P3-peer05 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)

Options: 
api                 : IME
apiVersion          : 1.2.2 (1573)
test filename       : /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 385000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 16384
clients per node    : 8
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 269.68 TiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
WARNING: Expected aggregate file size       = 296518942720000.
WARNING: Stat() of aggregate file size      = 193356536741888.
WARNING: Using actual aggregate bytes moved = 193356536741888.
read      316927     45.91      45.91      0.402839   581.29     0.139831   581.83     0   
Max Read:  316927.14 MiB/sec (332322.19 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       316927.14  316927.14  316927.14       0.00 7069481.53 7069481.53 7069481.53       0.00  581.83457         NA            NA     0  16384   8    1   0     1        1         0    0 385000    47008    47008 184399168.0 IME      0
Finished            : Thu Nov 14 08:28:17 2019